利用語意嵌入常識模板與蒙地卡羅樹搜尋法產生連貫性文章

Generate coherent text using semantic embedding, common sense templates and Monte-Carlo tree search methods

陳櫻仁(Ying-Ren Chen)

指導教授 ： 蘇豐文

國立清華大學/電機資訊學院/資訊工程學系所/碩士(2021年)

摘要

自動產生相關且可懂的文字是一項困難的研究，很多現存的自然語言生成系統沒有考慮到詞和詞之間的關係，且沒有理解常識知識。詞和詞或句子間的關係是超出預期的緊密結合，因此缺乏理解常識知識的語言生成系統往往會造出非預期中的段落或句子。為了改進這個問題，我們從ConceptNet裡自動抓取常識知識並結合詞嵌入及深度神經網路篇章連貫性模型到蒙地卡羅搜尋樹裡，在極大的搜尋空間及有限的時間資源裡來尋找次佳解，最後根據使用者給定的初始詞來生成具連貫性的段落。我們也觀察到以統計記數方法而建成的詞嵌入經過調整後，在相似及相關性的任務上比用預測來建成的詞嵌入模型還要精準。我們的詞嵌入在相關性任務上得到了0.679的Spearman分數，勝過了其他預訓練的詞嵌入。我們最後採用人工來評測文字連貫性，評測結果為採用篇章連貫性模型及修改擴增後的ConceptNet所生成的文字更具有連貫性。

關鍵字

自然語言生成 ； ConceptNet ； 常識知識 ； 詞嵌入 ； 蒙地卡羅樹搜尋法

並列摘要

The objective of our research was to generate a coherent, understandable text which is a challenging task. Many of current natural language generation systems that based on word appearance frequency didn't consider the relations between words and lack understanding of commonsense knowledge. The relations between words or sentences combined so closely and subtlety that are often beyond the system's expectation. Therefore, unexpected paragraphs or sentences may be generated that leads to the incoherence of the generated text. To remedy this problem, we extracted commonsense knowledge templated from ConceptNet automatically. We combined a constructed word semantic embedding model and a designed Deep Neural Network of discourse coherence model with Monte-Carlo Tree Search to find suboptimal branches in a large search space and the limited time. Our system can generate a more coherent paragraph given user's input concept. We also observed that with proper techniques, count-based word embedding can perform better than prediction-based one on similarity/relatedness tasks. We get 0.67 Spearman's score on relatedness task which outperforms other pre-trained word embeddings. We evaluated generated paragraphs by human rating, our model can generate more coherent paragraphs when using the discourse coherence model and refined ConceptNet.

並列關鍵字

Natural Language Generation ； ConceptNet ； commonsense knowledge ； word embedding ； Monte-Carlo Tree Search

參考文獻

[1] H. Liu and P. Singh, “Makebelieve: using commonsense knowledge to generate stories,” in AAAI/IAAI, (USA), pp. 957–958, American Association for Artificial Intelligence, 2002.

[2] S. Yu and E. Ong, “Using common-sense knowledge in generating stories,” in PRICAI 2012: Trends in Artificial Intelligence, (Berlin, Heidelberg), pp. 838–843, Springer Berlin Heidelberg, 2012.

[3] P. Yang, F. Luo, P. Chen, L. Li, Z. Yin, X. He, and X. Sun, “Knowledgeable storyteller: A commonsense-driven generative model for visual storytelling,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5356–5362, International Joint Conferences on Artificial Intelligence Organization, 2019.

[4] H. Zhang, Z. Liu, C. Xiong, and Z. Liu, “Grounded conversation generation as guided traverses in commonsense knowledge graphs,” in ACL 2020, Association for Computational Linguistics, 2020.

[5] R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge,” in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pp. 4444–4451, AAAI Press, 2017.