未登錄詞之向量表示法模型於中文機器閱讀理解之應用

An OOV Word Embedding Framework for Chinese Machine Reading Comprehension

羅上堡(Shang-Bao Luo) ； 李青憲(Ching-Hsien Lee) ； 涂家章(Jia-Jang Tu) ； 陳冠宇(Kuan-Yu Chen)

《中文計算語言學期刊》 23卷2期 (2018/12) Pp. 67-83

摘要

在使用深度學習（Deep Learning）方法於自然語言處理的問題時，我們通常會先將每一個詞以一個相對應的詞向量（Word Embedding）表示，再輸入至各式神經網路模型。當遭遇未登錄詞（Out-of-Vocabulary, OOV）的問題時，最常見的處理方式是略去該未登錄詞、以一個零向量表示或是用一個隨機產生的向量表示這個未登錄詞。就我們所知，在目前的研究裡，似乎仍未有一套合理且快速的做法，用於產生未登錄詞的詞向量表示法，並進一步地探索未登錄詞的詞向量對於任務成效的影響性。因此，本論文提出一套新穎的詞向量表示法學習技術，其目標是為未登錄詞產生一個較為合理且可靠的低維度向量表示法；除此之外，我們將進一步地把此一技術運用於中文機器閱讀理解任務之中，探究未登錄詞對於中文機器閱讀理解任務之影響，並驗證本論文所提出的詞向量表示法學習技術之成效。

關鍵字

自然語言處理 ； 詞向量表示法 ； 未登錄詞 ； 機器閱讀理解

並列摘要

When using Deep Learning methods in NLP-related tasks, we usually represent a word by using a low-dimensional dense vector, which is named the word embedding, and these word embeddings can then be treated as feature vectors for various neural network-based models. However, a major challenge facing such a mechanism is how to represent OOV words. There are two common strategies in practiced: one is to remove these words directly; the other is to represent OOV words by using zero or random vectors. To mitigate the flaw, we introduce an OOV embedding framework, which aims at generating reasonable low-dimensional dense vectors for OOV words. Furthermore, in order to evaluate the impact of the OOV representations, we plug the proposed framework into the Chinese machine reading comprehension task, and a series of experiments and comparisons demonstrate the good efficacy of the proposed framework.

並列關鍵字

Natural Language Processing ； Word Embedding ； Out-of-vocabulary ； Machine Reading Comprehension

參考文獻

Abadi, M.,Barham, P.,Chen, J.,Chen, Z.,Davis, A.,Dean, J.,Zheng, X.(2016).Tensorflow: Large-scale machine learning on heterogeneous distributed systems.Proceeding of OSDI'16 Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation.(Proceeding of OSDI'16 Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation).

Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. Retrived from arXiv:1607.06450

Bengio, Y.,Ducharme, R.,Vincent, P.,Jauvin, C.(2003).A neural probabilistic language model.Journal of machine learning research.3,1137-1155.

Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2016). Enriching word vectors with subword information. Retrieved from arXiv:1607.04606

Chen, K.-Y.,Wang, H.-M.,Chen, H.-H.(2015).A Probabilistic Framework for Chinese Spelling Check.Transactions on Asian and Low-Resource Language Information Processing.14(4),15.