Boosting the Accuracy of a Chinese Factoid Question Answering System with Hybrid Modules and Lightweight Methods

以混和方法模組與輕量級方法建構高正確率中文專名問答系統

李政緯(Cheng-Wei Lee)

指導教授 ： 許聞廉

國立清華大學/電機資訊學院/資訊工程學系/博士(2009年)

摘要

在資訊爆炸的時代，好的資訊搜尋技術幾乎等同於效率的代名詞。在眾多資訊搜尋技術中，專名問答(Factoid Question Answering)技術漸漸在學術圈內受到重視。從1999年開始，TREC、CLEF以及NTCIR等國際型評鑑會議開始興起，研究範圍涵蓋多種語言。中文雖然使用族群眾多，但在相關的研究上似乎仍與英文等主要語言有一段差距。在這一篇論文中，我們將探討此問題，尋求低成本可行的方式來提升中文專名問答系統的表現。我們將研究方向集中在兩個議題上：
(1) 使用整合式技術，整合以知識為基礎的方法與以機器學習為基礎的方法
大多數的問答系統相關研究都是單獨集中在以知識為基礎的方法，或是以機器學習為基礎的方法上，少有整合式技術在重要會議或期刊發表。為了填補此研究空缺，並驗證整合式技術對於實際應用的效益，我們選擇了問題分類器作為研究對象，分別以知識為基礎、以機器學習為基礎、以及使用整合式技術，開發了三個問題分類器。我們將幾組具備「異質」與「未見」特性的問題集，分別應用到這三個分類器。我們發現至少在這個受控制的實驗中，以知識為基礎的分類器表現優於以機器學習為基礎的分類器，同時整合式的分類器又優於以知識為基礎的分類器。驗證了知識與整合式技術的實用性。
(2) 輕量級問答系統技術
目前世上表現好的專名問答系統多少都會採用如句法剖析器、邏輯推理器等複雜費時的處理技術。這類型的技術雖有好處，但卻不適用在資源缺乏的語言或資源限制多的環境中。有鑑於此，我們試圖尋找有用的輕量級技術。我們提出了兩個輕量級問答系統技術，分別是「問題與答案關鍵詞共現總和法」(SCO-QAT, Sum of Co-occurrences of Question and Answer Terms)以及「以對齊法產生的表層模版」(ABSPs, Alignment-based Surface Patterns)。相較於其他以共現法為基礎的作法，SCO-QAT不需要額外的知識資料、不需要導入剔除規則供找不到共現頻率時之用、也不需要額外的工具支援，所有共現頻率資料都直接根據文句擷取模組所回傳的文句來計算。ABSPs則是使用一組根據問題答案集所訓練出來的文句模版，用來捕捉文句中問題關鍵字與答案的關係，計算信心分數用來過濾出可靠的答案。
經過測試，這兩個輕量級技術成功地在測試平台上將針對NTCIR-5資料集的RU正確率從0.445提升到0.535。同時在NTCIR-6資料集上也有0.5 RU正確率的好表現。

關鍵字

問答系統 ； 混和方法 ； 輕量級方法 ； 問題分類 ； 答案過濾 ； 答案排序

並列摘要

Factoid Question Answering (QA) is becoming an increasingly important research area in natural language processing. Since 1999, many international question answering contests have been held at conferences and workshops, such as TREC, CLEF, and NTCIR; and several languages have been tested in monolingual or cross-lingual question answering tasks. Although Chinese is growing in popularity worldwide, there seems to be a performance gap between Chinese question answering systems and some systems used for other languages. In this dissertation, our objective is to improve the performance of Chinese Factoid Question Answering systems. To this end, we investigate in the following two concepts.
(1) Hybrid Modules Comprised of Knowledge-based and Machine Learning based Methods
To date, most research on QA modules has focused on knowledge-based or machine learning based methods, possibly because hybrid methods are costly that both the knowledge-based and machine learning-based methods need to be adjusted, and it necessary to find an appropriate way to combine the methods in a hybrid model. To demonstrate the effect of hybrid modules, we developed a hybrid question classifier and used it to conduct a series of empirical experiments. Specifically, we compared the performances of the knowledge-based classifier, the machine learning based classifier and the hybrid classifier on several heterogeneous unseen questions from various sources. The results showed that the knowledge-based question classifier was more accurate than the machine learning-based classifier, but the proposed hybrid classifier achieved the highest accuracy.
(2) Lightweight Question Answering Methods
Nearly all the top performing systems use heavy methods that require sophisticated techniques, such as parsers or logic provers. However, such techniques are usually unavailable or unaffordable for under-resourced languages or in resource-limited situations. In contrast to state-of-the-art QA systems, we improve a top performing Chinese QA system by using lightweight methods effectively. We propose two lightweight methods, namely the Sum of Co-occurrences of Question and Answer Terms (SCO-QAT) and Alignment-based Surface Patterns (ABSPs). SCO-QAT is a co-occurrence-based answer ranking method that does not need extra knowledge, word-ignoring heuristic rules, or tools. It simply calculates co-occurrence scores based on the passage retrieval results. ABSPs are syntactic patterns trained from question-answer pairs with an alignment algorithm. They are used to capture the relations between terms; and the relations are used to filter answers. We attribute the success of the ABSP and SCO-QAT methods to the effective use of local syntactic information and global co-occurrence information.
By using SCO-QAT and ABSPs, we improved the RU-Accuracy of our testbed QA system, ASQA, from 0.445 to 0.535 on the NTCIR-5 dataset. The system also achieved the top 0.5 RU-Accuracy on the NTCIR-6 dataset. The result shows that lightweight methods are not only less expensive to implement, but also have the potential to achieve state-of-the-art performances.

並列關鍵字

Question Answering ； Hybrid Method ； Lightweight Method ； Question Classification ； Answer Filtering ； Answer Ranking

參考文獻

TSAI, T.-H., WU, S.-H., LEE, C.-W., SHIH, C.-W. AND HSU, W.-L. 2004. Mencius: A Chinese Named Entity Recognizer Using Maximum Entropy-based Hybrid Model. Computational Linguistics & Chinese Language Processing 9, 65-82.

COOPER, R.J. AND RUGER, S.M. 2000. A Simple Question Answering System. In Proc. of TREC.

DAY, M.-Y., LU, C.-H., YANG, J.-T.D., CHIOU, G.-F., ONG, C.-S. AND HSU, W.-L. 2005. Designing an Ontology-based Intelligent Tutoring Agent with Instant Messaging. In Proceedings of ICALT2005.

DAY, M.-Y., ONG, C.-S. AND HSU, W.-L. 2007. Question Classification in English-Chinese Cross-Language Question Answering: An Integrated Genetic Algorithm and Machine Learning Approach. In Proceedings of the the IEEE International Conference on Information Reuse and Integration (IEEE IRI 2007), Las Vegas, Nevada, USA, August, 13-15 2007, 203-208.

GUO, Y. 2004. Chinese Question Answering with Full-Text Retrieval Re-Visited Waterloo.