中文關鍵詞偵測的探討

在資訊爆炸的大數據時代，需要快速有效率獲取關鍵訊息，透過搜尋引擎之類的資料檢索，配合適當字詞（或關鍵詞）可準確及有效率地找出目標。然而定義關鍵詞取決於搜尋者的目標需求，也與各文章所屬領域有關，多半仍需仰賴專家意見決定哪些詞彙是關鍵詞。近年也有不少研究聚焦於關鍵詞的數位分析，藉由相關領域專家的標示與協助，以監督學習的方式決定關鍵詞，但成效仍有待加強，猜測與沒有挑選適當的解釋變數有關。本文以量化模型決定關鍵詞為研究目標，尤其著重於測試哪些變數與關鍵詞關聯性較高；此外，本文也將提出一種非監督學習的關鍵詞挑選方法，不需依賴專家意見。本文以關鍵詞偵測為研究主題，依據詞頻及文本頻率（TF-IDF，Term Frequency Inverse Document Frequency）、卡方檢定值（Chi-Square Test）、RAKE（Rapid Automatic Keyword Extraction）、吉尼係數（Gini Index）等測量值，比較監督學習、非監督學習模型的關鍵詞偵測效果。實證分析為《新青年》、《人民日報》、《自由時報》、《蘋果日報》等屬於白話文的文本，先由人文學者標記關鍵詞（視為標準答案），再以交叉驗證比較各監督學習模型的優劣。其中，監督學習包括常見的統計學習、機器學習模型，包括：羅吉士迴歸、分類樹、隨機森林、類神經網路、支持向量機、極限梯度提升。本文也考量在沒有專家意見輔助下，結合空間分析、逐次分析等統計方法，嘗試建立非監督學習的關鍵詞偵測方法，再與上述監督學習模型比較詞偵測準確性。研究結果顯示適當變數在偵測關鍵詞扮演重要角色，而成效評估的部分由監督學習模型結果較佳，F1值高出非監督學習模型約20%，而非監督學習模型的F1值也能達到45%左右的水準。

關鍵字

文字探勘；關鍵詞偵測；空間統計；非監督學習；交叉驗證

並列摘要

It becomes an essential skill to effectively search and obtain important information in the era of big data. With suitable words and phrases (or keywords), the search engines can help us to acquire most relevant information. However, there are no specific rules yet for defining keywords, depending on the study goal and the nature of texts, and people usually rely on expert’s opinions to determine the keywords. Even with the expert’s feedback, the accuracy of applying quantitative models for keyword extraction is usually lower than 50%. We think the key reason is lacking appropriate explanatory variables. In this study, we should explore if including more variables can increase the accuracy. Also, we will propose an unsupervised learning method, based on the ideas from spatial analysis and sequential analysis, to detect keywords. For the variables of keyword extraction, we choose Chi-Square Test, RAKE(Rapid Automatic Keyword Extraction), and Gini’s Index, in addition to the term frequency and text frequency (TF-IDF, Term Frequency Inverse Document Frequency) which is often used in the past studies. Both statistical and machine learning models are used in this study, including logistic regression, classification tree, Random Forest, Neural Network, Support Vector Machine, and Extreme Gradient Boosting. We choose articles from “New Youth Magazine”, “People’s Daily”, “The Liberty Times”, and “Apple Daily” in empirical study, and all texts belong to modern Chinese writing style. First, humanity scholars mark keywords (treated as true answers) and then we use cross-validation to evaluate the model performance. We found that adding more variables can increase the accuracy of keyword extraction and the supervised learning models have higher accuracy. Still, the proposed unsupervised learning model can still achieve about 45% of accuracy.