基於詞語分布均勻度的核心詞彙選擇

核心詞彙是不受文本類型、主題、應用情境等影響，穩定使用的詞彙。在自然語言中，核心詞彙的數量相對稀少，卻構成溝通內容的主要部份，因此是語言學習中重要的一環。傳統的核心詞彙選擇方法主要依據專家知識與經驗法則，在語料庫語言學興起後，詞頻與詞彙分布均勻度統計提供了客觀的統計數據協助核心詞彙的選取。在本論文中，我們提出一個多面向均勻度整合公式，使詞語均勻度的計算能夠同時考慮到不同的分類面向。其次，我們也針對傳統公式統計結果偏差的問題，提出詞頻正規化的方法。對於實驗的評估，我們提出了一個以異源語料庫評估核心詞彙的方法，可以比較各種統計公式的優缺點與特性。在實驗結果的部份，我們實際比較了多種不同的核心詞彙表選擇公式，分析不同公式的特質，並驗證了詞頻正規化的確能夠修正傳統公式的缺點。最後，我們也驗證了整合多面向均勻度的計算方法，確實可以選擇到更具核心特質的詞彙。

關鍵字

語料庫語言學；核心詞彙；邊緣詞彙；分布均勻度

並列摘要

Core vocabulary is a set of words that are stable used across different text types, theme, and application scenario. In natural language, the number of core vocabulary is relatively small, the core vocabulary, however, plays an important part in language learning because it constitutes a major part of communication content. The traditional core vocabulary selection method is mainly based on the expert knowledge and rule of experience. With the rise of corpus linguistics, word frequency and dispersion uniformity provide objective statistical data to assist the selection of core vocabulary. In this paper, we propose a formula that integrates multi-dimensional uniformity , so that the estimation of word uniformity can take different classification dimensions into account. Secondly, we also propose a method of word frequency normalization for the problem of deviation of the traditional method. For evaluation, a method of evaluating the core vocabulary with a heterogeneous corpus is proposed and it can compare the advantages, disadvantages, and characteristics of various statistical formulas. In the results, we actually compare the different core vocabulary selection formulas, analyzed the characteristics of different formulas, and verified the word frequency normalization can correct the shortcomings of the traditional formula. Finally, we also verified that the proposed method which integrates multi-dimensional uniformity can pick out the vocabulary with more core characteristics.

並列關鍵字

Corpus Linguistics ； Core Vocabulary ； Fringe Vocabulary ； Dispersion Uniformity

參考文獻

Steering Committee for the Test of Proficiency - Huayu[SC-TOP]. (2016). 8000 Chinese Words. Steering Committee for the Test Of Proficiency-Huayu,http://www.sc-top.org.tw/english/download.php

Google Scholar

Carroll, J. B.(1970).An alternative to Juilland's usage coefficient for lexical frequencies and a proposal for a standard frequency index.Computer Studies in the Humanities and Verbal Behaviour.3(2),61-65.

Google Scholar

Chinese Knowledge Information Processing Group=CKIP(1995).A Description to the Sinica Corpus.Technical Report.(Technical Report).,::Academia Sinica.

Google Scholar

Huang, C.-R.,Zhang, H.,Yu, S.-W.(2005).On predicting and verifying a basic lexicon: proposals inspired by distributional consistency.POLA forever: festschrift in honor of Professor William S-Y. Wang on his 70th birthday.(POLA forever: festschrift in honor of Professor William S-Y. Wang on his 70th birthday).: