Enhancement of Feature Engineering for Conditional Random Field Learning in Chinese Word Segmentation Using Unlabeled Data

Mike Tian-Jian Jiang ； Cheng-Wei Shih ； Ting-Hao Yang ； Chan-Hung Kuo ； Richard Tzong-Han Tsai ； Wen-Lian Hsu

《中文計算語言學期刊》 17卷3期 (2012/09) Pp. 45-85

https://doi.org/10.30019/IJCLCLP.201209.0003

並列摘要

This work proposes a unified view of several features based on frequent strings extracted from unlabeled data that improve the conditional random fields (CRF) model for Chinese word segmentation (CWS). These features include character-based n-gram (CNG), accessor variety based string (AVS) and its variation of left-right co-existed feature (LRAVS), term-contributed frequency (TCF), and term-contributed boundary (TCB) with a specific manner of boundary overlapping. For the experiments, the baseline is the 6-tag, a state-of-the-art labeling scheme of CRF-based CWS, and the data set is acquired from the 2005 CWS Bakeoff of Special Interest Group on Chinese Language Processing (SIGHAN) of the Association for Computational Linguistics (ACL) and SIGHAN CWS Bakeoff 2010. The experimental results show that all of these features improve the performance of the baseline system in terms of recall, precision, and their harmonic average as F1 measure score, on both accuracy (F) and out-of-vocabulary recognition (FOOV). In particular, this work presents compound features involving LRAVS/AVS and TCF/TCB that are competitive with other types of features for CRF-based CWS in terms of F and FOOV, respectively.

並列關鍵字

Conditional Random Fields ； Word Segmentation ； Accessor Variety ； Term-contributed Frequency ； Term-contributed Boundary

參考文獻

Chang, J.-S.,Su, K.-Y.(1997).An Unsupervised Iterative Method for Chinese New Lexicon Extraction.Proc. Computational Linguistics and Chinese Language Processing.(Proc. Computational Linguistics and Chinese Language Processing).

Chang, T.-H.,Lee, C.-H.(2003).Automatic Chinese unknown word extraction using small-corpus-based method.Proc. International Conference on Natural Language Processing and Knowledge Engineering.(Proc. International Conference on Natural Language Processing and Knowledge Engineering).

Chien, L.-F.(1997).PAT-tree-based Keyword Extraction for Chinese Information Retrieval.Proc. 20th Annnual International ACM SIGIR Conference on Research and Development in Information Retrieval.(Proc. 20th Annnual International ACM SIGIR Conference on Research and Development in Information Retrieval).

Cohen, P.,Adams, N.,Heeringa, B.(2007).Voting Experts: An Unsupervised Algorithm for Segmenting Sequences.Intelligent Data Analysis.11(6),607-625.

Emerson, T.(2005).The Second International Chinese Word Segmentation Bakeoff.Proc. 4th SIGHAN Workshop on Chinese Language Processing.(Proc. 4th SIGHAN Workshop on Chinese Language Processing).