Chinese Word Segmentation by Classification of Characters

Chooi-Ling Goh ； Masayuki Asahara ； Yuji Matsumoto

《中文計算語言學期刊》 10卷3期 (2005/09) Pp. 381-396

https://doi.org/10.30019/IJCLCLP.200509.0006

並列摘要

During the process of Chinese word segmentation, two main problems occur: segmentation ambiguities and unknown word occurrences. This paper describes a method to solve the segmentation problem. First, we use a dictionary-based approach to segment the text. We apply the Maximum Matching algorithm to segment the text forwards (FMM) and backwards (BMM). Based on the difference between FMM and BMM, and the context, we apply a classification method based on Support Vector Machines to re-assign the word boundaries. In so doing, we use the output of a dictionary-based approach, and then apply a machine-learning-based approach to solve the segmentation problem. Experimental results show that our model can achieve an F-measure of 99.0 for overall segmentation, given the condition that there are no unknown words in the text, and an F-measure of 95.1 if unknown words exist.

並列關鍵字

Chinese ； word segmentation ； segmentation ambiguity ； unknown word ； maximum matching algorithm ； support vector machines

參考文獻

Asahara, M.,C.L. Goh,X.J. Wang,Y. Matsumoto(2003).Combining Segmenter and Chunker for Chinese Word Segmentation.(Proceedings of Second SIGHAN Workshop on Chinese Language Processing).

Chen, K.J.,M.H. Bai(1997).Unknown Word Detection for Chinese By a Corpus-based Learning Method.(Proceedings of ROCLING X).

Chen, K.J.,W.Y. Ma(2002).Unknown Word Extraction for Chinese Documents.(Proceedings of COLING 2002).

Fu, G.H.,K.K. Luke(2003).An Integrated Approach for Chinese Word Segmentation.(In Proceedings of PACLIC 17).

Fu, G.H.,X.L. Wang(1999).Unsupervised Chinese Word Segmentation and Unknown Word Identification.(Proceedings of NLPRS).