The Use of Clustering Techniques for Language Modeling-Application to Asian Language

Jian-Feng Gao ； Joshua T. Goodman ； Jiang-Bo Miao

《中文計算語言學期刊》 6卷1期 (2001/02) Pp. 27-60

https://doi.org/10.30019/IJCLCLP.200102.0002

並列摘要

Cluster-based n-gram modeling is a variant of normal word-based n-gram modeling. It attempts to make use of the similarities between words. In this paper, we present an empirical study of clustering techniques for Asian language modeling. Clustering is used to improve the performance (i.e. perplexity) of language models as well as to compress language models. Experimental tests are presented for cluster-based trigram models on a Japanese newspaper corpus and on a Chinese heterogeneous corpus. While the majority of previous research on word clustering has focused on how to get the best clusters, we have concentrated our research on the best way to use the clusters. Experimental results show that some novel techniques we present work much better than previous methods, and achieve more than 40% size reduction at the same level of perplexity.

參考文獻

Brown, Peter F.,Della Pietra, Stephen A.,Pietra Della, Vincent J.,Jelinek, Fredick,Lafferty, John D.,Cocke, John,Roossin, Paul S.,Mercer, Robert L.(1990).A Statistical Approach to Machine Translation.Computational Linguistics.16(2),79-85.

Brown, Peter F.,Della Pietra, Vincent J.,deSouza, Peter V.,Lai, Jennifer C.,Mercer, Robert L.(1992).Class-Based N-gram Models of Natural Language.Computational Linguistics.18(4),467-479.

Chow, Y. L.,Bellegarda, J. R.,Butzberger, J. W.,Coccaro, N. B.,Naik, D.(1996).ICASS-96.

Kenneth, K. K. W., K. W.(1988).ACL Proc. 2nd Conf. on Applied Natural Language Processing.

Fung, P.,Nguyen, L.,Placeway, P.,Schwartz, R.(1993).Proceedings of ICASSP-93.