A Maximum Entropy Approach for Semantic Language Modeling

Chuang-Hua Chueh ； Hsin-Min Wang ； Jen-Tzung Chien

《中文計算語言學期刊》 11卷1期 (2006/03) Pp. 37-55

https://doi.org/10.30019/IJCLCLP.200603.0004

並列摘要

The conventional n-gram language model exploits only the immediate context of historical words without exploring long-distance semantic information. In this paper, we present a new information source extracted from latent semantic analysis (LSA) and adopt the maximum entropy (ME) principle to integrate it into an n-gram language model. With the ME approach, each information source serves as a set of constraints, which should be satisfied to estimate a hybrid statistical language model with maximum randomness. For comparative study, we also carry out knowledge integration via linear interpolation (LI). In the experiments on the TDT2 Chinese corpus, we find that the ME language model that combines the features of trigram and semantic information achieves a 17.9% perplexity reduction compared to the conventional trigram language model, and it outperforms the LI language model. Furthermore, in evaluation on a Mandarin speech recognition task, the ME and LI language models reduce the character error rate by 16.9% and 8.5%, respectively, over the bigram language model.

並列關鍵字

Language Modeling ； Latent Semantic Analysis ； Maximum Entropy ； Speech Recognition

參考文獻

Bellegarda, J.(2000).Exploiting latent semantic information in statistical language modeling.(Proceedings of the IEEE).

Bellegarda, J.,J. Butzberger, Y. Chow, N. Coccaro,,D. Naik(1996).A novel word clustering algorithm based on latent semantic analysis.(IEEE Proceedings of International Conference on Acoustic, Speech and Signal Processing (ICASSP)).

Berger, A.,S. Della Pietra,V. Della Pietra(1996).A maximum entropy approach to natural language processing.(Computational Linguistics).

Berry, M.,S. Dumais,G. O`Brien(1995).Using linear algebra for intelligent information retrieval.SIAM Review.37(4),573-595.

Chelba, C.,F. Jelinek(2000).Structured language modeling.(Computer Speech and Language).