- 期刊
- OpenAccess
Combining Mutual Information and Entropy for Unknown Word Extraction from Multilingual Code-Switching Sentences
摘要
In multilingual environments, a single statement may include content from more than one language, a phenomenon known as code-switching. Among speakers of Mandarin Chinese, code switching is a frequent occurrence in daily life, and this mixing of different languages poses serious challenges for language processing. This paper collects text corpora including code switching between Mandarin and English and Mandarin and Taiwanese, where Mandarin is the dominant language. Mutual information and entropy are then used as a basis for an algorithm to identify unknown words from multilingual texts which are then automatically referenced for multilingual inclusions. Experimental results show that the proposed method effectively filters unrelated new words, thus improving the accuracy of extracting unknown words.
延伸閱讀
- 張富鈞（2018）。A Study on Automatic Unknown Word Extraction Based on Statistical Approach〔碩士論文，國立暨南國際大學〕。華藝線上圖書館。https://www.airitilibrary.com/Article/Detail?DocID=U0020-1908201804381200
- Alam, M., Goni, O., Shameem, A., Islam, S., Datta, N. K., Ahmed, S., & Moazzam, G. (2021). An Approach for the Normalization of Short Message Service to Detect Shorter Form of Words and Find out Actual Word. International Journal of Electronics and Information Engineering, 13(3), 111-118. https://doi.org/10.6636/IJEIE.202109_13(3).04
- Chueh, C. H., Wang, H. M., & Chien, J. T. (2006). A Maximum Entropy Approach for Semantic Language Modeling. 中文計算語言學期刊, 11(1), 37-55. https://doi.org/10.30019/IJCLCLP.200603.0004
- Han, C. H. (2016). Evaluating the Effects of Interface Designs with Human-Generated Annotation Support for Second Language Reading [master's thesis, National Tsing Hua University]. Airiti Library. https://www.airitilibrary.com/Article/Detail?DocID=U0016-0901201710384176
- Wang, Y. A. (2021). Investigating Positional Encoding in Pre-Trained Language Models [master's thesis, National Taiwan University]. Airiti Library. https://doi.org/10.6342/NTU202100158