- 學位論文
基於搜尋引擎與文本探勘之相似文件檢測系統
Similar Document Detection System Based on Search Engine and Text Mining Methods
摘要
自83學年度開始全國碩博士論文建檔計畫，直至109學年度，台灣碩博士論文加值系統 (NDLTDT) 已經累積了論文共計1,241,363篇，其中具全文授權的論文 561,731篇。這26年來，平均每年產生42,804篇論文，每年在論文上傳短短一兩個月的畢業期間，須要比對數萬篇論文與百萬篇論文之相似度，有效從中挑出有抄襲疑慮的論文，是很大的挑戰。本論文自動化蒐集擷取 NDLTDT 的龐大資料庫，設計與開發SDDS (Similar Document Detection System) 系統，基於搜尋引擎與文本探勘技術，分析NDLTDT所有論文摘要，並比對兩兩相似度 (pairwise similarities)，以輔助檢查論文是否有抄襲之疑慮。實驗結果發現，摘要相似度大於50%的論文，占了23%的比例，經檢查部分論文內文後，成功找出有抄襲疑慮的論文。基於文本探勘關鍵字及研究領域之關聯，提出分析關鍵字重要的的分法，也有效提升SDDS 檢測抄襲的回收率 (recall rate)。
並列摘要
National Digital Library of Theses and Dissertations in Taiwan (NDLTDT) was established in 1994. Up to now, there are 1,241,363 theses, including 561,731 full texts. In average, 42,804 theses are published per year in Taiwan. It's a big challenge to compare tens of thousands of theses with millions published theses and rapidly detect highly similar theses within one or two months, usually in June and July for every year. In this thesis, we crawl NDLTDT databases, design and implement the Similar Document Detection System (SDDS) based on search engine and text mining methods. SDDS creates the full-text index for all abstracts of NDLTDT and estimates pairwise similarities among all theses based on string and keyword matchings. Given an abstract text string, SDDS can rapid detect similar theses within ??? seconds. Therefore, we do large-scale experiments to detect highly similar abstracts of theses. The result show that 23% abstracts are 50% similar. We seriously reviewed some abstracts and observed that some theses can be regarded as plagiarism. Consequently, SDDS is useful to rapidly detect similar abstracts before students submit their theses to NDLTDT.
參考文獻
延伸閱讀
- 吳建興（2010）。以關鍵字序列型樣探勘為基礎之文件檢索方法〔碩士論文，淡江大學〕。華藝線上圖書館。https://doi.org/10.6846/TKU.2010.00505
- Cheng, Y. L. (2006). 透過文件特徵－字型、位置及引用文獻搜尋科學文件 [master's thesis, National Central University]. Airiti Library. https://www.airitilibrary.com/Article/Detail?DocID=U0031-0207200917335649
- 蔡明成（2004）。建立文獻識別字進行研究趨勢之探勘-以資料庫相關文獻為例〔碩士論文，元智大學〕。華藝線上圖書館。https://www.airitilibrary.com/Article/Detail?DocID=U0009-0112200611314289
- Hoque, F. A., Mohebujjaman, M., & Noman, N. (2011). Informative Motif Detection Using Data Mining. Research Journal of Information Technology, 3(1), 26-32. https://www.airitilibrary.com/Article/Detail?DocID=20413114-201103-201510230016-201510230016-26-32
- Anbu, M., & Mala, G. S. A. (2015). Investigation of Software Defect Prediction Using Data Mining Framework. Research Journal of Applied Sciences, Engineering and Technology, 11(1), 63-69. https://www.airitilibrary.com/Article/Detail?DocID=20407467-201509-201601200030-201601200030-63-69