- 學位論文
透過多階段的資料增強與回饋機制來優化大型語言模型的推理與可用性
Enhancing Large Language Models Across Training Stages via Synthetic Data and Iterative Feedback
若您是本文的作者，可授權文章由華藝線上圖書館中協助推廣。
摘要
大型語言模型在各種任務中展現出卓越的能力，但其效能與可用性仍可透過精細優化進一步提升。本論文專注於大型語言模型訓練流程中的兩個關鍵階段：預訓練與後訓練。在預訓練階段，我們提出特定領域的資料整理與模型開發策略，以打造適用於特定語言與情境的大型語言模型，例如針對繁體中文的資料處理與評測方法及 Taiwan-LLM 模型。在後訓練階段，我們透過合成資料生成、偏好最佳化與迭代回饋來提升模型的能力，包括用於意圖識別的資料增強、用於自動評測與對齊的 LLM-Eval，以及用於逐步偏好最佳化的 Step-KTO。實驗結果顯示，合成資料增強可提升大型語言模型在低資源環境下的穩健性，而迭代回饋與精心設計的獎勵信號則能改善推理能力與輸出品質。本研究展示了一套完整的多階段強化框架，從預訓練到後訓練，以打造更符合語言、文化及實際應用需求的 LLM。
並列摘要
Large Language Models (LLMs) have demonstrated remarkable capabilities, but their performance, alignment, and trustworthiness can still be significantly improved through fine-grained optimization across multiple training stages. This thesis focuses on enhancing LLMs in two key stages: pre-training and post-training. In pre-training, we introduce domain-specific data curation and model development strategies to produce LLMs tailored for specific languages and contexts, such as Traditional Mandarin processing and benchmarking (e.g., the Taiwanese Mandarin Language Understanding benchmark and Taiwan-LLM). In post-training, we refine models using synthetic data generation, preference optimization, and iterative feedback, including methods like In-Context Data Augmentation for intent detection, LLM-Eval for automatic evaluation, and Step-KTO for stepwise preference optimization. Experimental results show that synthetic data augmentation enhances robustness in low-resource settings, while iterative feedback and well-designed reward signals improve reasoning and output quality. This framework demonstrates how multi-stage enhancements can create LLMs that are linguistically, culturally, and pragmatically aligned with user needs.
參考文獻
fasteval, Fasteval, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL: https://arxiv.org/abs/2303.08774.
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive Science, 9(1):147–169, 1985. doi:10.1207/S15516709COG0901_7. URL: https://doi.org/10.1207/s15516709cog0901_7.
Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. Reason first, then respond: Modular generation for knowledge-infused dialogue. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7112–7132. Association for Computational Linguistics, 2022. doi:10.18653/V1/2022.FINDINGS-EMNLP.527. URL: https://doi.org/10.18653/v1/2022.findings-emnlp.527.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.