- 學位論文
輕量化微調的預訓練語言模型引導之系統性分析
Systematic Analysis of Pre-trained Language Model Priming for Parameter-efficient Fine-tuning
摘要
隨著預訓練語言模型(Pre-trained Language Model)的參數量變得越來越大，輕量化微調(Parameter-Efficient Fine-tuning)顯得更為重要，但在少樣本學習(Few-Shot Learning)的情境下進行輕量化微調的效果卻遠遠不及微調整個預訓練模型。 為了解決這個問題，本研究提出在進行輕量化微調前加入一個稱為「引導」(Priming)的訓練過程來強化預訓練語言模型的量化微調效果，並且在一個包含160個不同自然語言處理任務的少樣本資料集上驗證了本方法的有效性。相較於直接進行輕量化微調，經過引導的模型在ARG(Average Relative Gain)分數上達到了近30%的進步量，其表現也超越了其他的輕量化微調基石模型。 除此之外，我們針對引導模型的方法進行了系統性的實驗，分析了在引導階段使用不同訓練演算法和訓練不同參數對於引導效果的影響，並找出最有效的引導方法。本研究的結果將能有效增強輕量化微調在少樣本學習上的表現，並使得大型預訓練語言模型的微調和使用更加有效率。
並列摘要
As the parameter size of pre-trained language models (PLMs) continues to grow, parameter-efficient fine-tuning becomes more important. However, the effectiveness of parameter-efficient fine-tuning is far inferior to that of fine-tuning the entire pre-trained model in the context of few-shot learning. To address this issue, this study proposed a training process called "priming" to enhance the effectiveness of parameter-efficient fine-tuning by strengthening the pre-trained language model before performing the downstream fine-tuning. The effectiveness of this method was verified on a few-shot dataset consisting of 160 different NLP tasks. Compared to directly performing parameter-efficient fine-tuning, the primed model achieved an improvement of nearly 30% in ARG (Average Relative Gain) score and outperformed other parameter-efficient fine-tuning baselines. In addition, we conducted systematic experiments to analyze the impact of different training algorithms and different upstream trainable parameters and identify the most effective priming method. The results of this study will effectively enhance the performance of parameter-efficient fine-tuning in few-shot learning and make fine-tuning and usage of large-scale pre-trained language models more efficient.