- 學位論文
利用熟練者知識加速強化式學習演算法之效率─以西洋雙陸棋作為探討實例
Improving Learning Efficacy of Reinforcement Learning from Seniors' Knowledge ─ A Case Study on TD-GAMMON
摘要
強化式學習(reinforcement learning)是一種特別的機器學習方法。它和一般監督式學習(supervised learning)有很大的不同，它可以不需要靠外部的範例來進行self-play學習，這代表著它可以在未知的環境中藉由與環境互動自我學習，這是監督式學習所作不到的。 然而我們通常並不處於一完全未知的環境。換言之，我們往往有許多與環境有過經驗的前輩知識可以參考。以下棋為例，棋師、棋譜、下棋程式，都可視為經驗的來源，而且各自擁有不同的棋藝。不加以利用殊為可惜。這時如何結合self-play與利用有經驗者的知識有效率的學習並達到青出於藍的效果是我們研究的目標。 本論文將以以TD-Gammon為例驗證我們的概念與想法。TD-Gammon是強化式學習一個很有代表性的範例。它結合TD($lambda $)與類神經網路實現函數逼近(function approximation)來學習玩西洋雙陸棋，已可以在150萬次self-play中學習達到接近世界級棋手的強度。我們在TD-Gammon的self-play的過程中，加入指導者對目前棋局的觀點提供建議，藉此提升學習的效果。另外，我們亦實驗了多指導者參與時對學習效能產生的影響，實驗並討論其結果。 關鍵詞：強化式學習; TD-Gammon
並列摘要
Reinforcement learning is a special machine learning method. It is different from supervised learning. It can learn by self-play without using examples. This represents that it can study in an unknown environment, but supervised learning can’t. But the questions we usually meet aren’t located in the unknown environment. In other words, we usually have a lot of experiences that can be referred. Take chess as the example, we have many sources of experiences. In this case, if we don’t use those knowledge may waste those resources. Therefore the goal we study is how to combine self-play and those knowledge. We take TD-Gammon as the example, TD-Gammon is one of the most impressive applications of reinforcement learning. The learning algorithm in TD-Gammon was a straightforward combination of the TD($lambda $) algorithm and nonlinear function approximation using a multilayer neural network trained by backpropagating TD errors. TD-Gammon learned to play extremely well, near the level of the world's strongest grandmasters. We entered seniors' knowledge in TD-Gammon self-play process, and improving learning efficacy. Moreover, we attempt improving learning efficacy further by using more seniors. Keywords: Reinforcement learning; TD-Gammon
參考文獻
延伸閱讀
- 張哲維（2009）。探討不同類型多媒體教材對低工作記憶能力者之學習成效－以國中數學『式子的化簡』為例〔碩士論文，國立交通大學〕。華藝線上圖書館。https://doi.org/10.6842/NCTU.2009.00355
- 張紘綸（2011）。運用分支截斷與指標應變以改善電腦象棋之效能〔碩士論文，國立中正大學〕。華藝線上圖書館。https://www.airitilibrary.com/Article/Detail?DocID=U0033-2110201613494379
- 郭哲宇（2006）。電腦象棋擴大空步剪裁演算法的設計及實作〔碩士論文，國立臺灣師範大學〕。華藝線上圖書館。https://www.airitilibrary.com/Article/Detail?DocID=U0021-0712200716121221
- 汪昌賢（2011）。Self-Organizing Reinforcement Learning Model〔碩士論文，國立臺灣大學〕。華藝線上圖書館。https://doi.org/10.6342/NTU.2011.02661
- 徐照揚（2023）。Service Intention Recognition System based on Reinforcement Learning〔碩士論文，中原大學〕。華藝線上圖書館。https://doi.org/10.6840/cycu202301105