人工神經網路演化的熱力學行為

The thermodynamic behaviors of the evolution of artificial neural networks

李承鴻(Cheng-Hung Li)

指導教授 ： 張正宏

國立交通大學/理學院/物理研究所/碩士(2020年)

摘要

本論文探討人工神經網路在訓練過程權重演化的熱力學行為。我們建構簡單人工神經網路，將網路權重狀態標示於權重空間，觀察權重狀態在訓練過程的演化，發現它會如同物理系統收斂到平衡點，特別的是此收斂過程是以階梯式進行。到達平衡點附近，權重狀態不會停在平衡點，而是會受到隨機力作用在平衡點附近擾動，我們發現該隨機力維度比權重空間低，不是完全隨機。我們分析了階梯式演化與有序隨機力的來源。最後利用中央極限定理，累加隨機力成高斯分布，算出隨地而異的乘性噪聲的共變異數矩陣，奠立了使用朗之萬方程、最小作用量、以及路徑積分來描述人工神經網路演化的基礎。

關鍵字

人工神經網路演化 ； 熱力學 ； 隨機熱力學 ； 噪聲分析

並列摘要

In this thesis, we discuss the thermodynamic behaviors of the weight evolution of artificial neural networks during the training process. We set up the simple artificial neural networks and represent their weight state in the weight space. After tracing the evolution of the weight state during the training process, we find that that state will converge to the equilibrium point like physical systems. A special feature during this process is that the convergence is cascaded. When the weight state comes close to the equilibrium point, it will not stop at that point. Instead, it will be spread out by the random force near the equilibrium point. We found that the random force has a dimension less than that of the weight space and thus is not completely random. Finally, the central limit theorem is used to sum up random forces into a Gaussian distribution and the covariance matrix of multiplicative noise is calculated, which varies from the weight state. These set up the foundation of describing the evolution of artificial neural networks by using Langevin equation, the least action, and the path integral formalism.

並列關鍵字

the evolution of artificial neural networks ； thermodynamic ； stochastic thermodynamic ； the analysis of noise

參考文獻

[1] A. H.Cater, Classical And Statistical Thermodynamics, Pearson international ed.(Pearson Education, New Jersey, 2009),Chapter 6, pp.85-97.

[2] D. V.Schroeder, An Introduction to Tthermal Physics, (Addison-Wesley, U.S.A, 1999), Chapter 2, pp.74-84.

[3] B. Alberts, A. Johnson, and J. Lewis, Molecular Biology of the Cell, 4th ed.( Garland Science, New York, 2002).

[4] U.Seifert, Stochastic thermodynamics, (Institute for Theoretical Physics, University of Stuttgart, 2008), B5,pp.2-5.

[5] H. Bindu and Srilatha, J Stem Cell Res Ther. 1(3) ,1000115, (2011).