深度學習之訊息理論分析

Information-theoretic analysis on Deep Learning

黃師揚(Ng, Suiong)

指導教授 ： 張正宏

國立交通大學/理學院/物理研究所/碩士(2018年)

摘要

人工智慧裡的深度學習近年來帶給科技界重大的衝擊,深度神經網路的學習過
程本質相同於統計物理的趨向平衡問題。對於深度學習動態系統的探討有助於瞭解與學習機制有關的生物問題,例如腦神經網路的成型及生物新演化論。本論文運用資訊論工具來探討此問題,包含以互訊息剖析深度神經網路動態系統的兩相變換,以及運用訊息瓶頸理論觀查該動態系統的最後收斂點。

關鍵字

深度學習 ； 訊息理論 ； 訊息瓶頸 ； 機器學習 ； 信息理論 ； 信息瓶頸 ； 深度神經網路

並列摘要

The recent breakthrough of Deep Learning has had a great impact on science and technology. The training process of Deep Neural Network is essentially the same as the problem of approaching equilibrium in Statistical Physics. The objective of this thesis is to understand this process by utilizing tools in Information Theory. The content includes using Mutual Information to analyze the phase transition during the training process and applying the Information Bottleneck theory to understand how the training dynamics converges to its final state.

並列關鍵字

Deep Learning ； Information Theory ； Information Bottleneck ； Machine Learning ； Deep Neural Network ； DNN

參考文獻

[1] N. Tishby and N. Zaslavsky, “Deep Learning and the Information Bottleneck Principle,””IEEE Information Theory Workshop (ITW)”, pp. 1–5, 2015.

[2] R. Shwartz-Ziv and N. Tishby, “Opening The Black Box of Deep Neural Networks via Information,” arXiv:1703.00810, 2017.

[3] P. Mehta and D. J. Schwab, “An exact mapping between the variational renormalization group and deep learning,” arXiv:1410.3801, 2014.

[4] H. W. Lin, M. Tegmark, and D. Rolnick, “Why Does Deep and Cheap Learning Work So Well?” Journal of Statistical Physics, vol. 168, pp. 1223–1247, 2017.

[5] A. Paul and S. Venkatasubramanian, “Why does Deep Learning work? - A perspective from Group Theory,” arXiv:1412.6621, 2014.