多層感知器對輸入與權值誤差的敏感度分析及倒傳遞(BP)演算法與進化策略(ES)演算法的改善

Sensitivity Analysis of the Multilayer Perceptron due to the Errors of the Inputs and Weights & Improvements in BP and ES Algorithms

楊盛松(Sheng-Sung Yang)

指導教授 ： 賀嘉律

國立中央大學/資訊電機學院/電機工程學系/博士(2007年)

摘要

多層感知器(Multilayer Perceptron)，簡稱為MLP，常被用於一些演算法中，例如：倒傳遞演算法(Back-Propagation Algorithm, 簡稱BP)、進化演算法(Evolutionary Algorithm, 簡稱EA)及快速學習法(Extreme Learning Machine, 簡稱ELM)等。其中倒傳遞演算法及進化演算法是最常使用MLP架構的，且它們的性能表現往往會受到MLP架構的影響。因此，決定MLP的架構(層數及每一層神經元的個數)對這些演算法是一件非常重要的事。本論文的重點就是探討當MLP中的輸入(input)及層與層神經元之間的權值(weight)改變時，MLP輸出的變化量，亦即輸出對輸入變化及層與層神經元之間權值變化的敏感度(sensitivity)，針對不同的MLP架構，此敏感度也不同。藉由此敏感度的大小，可選擇一個較適合的MLP架構。對MLP敏感度的研究，我們採用中央極限定理(Central Limit Theorem, 簡稱CLT)來幫助我們做統計上的運算，同時CLT也可用於計算分開複數(Split-Complex)MLP架構的敏感度，此種MLP架構簡稱為Split-CMLP，可用於複數訊號系統，例如QPSK訊號系統等。因此，本論文同時分析一般MLP及Split-CMLP對輸入變化及層與層神經元之間權值變化的敏感度。在論文的後半部份，我們結合了階層式(hierarchical)結構及倒傳遞演算法(BP)來改善標準BP演算法的性能，此新的演算法稱為HBP演算法；同時也為目前使用極為廣泛的進化演算法-進化策略(Evolutionary Strategy, 簡稱ES)決定運算參數，以增進其性能。

關鍵字

敏感度 ； 多層感知器 ； 倒傳遞演算法 ； 進化策略演算法

並列摘要

Multilayer Perceptron (MLP) is often used in some algorithms such as Back-Propagation (BP) algorithm, Evolutionary algorithm (EA), Extreme Learning Machine (ELM) algorithm, etc. Among these algorithms, BP and EA algorithms are more commonly operated in MLP structures to implement some applications than ELM is. Furthermore, the used MLP structures always affect the performances of these algorithms. Therefore, it is a substantial work to decide a feasible MLP structure (i.e., to decide number of layers and number of neurons in each layer) for each one of these algorithms. The main work of this dissertation is to analyze the adjustment of the output of the MLP due to the adjustments of the inputs and the weights between the neurons in adjacent layers (i.e., to analyze the sensitivity of the MLP due to the errors of the inputs and weights). Different MLP structure will lead to different sensitivity value. Based on the sensitivity values, it is feasible to choose a proper MLP structure for the related algorithm. In order to study the sensitivity of a MLP, we use the Central Limit Theorem (CLT) in the statistical computation of the sensitivity. Moreover, the CLT can also be extended to the sensitivity computation of the split-complex MLP (Split-CMLP); the Split-CMLP can be used in a complex signal system such as QPSK signal system, etc. Therefore, we analyze the sensitivity of both MLP and Split-CMLP in this dissertation. On the other hand, we combine the hierarchical structure and BP algorithm in this dissertation to improve the performance of the standard BP algorithm, and this new algorithm is named as HBP algorithm. Additionally, we also introduce an approach in this dissertation to decide the operation parameters of the Evolutionary Strategy (ES) algorithm-the most popular one of the Evolutionary algorithms, to improve its performance.

並列關鍵字

evolutionary strategy ； back-propagation ； multilayer perceptron ； sensitivity

參考文獻

[39] C. H. Chang, S. Siu and C. H. Wei, “Decision feedback equalization using complex backpropagation algorithm,” in Proc. of IEEE International Symposium on Circuits and Systems, Hong Kong, pp. 589-592, June 1997.

[1] M. Stevenson, R. Winter, and B. Widrow, “Sensitivity of feedforward neural networks to weight errors,” IEEE Trans. Neural Networks, vol. 1, pp. 71-80, Mar. 1990.

[3] S. W. Piche, “The selection of weight accuracies for Madalines,” IEEE Trans. Neural Networks, vol. 6, pp. 432-445, Mar. 1995.

[4] S. Hashem, “Sensitivity analysis for feedforward artificial neural networks with differentiable activation functions,” in Proc. IJCNN’92, vol. 1, Baltimore, MD, 1992, pp. 419-424.

[5] L. Fu and T. Chen, “Sensitivity analysis for input vector in multilayer feedforward neural networks,” in Proc. IEEE Int. Conf. Neural Networks, vol. 1, San Francisco, CA, 1993, pp. 215-218.