整合多種激活函數的硬體實現

The Hardware Design for Integration of Multiple Activation Functions

張智翔(Chih-Hsiang Chang)

指導教授 ： 黃世旭

中原大學/電機資訊學院/電子工程研究所/碩士(2019年)

https://doi.org/10.6840/cycu201900578

摘要

由於近幾年人工智慧（Artificial Intelligence, AI）的興起，尤其是深度學習（Deep Learning），它可以應用在許多領域，如電腦視覺、語音識別、自然語言處理等等。為了提高效率，AI加速器的需求也在逐漸增加。在神經網路（Neural Network, NN）中，激活函數（Activation Function, AF）可以在各節點輸入輸出之間生成非線性映射，讓神經網路的學習結果更好。在此篇論文中，我們利用了激活函數Hyperbolic Tangent特性來減少硬體成本，並利用激活函數Hyperbolic Tangent及Sigmoid之間的數學關係式，將多個激活函數整合成單一硬體架構，共享部分的模塊（Module）來減少硬體成本。另外，我們也提出新的近似乘法器設計，結合分段線性逼近法（Piecewise Linear Approximation, PWL），可以用更少的面積及功耗來實現任何的激活函數。

關鍵字

深度學習 ； 神經網路 ； 激活函數 ； 硬體實現

並列摘要

In recent year, Artificial Intelligence (AI) technology grows up dramatically. Especially deep learning technology can be applied to many fields, such as computer vision, speech recognition, natural language processing, and so on. In a neural network, the activation function defines the output of that node. In this thesis, we reduce hardware cost of Hyperbolic Tangent function by utilizing its mathematical feature. Moreover, we also propose a hardware design for the integration of both Hyperbolic Tangent function and Sigmoid function according to their mathematical relationship. We try to save the hardware cost by sharing some modules. Besides, we also propose a new approximate multiplier. By incorporating our proposed approximate multiplier with piecewise linear approximation, we can implement any activation function with smaller area and smaller power.

並列關鍵字

Deep Learning ； Neural Network ； Activation Function ； Hardware Implementation.

參考文獻

[1] I. Sutskever, O. Vinyals, and Q.V. Le, “Sequence to sequence learning with neural networks”, Proc. 2014 Advances in Neural Information Processing Systems (NIPS), ARXIV eprint, December 2014, pp. 3104-3112, arXiv:1409.3215.

[2] K. Karpathy, and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions”, Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, June 2015, pp. 3128-3137, doi:10.1109/CVPR.2015.7298932.

[3] A. Graves, A. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks”, Proc. 2013 IEEE international Conference on Acoustics, Speech and Signal Processing, IEEE press, May 2013, pp. 6645-6649, doi: 10.1109/ICASSP.2013.6638947.

[4] B. Lee, and N. Burgess, “Some Results on Taylor-Series Function Approximation on FPGA”, in Proc. of Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, vol. 2, pp. 2198-2202, Nov. 2003.

[5] K. Basterretxea, J. Tarela, I. D. Campo, "Digital design of sigmoid approximator for artificial neural networks", Electron. Lett., vol. 38, no. 1, pp. 35-37, Jan. 2002.