支持任意雙線性乘積之向量神經元

Vector-neuron-based Learning with Arbitrary Bilinear Products

范哲誠(Zhe-Cheng Fan)

指導教授 ： 張智星

國立臺灣大學/電機資訊學院/資訊工程學研究所/博士(2019年)

https://doi.org/10.6342/NTU201902888

摘要

向量值神經網路學習近年來在深度學習領域中逐漸成為一個新主題。傳統上，多筆多維度的訓練資料在放進神經網路訓練之前，會被串接成一筆高維度的向量。然而，這樣串接資料的方式會造成在不同筆資料中，相同維度的資訊關係沒有學習好，進而導致訓練效果不是最佳的。因此，在本論文中，我們提出一個新的神經網路架構，稱為任意雙線性積神經網路。在此架構中，每個神經元處理向量資訊，成為向量神經元，並且可以將不同的任意雙線性積應用在此架構中。
除此之外，我們將向量神經元配合循環群代數的概念，應用在捲積神經網路，提出深度循群環網路。在此架構中，訓練資料，輸入資料、輸出資料、特徵圖、捲積核都是三維的矩陣。我們透過高光譜影像去躁、歌聲分離以及影像修補的實驗來驗證所提出來的架構。實驗結果顯示，我們提出來的架構與傳統的神經網路比較下，擁有較好的效果，同時也驗證在訓練的過程中，向量之間關連性是有被學習到的。

關鍵字

向量神經學習 ； 向量神經網路 ； 雙線性積 ； 任意雙線積神經網路 ； 倒傳遞 ； 循環代數 ； 深度循環群網路

並列摘要

Vector-valued neural learning has emerged as a promising direction in deep learning recently. Traditionally, training data for neural networks (NNs) are formulated as a vector of scalars; however, its performance may not be optimal since associations among adjacent scalars are not modeled. In this dissertation, we propose a new vector neural architecture called the Arbitrary BIlinear Product Neural Network (ABIPNN), which processes information as vectors in each neuron, and the feedforward projections are defined using arbitrary bilinear products. Such bilinear products can include circular convolution, seven-dimensional vector product, skew circular convolution, reversed-time circular convolution, or other new products not seen in previous work. Besides, we also employ vector values into convolutional neural networks (CNNs), called deep cyclic group network (DCGN) which uses the cyclic group algebra for convolutional vector-neuron learning. The input to DCGN is a three-way tensor, where the mode-3 dimension corresponds to the dimensionality of the input data. As a proof-of-concept, we apply our proposed network to multispectral image denoising, singing voice separation and image inpainting. Experimental results show that ABIPNN or DCGN obtains substantial improvements when compared to conventional NNs or CNNs, suggesting that associations are learned during training.

並列關鍵字

Vector neural learning ； vector neural network ； bilinear products ； arbitrary bilinear product neural networks ； backpropagation ； deep cyclic group algebra ； deep cyclic group networks

參考文獻

[1] Chase Gaudet and Anthony Maida. Deep quaternion networks. arXiv preprint arXiv: 1712.04604, 2017. [Online]. Source code available at: https:// github.com/gaudetcj/DeepQuaternionNetworks.

[2] D. P. Mandic and V. S. L. Goh. Complex Valued Nonlinear Adaptive Filters: Non- circularity, Widely Linear and Neural Models. Wiley, New York, NY, 2009.

[3] T. Nitta. Complex-Valued Neural Networks: Utilizing High-Dimensional Parameters. Information Science Reference, Hershey, PA, 2009.

[4] A. Hirose. Complex-Valued Neural Networks. Springer, Berlin, 2013.

[5] A. Hirose. Complex-Valued Neural Networks: Advances and Applications. John Wiley and Sons, New York, 2013. [6] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, J.-F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. J Pal. Deep complex networks. In Proc. Int. Conf. Learn. Representations (ICLR), 2018.