A Scalable and Programmable Continuous Restricted Boltzmann Machine in VLSI

連續值局限型波茲曼模型積體電路系統之模組化及程式化設計

盧峙丞(Chih-Cheng Lu)

指導教授 ： 陳新

國立清華大學/電機資訊學院/電子工程研究所/博士(2010年)

https://doi.org/10.6843/NTHU.2010.00513

摘要

Probabilistic models use stochasticity to generalise the natural variability of data, and have been shown promising for reasoning biomedical data or for solving weakly-constrained problems such as pattern recognition. Realising probabilistic models in the Very-Large-Scale-Integration (VLSI) is thus attractive for the application like intelligent sensor fusion in implantable devices. However, only a few probabilistic models are amenable to VLSI implementation, and most of which relies greatly on precise computation of Bayesian rules or vector products, which becomes infeasible as transistor noise and hardware non-ideality grow.

This study presents the VLSI system of a scalable and programmable Continuous Restricted Boltzmann Machine (CRBM), a probabilistic model proved to be both useful and hardware-amenable. The probabilistic VLSI system utilises noise to induce continuous-valued stochastic behaviour in VLSI, and based on the CRBM algorithm, the stochasticity in VLSI can be adapted to represent the natural variability in data for pattern recognition. Moreover, as the noise injection makes a circuit’s inputs decide its “output probability” only, the stochasticity in VLSI would further discourage the propagation of noise and computation errors. The full system containing 10 stochastic units, and 25 adaptable connections has been designed and fabricated with the TSMC 0.35m 2P4M CMOS process. Modular design has been employed such that the system can expand its size by inter-connecting multiple chips, extending its application to reasoning high-dimensional, complex data, e.g. real biomedical signals. The system’s capability of modelling and classifying real biomedical data is examined in the context of sorting neuronal spikes. Furthermore, to enhance the system’s performance and flexibility, parameters in the system are not only adaptable by chip-in-a-loop training, but also externally-programmable through a computer.

The capability of the system to model high-dimensional biomedical data, as well as the feasibility of using noise-induced stochastic behaviour to enhance the robustness of analogue computation will be examined and discussed. The latter is especially important when the VLSI technology heads towards the deep-sub-micron era, wherein transistor noise and hardware nonidealities are expected to increase dramatically. The feasibility will then support the suggestion that transistor noise could be used rather than suppressed in future computation, as is what biological neurons have being doing.

關鍵字

超大型積體電路 ； 雜訊 ； 機率型模型 ； 波茲曼模型 ； 可程式化及模組化系統 ； 隨機系統

並列摘要

機率型演算法利用其隨機性來概括信號資料的變異，已經廣泛的應用在生醫訊號分析與解決如圖形辨識等問題上，所以實現在超大型積體電路上之機率型演算法對於植入式之智慧型感測資料融合系統或生醫檢測儀器是很有潛力的。然而，適合實現在超大型積體電路上的機率型演算法並不多，且大部分的機率型演算法需很準確的計算貝斯定理(Bayesian rule)或向量乘積(vector product)，如此，當元件雜訊或是電路非線性度增加時，硬體實現的困難度便會隨之提升。

本研究提出一個可程式化及模組化之「連續值局限型波茲曼模型」 (Continuous Restricted Boltzmann Machine, CRBM)，此模型已證明適合於硬體。此機率型之超大型積體電路系統利用雜訊注入來產生機率行為，且基於此CRBM 演算法，在超大型積體電路上之隨機性可以被訓練 來表示資料感測信號之變異並用在圖形之辨識上。由於電路的輸入只決定其“輸出之機率”，此特性可以阻擋雜訊與計算之錯誤往下一級傳遞。系統使用TSMC 0.35m 2P4M標準製程實現，包含十個隨機型運算單元與二十五個可調整權重參數之突觸連結。模組化設計使得系統可以多晶片相互連結，擴充運算單元數目來處理高維度與複雜之資料，如真實世界之生醫信號。而系統之重建與生醫信號辨識能力則透過分類神經脈衝展示出來。另外，為了增加系統效能與彈性，系統參數不但可以經由外部電腦來寫入，也可以經由晶片迴路訓練來設定。

在此，我們將會探討系統處理高維生醫信號之能力，並且討論利用雜訊注入之隨機行為來提高類比運算穩健性的可能性。尤其是機率行為，在未來超大型積體電路製造技術邁入深次微米時代會顯得更重要，屆時元件雜訊跟電路非線性度預期會更嚴重。就如同真實生物神經訊號的運算方式，雜訊可以使用在運算過程而非抑制運算的準確度。

並列關鍵字

VLSI Implementation ； Noise ； Probabilistic Model ； Boltzamnn Machine ； Scalable and Programmable Systems ； Stochastic systems

參考文獻

[1]A. Schwartz, "Cortical neural prosthetics," Annual review neuroscience, vol. 27, pp. 487-507, July 2004.

[2]K. Wise, D. J. Anderson, J. F. Hetke, D. R. Kipke, and K. Najfi, "Wireless implantable microsystems: high-density electronic interfaces to the nervous system," Proceedings of the IEEE, vol. 92, no. 1, pp. 76-97, 2004.

[4]N. H. Hamid, A. F. Murray and S. Roy, "Time-domain modeling of low-frequency noise in deep-submicrometer MOSFET," IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 55, no. 1, pp. 245-257, 2008.

[5]E. Johannessen, L. Wang, C. Wyse, D. R. S. Cumming, and J. M. Cooper, "Biocompatibility of a lab-on-a-pill sensor in artificial gastrointestinal environments," IEEE Transactions on Biomedical Engineering, vol. 53, no.11, pp. 2333-2340, 2006.

[7]H. Chen and A. Murray, "Continuous restricted Boltzmann machine with an implementable training algorithm," IEE Proceedings-Vision Image and Signal Processing, vol. 150, no. 3, pp. 153-159, 2003.