在異質架構下深度學習利用多為數組中的稀疏度做動態調整

Adaptive runtime exploiting sparsity in tensor of deep learning on heterogeneous systems

彭國祐(Kuo-You Peng)

指導教授 ： 徐慰中

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2017年)

https://doi.org/10.6342/NTU201700007

摘要

深度學習在現今應用越來越廣，但隨著神經網路越來越深層，所需要的計算也越來越多，所需要的記憶體也越來越多。因此，如何減少計算與記憶體就是一個很重要的議題。
本文就是在探討在神經網路擁有高稀疏度的情況下，我們該如何在異質系統架構上利用高稀疏度來達到減少計算與記憶體容量

關鍵字

深度學習 ； 稀疏度 ； 異質架構

並列摘要

Heterogeneous computing achieves high performance by exploiting high parallelism and special type of computation (such as SIMD operations) available in applications on best fit computation devices. For example, massive and regular SIMD operations can be more efficiently computed on GPU. However, the performance of heterogeneous program can be degraded when the portion assigned to GPU encounters irregular tasks.
Deep learning is an application that has the characteristics of high parallelism but may also encounter irregular tasks. This study introduces a method which could reduce computation and improve the performance in the deep learning application by recording information on the runtime. By using collected information, we can adaptive changing workload when we encounter irregular tasks.
When deep learning encounters irregular tasks, using our method could split the workload of the deep learning application into two parts: dense workload and sparse workload. The dense workload will be deployed on the GPU device, and the sparse part is sent for the CPU. In this way, GPU gets better computing efficiency, and CPU is more competent in handling sparse part than GPU.

並列關鍵字

Adaptive runtime ； deep learning ； heterogeneous systems

參考文獻

[1] MIT Technology Review, “Deep learning” [online]. Available:

[2] CS231n Convolutional Neural Networks for Visual Recognition [online] Available:

[3] Xavier Glorot, A. B., Yoshua Bengio (2011). "Deep Sparse Rectifier Neural Networks."

[7] AMD, Heterogeneous Computing, What is Heterogeneous System Architecture (HSA) , [online] Available:

[8] GOOGLE, Tensorflow, [online] Available: