對分散式深度學習的計算與傳輸之排程優化

Computation and Communication Scheduling Optimization for Distributed Deep Learning Systems

蔡慶源(Ching-Yuan Tsai)

指導教授 ： 劉邦鋒

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2018年)

https://doi.org/10.6342/NTU201801485

摘要

深度學習是一種可以解決複雜問題的技術。因為數據的增長和模型的複雜性，大規模的深度學習已經成了一個重要的問題。分佈式深度學習是一種有效的方式訓練一個大型模型。在分散式環境下，網絡帶寬是性能瓶頸。本文的重點是如何安排網路活動以減少訓練時間。我們提出一些調度程序，並獲得最多25 % 的加速。

關鍵字

深度學習 ； 機器學習 ； 參數伺服器 ； 網路瓶頸

並列摘要

Deep learning is a technique that can solve complex problem. Due to the growth of data and model complexity, large-scale deep learning has became an important issue. Distributed deep learning is a efficient way to train a large model. Under distributed environment, network bandwidth is a performance bottleneck. This paper focus on how to schedule network events to reduce training time. We propose some schedulers and get at most 25% speedup.

並列關鍵字

Machine learning ； Deep learning ； Parameter server ； Network limitation

參考文獻

[1] Spark mllib. http://spark.apache.org/mllib/. Accessed: 2018.

[2] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.

[3] A. F. Aji and K. Heafield. Sparse communication for distributed gradient descent. arXiv preprintarXiv:1704.05021, 2017.

[4] D. Alistarh, J. Li, R. Tomioka, and M. Vojnovic. Qsgd: Randomized quantization for communication-optimal stochastic gradient descent. arXiv preprintarXiv:1610.02132, 2016.

[5] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprintarXiv:1607.06450, 2016.