非獨立同分布之影像資料使用於半監督分散式學習的效能優化

Performance Optimization of Semi-supervised Distributed Learning from Non-IID Data

王捷生(Chieh-Sheng Wang)

指導教授 ： 逄愛君

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2019年)

https://doi.org/10.6342/NTU201903744

摘要

現代的移動終端裝置，例如:手機和行車記錄器，無時無刻都在收 集大量的資料，而這些資料對於很多物聯網和人工智慧的應用是很有 價值的。然而隨著對於隱私問題的日漸重視，使用者不願意將他們的 資料上傳到伺服器做進一步的分析，這牽涉到深度模型的訓練。因此 Google 提出的聯邦學習在不需要回傳資料的情況下，利用分散式學習 的方式訓練一個深度學習模型。然而在現實環境下，終端裝置上的資 料皆為非獨立通分布，會造成訓練時引起權重分歧並最終導致續練出 來的模型性能降低。在這篇論文中，我們提出一種創新的聯邦學習方 案，並設計了一個名為聯邦交換(FedSwap)的新操作，在聯邦訓練期 間替換一些原本的聯邦平均(FedAvg)的操作，以減輕權重分歧所造 成的影響。我們將我們的方法實作圖像分類在 CIFAR-10 標準資料上 和物體偵測在現實世界影像資料上。實驗結果顯示，圖像分類的準確 率提高了 3.8%，物體偵測可以提高 1.1%。

關鍵字

分散式學習 ； 聯邦學習 ； 邊緣運算 ； 非獨立同分佈 ； 半監督式學習

並列摘要

Modern mobile end devices, such as smartphones and dash cams, are collecting numerous data that can be useful for various IoT and AI appli- cations. However, as the concern of privacy rises, users are not willing to provide their data to the server for further data analysis, which involves deep learning model training. Therefore, Google proposes Federated Learning, of- fering distributed end devices a way to train a deep learning model without passing their private data to a centralized server. However, in the real world circumstances, the data on the end devices are non-independent and identi- cally distributed (non-IID) such that it may cause weight divergence during training and eventually result in a considerable decrease in the model perfor- mance. In this thesis, we propose an innovative Federated learning scheme, in which we design a new operation called Federated Swapping (FedSwap) to replace some Federated Averaging (FedAvg) operations based on a few shared data during federated training to alleviate the adverse impact of weight divergence. We implement our method on both image classification using CIFAR-10 benchmark data and object detection using the real world video data. Experiment results show that the accuracy of image classification is increased by 3.8%, and the object detection task can be improved by 1.1%.

並列關鍵字

distributed learning ； federated learning ； edge computing ； non-IID ； semi-supervised learning

參考文獻

[1] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Distributed deep neural networks over the cloud, the edge and end devices. In 2017 IEEE 37th Inter- national Conference on Distributed Computing Systems (ICDCS), pages 328–339, 2017.

[2] Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, pages 685– 693, 2015.

[3] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.

[4] Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.

[5] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4119–4128, 2018.