適用於異質性資料之聯邦元學習架構

Federated Meta-Learning on Data Heterogeneity

陳昇(Sheng Chen)

指導教授 ： 逄愛君

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2021年)

https://doi.org/10.6342/NTU202102362

摘要

近年來行動裝置在市面上佔有率的大幅成長，使得我們得以收集到大量的數據，而在大數據潮流下，倘若我們能利用這些數據進行深度學習訓練，便能大幅改善裝置的用戶體驗與效能。相對於傳統的集中式學習，聯邦式學習僅傳送模型參數而非資料原始數據，使得它大幅降低資料傳輸成本，並且保障用戶的資料隱私與安全性，讓我們得以在各用戶端訓練深度學習模型。然而由於每個用戶的使用習慣不同，導致擁有的資料呈非獨立同分布，而這樣的資料異質性會使得聯邦式學習模型精度大幅下降。本篇論文將元學習引入聯邦式學習架構，以元學習能夠在用戶端快速適應該用戶資料分布達成模型個人化，以提升整體模型精度。由實驗得以看出，元學習在資料異質性降低且資料類別過多時表現不佳。於是我們破壞元學習時訓練任務的限制，以更有彈性的參數設定來訓練模型，並提出一量尺來量化不同參數訓練下模型的效能。同時我們在辨識階段導入集成學習的精神，使用多個不同的個人化模型合作以作出更準確的預測。從實驗看出我們的方法能在多種非獨立同分布的資料分布下皆增進效能，並且能具備高擴展性，同時證實所提出的量尺能夠幫我們找到最合適的元學習的訓練參數。

關鍵字

聯邦式學習 ； 元學習 ； 資料異質性 ； 非獨立同分布 ； 邊緣運算

並列摘要

Federated learning is currently a widely-used framework to perform machine learning tasks over distributed data under privacy constraints. However, in real-world scenarios, data is often distributed heterogeneously, i.e., Non-IID data, which leads to data diversity and accuracy degradation. As meta-learning utilizes data diversity, federated meta-learning becomes a popular framework to improve model accuracy in recent years. As meta-learning trains models using few-shot tasks, N-way K-shot tasks, data diversity drops, and we require a larger N setting when data is mildly Non-IID, which leads to accuracy reductions. Even though reducing N causes models unable to cover all the classes a user possesses, the training process becomes more flexible. We introduce Better-Than-Random-Guess (BTRG), a metric that helps us find the ideal N setting when training federated meta-learning models. We propose Meta-Learning-Ensemble (Metalens), which allows multiple finetuned models cooperate using ensemble learning in the inference phase. We show that our method outperforms other methods and has better scalability than other meta-learning-based methods.

並列關鍵字

Federated Learning ； Meta-Learning ； Data Heterogeneity ； Non-IID ； Edge Computing

參考文獻

G. J. Annas. Hipaa regulations—a new era of medical­record privacy?, 2003.

M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.

L. Breiman. Bagging predictors. Machine learning, 24(2):123–140, 1996.

F. Chen, M. Luo, Z. Dong, Z. Li, and X. He. Federated meta­learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876, 2018.

A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning: A meta­learning approach. arXiv preprint arXiv:2002.07948, 2020.