基於重新加權損失函數與預測重排序解決深度學習類別訓練資料失衡問題

Re-weighting Loss Functions and Re-ranking Predictions for the Class Imbalance Problem in Deep Learning

蔡尚倫(Shang-Lun Tsai)

指導教授 ： 王勝德

國立臺灣大學/電機資訊學院/電機工程學研究所/碩士(2020年)

https://doi.org/10.6342/NTU202002524

摘要

數據不平衡是指訓練數據集中類的傾斜分佈。深度學習演算法訓練類別數據量不平衡資料集時，往往在少數類別上表現較差的預測結果。不幸的是真實世界應用中比如異常偵測，往往存在不同類別間訓練數據差距繁多之疑慮。本論文中，我們針對上述問題提出兩種改善方法分別應用於訓練階段與預測階段。訓練階段時，我們提出類別資訊本體相關之重新加權損失函數，藉由信息增益重新定義各類別之相對權重，使訓練模型將注意力放在少數類別上。預測階段時，我們針對先前的研究與實驗觀察，發現深度學習訓練不平衡資料集所產生之缺陷，藉此延伸解決方案及猜想，提出基於特徵比對之重排序預測，運用先前訓練之特徵抽取器，擷取各類別之特徵範本作為輔助集，藉以比對測試資料與其餘弦相似度，綜合上述輔助預測與原有模型預測結果，採取多數決結果作為最終重排序之預測。實驗上我們使用殘差網路架構作為訓練模型，透過long-tailed CIFAR-10與long-tailed CIFAR-100來檢驗我們提出之方法。實驗結果表明在不平衡數據集上，我們提出的方法顯著提升Top-1準確度。

關鍵字

不平衡分類 ； 重新加權損失函數 ； 信息理論 ； 特徵比對 ； 重排序預測 ； 深度學習

並列摘要

The skewed distribution of classes in a training dataset refers to as data imbalance. Deep learning algorithms usually suffer poor performance in rare or minor classes when facing the imbalanced datasets. Unfortunately, real world applications typically have the data imbalance issue, such as anomaly detection. In this thesis, we propose two improved methods for the training and the testing stages to solve the data imbalance problem. For the training process, we introduce a Self-information Relevant Re-weighting Loss Function, a theoretical method that redefines the relative weighting of each class through information gained and makes the training model pay attention to minor classes. For the prediction process, based on the previous research and experimental observations, we found the deficiencies generated by deep learning algorithms when training imbalanced datasets. In our solution, the re-ranking method bases on robust feature matching and the voting mechanism. The feature extractor extracts the feature paradigms from each class as a support set, and each type of feature paradigm is used to match the test data with its cosine similarity. The prediction results generated by the support set compare with the original model prediction by majority voting to obtain the final re-ranking prediction. In the experiments, we use the residual network architecture as our training model, and we validate our proposed method through long-tailed CIFAR-10 and long-tailed CIFAR-100. Experimental results show that our method could be able to achieve significant improvement of Top-1 accuracy on imbalanced datasets.

並列關鍵字

imbalanced classification ； re-weighting loss function ； information theory ； feature matching ； re-ranking prediction ； deep learning

參考文獻

“Github/PlotNeuralNet ,” https://github.com/HarisIqbal88/PlotNeuralNet.

“Types of frequency distribution,” https://getnave.com/blog/frequency-distribution-types/.

Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” in IEEE Conference on Computer Vision and Pattern Recognition, 2019.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Neural Information Processing Systems, 2012.

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations, 2015.