設計高效且簡明的目標函數以提升增量學習之效能

Improving Incremental Learning with Simple and Efficient Design of Objectives

雷承勲(Cheng-Hsun Lei)

指導教授 ： 邱維辰

國立交通大學/資訊學院/網路工程研究所/碩士(2020年)

摘要

增量學習在影像分類的應用，主要目的是從持續新增的類別中，學習一個分類器，使該分類器能同時對所有學習過的類別之影像進行分類；且在學習的過程中，因受限於儲存設備的大小，只能從之前學習過的類別，保留部分資料，無法重複使用全部的資料做訓練。故對於增量學習而言，最重要的一個問題在於，如何使模型持續且有效地學習新類別，同時保有對過去已學習之類別的辨識能力，也可理解為防止模型產生災難性遺忘。過去眾多的研究，已致力於研發各種目標函數及正則化的策略，而在此篇論文中，我們首先從一個公平且統一的觀點，對先前的相關研究進行分析，統整並對其中不同的目標函數及正則化方法做區分，進而設計一個簡單且高效的目標函數，以提升模型在增量學習中的分類效能。關於此篇論文中所使用的目標函數，包含了二分類交叉熵損失函數，以及一個從特徵圖觀點進行設計的正則化項，並且加入了修改版的餘弦正規化，目的是讓不同類別的特徵能在特徵空間中相互遠離。同時，我們提供了大量的實驗，使用了CIFAR-100以及ImageNet兩個數據集，並結合多種不同的實驗設定，其實驗結果顯示，我們的方法在減少分類錯誤、減輕災難性遺忘以及平衡不同類別之分類準確率上，有著良好的表現，且優於目前最先進的作法。

關鍵字

增量學習 ； 影像分類 ； 餘弦正規化 ； 知識蒸餾 ； 特徵圖正則化

並列摘要

Incremental learning in classification aims to learn a classifier from the data of different classes that arrive sequentially over time, with only limited memory for preserving the seen data samples. A central problem in incremental learning is how to teach the model to learn new classes while keeping its capability of recognizing the seen classes, i.e. to prevent the model from catastrophic forgetting. Many research efforts have been focused on developing various objectives and regularization strategies. In this paper we first analyze these prior works from a unified perspective, categorizing their objectives and regularizations into different granularities. We then propose a simple yet efficient design of the objectives for advancing the classification performance upon incremental learning. Our objectives include a binary cross entropy loss and a regularization term built from the graph perspective, with a rectified cosine normalization step applied to feature representations. Extensive experiments are conducted on both CIFAR-100 and ImageNet datasets under various settings. The experimental results demonstrate that our method outperforms the state-of-the-art approaches in reducing the classification error, easing catastrophic forgetting, and encouraging evenly balanced accuracy over different classes.

並列關鍵字

Incremental Learning ； Image Classification ； Cosine Normalization ； Knowledge Distillation ； Feature Graph Regularization

參考文獻

[1] Yi-Hsin Chen, Wen-Hsiao Peng, and Ming-Feng Chang. “Incremental Learning with Rectified Feature-Graph Preservation”. In: Master’s Thesis, National Chiao Tung

University (2020).

[2] Francisco M Castro et al. “End-to-end incremental learning”. In: European Conference on Computer Vision (ECCV). 2018.

[3] Saihui Hou et al. “Learning a Unified Classifier Incrementally via Rebalancing”. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.

[4] Lufan Li et al. “An incremental face recognition system based on deep learning”.