利用視覺Transformer之多標籤深度視覺語義嵌入模型

Multi-Label Deep Visual-Semantic Embedding with Visual Transformer

來毓庭(Lai Yu-Ting)

指導教授 ： 葉梅珍

國立臺灣師範大學/理學院/資訊工程學系/碩士(2021年)

https://doi.org/10.6345/NTNU202101778

摘要

多標籤影像分類是一項具挑戰性的工作，目標是同時找出不同大小的物件並且辨識正確的標籤。然而，常見的做法是使用整張影像抽取特徵，較小物體的資訊可能會因此被稀釋，或是成為雜訊，造成辨識困難。在先前的研究裡顯示，使用關注機制和標籤關係能各自增進特徵擷取和共生關係，以取得更強健的資訊，幫助多標籤分類任務。
在本工作中，我們使用Transformer之架構，將視覺區域特徵關注至全域特徵，同時考慮標籤之間的共生關係，最後將加權後之新特徵產生出一動態的語義分類器，在語義空間內分類得出預測標籤。在實驗中，顯示我們的模型可達到很好的成效。

關鍵字

多標籤分類 ； 視覺語義嵌入模型 ； 關注機制

並列摘要

Multi-label classification is a challenge task since we must identify many kinds of objects in different scales. While using global features of an image may discard small object information, many researches have shown that an attention mechanism improves feature extraction and that label relations reveal label co-occurrence, both of which benefit a multi-label classification task.
In this work, we extract attended features from one image by Transformer and simultaneously consider labels’ co-occurrence. Then, we use the attended features to generate a classifier applied on the semantic space to predict the labels. Experiments validate the proposed method.

並列關鍵字

multi-label classification ； visual-semantic embedding ； Transformer

參考文獻

[1]M. Yeh, Y. Li, “Multilabel Deep Visual-Semantic Embedding”, In TPAMI, 2019

[2]Z. M. Chen, X. S. Wei, P. Wang and Y. Guo, “Multi-label image recognition with graph convolutional networks”, In CVPR, 2019.

[3]V. O. Yazici, A. Gonzalez-Garcia, A. Ramisa, B. Twardowski, and J. V. D. Weijer, “Orderless recurrent models for multi-label classification”, In CVPR, 2020.

[4]J. Zhang, Q. Wu, C. Shen, J. Zhang, and J. Lu, “Multilabel image classification with regional latent semantic dependencies”, In IEEE Transactions on Multimedia 20(10): 2801-2813, 2018.

[5]F. Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang, “Learning spatial regularization with image-level supervisions for multi-label image classification”, In CVPR, 2017.