物體關係注意力網絡實現圖像段落生成

Object Relation Attention for Image Paragraph Captioning

楊力權(Li-Chuan Yang)

指導教授 ： 許永真

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2020年)

https://doi.org/10.6342/NTU202001536

摘要

圖片段落描述(Image Paragraph Captioning)的目標是對一張圖片產生複數個句子。這個題目衍生自圖片描述生成(Image Captioning)，他們的不同之處在於生成單一或複數個句子以描述一張圖片。因此圖片段落生成是一個更具挑戰性的題目，因為比起單一的描述句子，一個好的段落必須更長、富含更多資訊、還有通順的前後文語意。這些既複雜又困難的挑戰使圖片段落描述還是一個未飽和的題目，而本文提出的方法是為了改善段落生成的關係與品質。在對圖片作物件偵測(Object Detection)後，我們將偵測方框有重疊的物件用兩個不同的方向連接形成多個配對，將物件與配對送入模型得到強化後的物件特徵，再將經強化的物件特徵送入語言模型以生成段落，希望在強化的階段能學習物件與物件的關係。我們的實驗顯示本文的方法能透過訓練抽出及強化重要的物件關係，並幫助語言模型生成更好的段落品質。

關鍵字

物體 ； 關係 ； 專注力 ； 影像 ； 文本生成 ； 深度學習

並列摘要

Image paragraph captioning aims to automatically generate a paragraph from a given image. It is an extension of image captioning in terms of generating multiple sentences instead of a single one, and it is more challenging because paragraphs are longer, more informative, and more linguistically complicated. Because a paragraph consists of several sentences, an effective image paragraph captioning method should generate consistent sentences rather than contradictory ones. It is still an open question how to achieve this goal, and for it we propose a method to incorporate objects' spatial coherence into a language-generating model. For every two overlapping objects, the proposed method concatenates their raw visual features to create two directional pair features and learns weights optimizing those pair features as relation-aware object features for a language-generating model. Experimental results show that the proposed network extracts effective object features for image paragraph captioning and achieves promising performance against existing methods.

並列關鍵字

object ； relation ； attention ； image ； captioning ； deep learning

參考文獻

[1]  P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In ECCV. Springer, 2016.

[2]  P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In CVPR, 2018.

[3]  S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Sum- marization, 2005.

[4]  J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.

[5]  M. Chatterjee and A. G. Schwing. Diverse and coherent paragraph generation from images. In ECCV, 2018.