基於自動情境標註之圖像檢索工具發展與數位人文應用研究

Developing an Image Retrieval Tool based on Automatic Context Annotation for Digital Humanities Research

趙映翔(Zhao Ying-Xiang)

指導教授 ： 陳志銘

國立政治大學/文學院/圖書資訊與檔案學研究所/碩士(2021年)

https://doi.org/10.6814/NCCU202101403

摘要

「圖像檢索」在資訊蓬勃發展的現代，已經成為數位人文研究的重要方式之一。而影響傳統「基於文本的圖像檢索工具(Text-Based Image Retrieval, TBIR)」之圖像檢索效能的主要問題，為人工所注入代表圖像後設資料(metadata)與使用者所下檢索詞之間的語意鴻溝(semantic gap)。隨著電腦視覺技術快速發展而衍伸出的自動圖像標註(automatic image annotation)，由機器為其自動添加後設資料以降低的語意鴻溝。然而自動圖像標註的物件標註僅能找到具有該物件特徵，對於使用者的圖像檢索及圖像理解的幫助有限，進而促成本研究探索「自動情境標註」為減少圖像情境與人之間的語意鴻溝，並發展出得以有效輔助人文學者進行圖像檢索及圖像解讀之數位人文工具。
因此，本研究發展出「基於自動情境標註之圖像檢索工具(Image Retrieval Tool Based on Automatic Context Annotation, IRT-ACA)」。該系統的核心技術採用Mask R-CNN、TF-IDF及SVM，主要目的為圖像中的實體物件識別，以及抽象的情境識別，並將所得之數據以標籤化形式提供使用者用於圖像檢索與瀏覽，讓使用者得以可以快速萃取數位圖像中的實體物件以及抽象情境之訊息。進而促進人文學者更有效率地解讀圖像情境。
為驗證本研究發展之IRT-ACA是否有助於人文學者進行圖像解讀，本研究採用實驗研究法之對抗平衡設計，將實驗對象分為兩組，根據不同的系統使用順序來依次操作「IRT-ACA」與「基於文本的圖像檢索工具(Text-Based Image Retrieval, TBIR)」來完成檢索任務學習單。並透過行為歷程記錄來完整記錄實驗對象的系統操作行為、科技接受度問卷來反映實驗對象對於系統的實際感受，以及半結構式訪談來瞭解實驗對象的想法與建議，透過多種方法進行交互驗證，以瞭解本研究發展之IRT-ACA與TBIR在自動情境標註之準確度、解讀圖像情境之成效以及科技接受度上的差異。
研究結果發現：第一，IRT-ACA的自動情境標註準確度已足以有效輔助使用者解讀圖像情境；第二，使用TBIR與IRT-ACA在解讀圖像情境之成效上達顯著差異，並且IRT-ACA顯著優於TBIR；第三，使用TBIR與IRT-ACA在整體科技接受度上達顯著差異，並且IRT-ACA顯著優於TBIR，但其中的系統易用性未達顯著差異。從訪談分析中顯示，實驗對象對於兩個系統的操作難意度及使用流暢性上均感到滿意，因此給予系統易用性分數差異不大；第四，IRT-ACA的標籤型檢索比起自由下達檢索詞的檢索更能促進實驗對象的檢索意願；第五，使用IRT-ACA高分組使用者之檢索行為更充分使用到所有檢索功能；第六，IRT-ACA使用者之查看圖像至筆記紀錄之轉移率高於TBIR。

關鍵字

數位人文 ； 圖像檢索 ； 語意鴻溝 ； 物件標註 ； 情境標註 ； 深度學習 ； 自動圖像標註 ； Mask R-CNN ； TF-IDF ； SVM ； 行為分析

並列摘要

Image retrieval has become one of the significant approaches in digital humanities research in the digital age. The main problem affecting the performance of Text-Based Image Retrieval (TBIR) is the semantic gap between the manually determined metadata for images and the users’ search terms or keywords. With the rapid development of computer vision technology in recent years, automatic image annotation developed by machine learning schemes can reduce the semantic gap between humans through automatically adding metadata based on the identified image objects’ tags. However, the object tags determined by automatic image annotation can only find the characteristics of image objects, which is of little help to users’ image retrieval and image comprehension because they are still too low level from human’s perspectives. It prompted this research to develop automatic context annotation as a digital humanities tool that can effectively assist humanities scholars in image context interpretation by reducing the semantic gap between the subject of the image context and humans.
Therefore, this research developed an Image Retrieval Tool Based on Automatic Context Annotation (IRT-ACA). The core technology of the tool is Mask R-CNN, TF-IDF, and SVM, which aims to identify physical objects and abstract contexts hidden in images and provide users with more rich metadata in the form of object and contextual tags for image retrieval and browsing, so that users can quickly extract needed information from images, thus facilitating the more efficient interpretation of image contexts by humanists.
To verify whether IRT-ACA developed in this research is beneficial to humanities scholars in image interpretation, this research utilized a counterbalanced design of the experimental research method to examine the research questions. Users were divided into two groups and operated the IRT-ACA and the TBIR tool alternately to complete the two designed image retrieval tasks. Besides, the behavioral history recorder was used to record the system operation behavior of the users using the IRT-ACA and the TBIR completely. The technology acceptance questionnaire was used to reflect the actual feelings and perceptron of the experimental subjects towards the two systems. The semi-structured interview was used to understand the thoughts, ideas, and suggestions of the users who alternately used the two image retrieval systems.
The research results are summarized as follows. First, the accuracy of the automatic context annotation of IRT-ACA was sufficient to interpret the image context effectively. Second, there was a significant difference in the effectiveness of interpreting the image context between TBIR and IRT-ACA as well as IRT-ACA is significantly superior to TBIR. Third, there was a significant difference in the overall technology acceptance and perceived usefulness between TBIR and IRT-ACA as well as IRT-ACA is significantly superior to TBIR, but there was no significant difference in the perceived ease of use between TBIR and IRT-ACA. The analysis results show that users were satisfied with the ease of operation and the smoothness of using the two systems, so the difference in the scores of perceived ease of use was not significant. Fourth, the labeled retrieval provided by the IRT-ACA promoted the research subjects’ willingness to retrieve more than the free retrieval of using keywords. Fifth, the retrieval behaviors of the IRT-ACA users with high image interpretation performance made full use of all retrieval functions, including contextual tags’ search, object tags’ search, full-text search, and title research. Sixth, the transfer rate from viewing images to take notes of the users who used IRT-ACA is higher than that of users who used TBIR.

並列關鍵字

Digital humanities ； Image retrieval ； Semantic gap ； Object annotation ； Contextual annotation ； Deep learning ； Automatic image annotation ； Mask R-CNN ； TF-IDF ； SVM ； Behavioral analysis

參考文獻

陳勇汀（2017）。行為順序檢定：滯後序列分析/ Behavior Analysis: Lag Sequential Analysis。檢自https://pulipulichen.github.io/HTML-Lag-Sequential-Analysis/

項潔、陳麗華（2014）。數位人文－學科對話與融合的新領域。數位人文研究與技藝（頁9-23）

Agosti, M., Ferro, N., Orio, N., & Ponchia, C. (2014). CULTURA outcomes for improving the user’s engagement with cultural heritage collections. Procedia Computer Science, 38, 34-39. doi:10.1016/j.procs.2014.10.007

Beaudoin, J. E. (2014). A framework of image use among archaeologists, architects, art historians and artists. Journal of Documentation, 70(1), 119-147. doi:10.1108/JD-12-2012-0157

Beaudoin, J. E., & Brady, J. E. (2011). Finding visual information: a study of image resources used by archaeologists, architects, art historians, and artists. Art Documentation: Journal of the Art Libraries Society of North America, 30(2), 24-36. doi:10.1086/adx.30.2.41244062