引入語意知識輔助影像生活紀錄之辨識與檢索

Incorporating Semantic Knowledge for Image-Based Lifelog Recognition and Retrieval

傅敏桓(Min-Huan Fu)

指導教授 ： 陳信希

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2019年)

https://doi.org/10.6342/NTU201902899

摘要

近年來，穿戴式科技的進步帶動了生活記錄的風潮。這些由穿戴式裝置收集的資料呈現個人生活經驗的不同面向，提供生活型態分析和記憶回復一個新的資料媒介。然而，建置一個方便組織與存取影像生活紀錄的系統並不容易，主要的原因在於影像生活紀錄和語意事件描述之間所呈現的語意鴻溝。在本篇論文中，我們嘗試引入外部語意知識減少語意鴻溝。
我們考慮兩種生活紀錄的存取形式：(1) 自動偵測和辨識生活記錄者之日常活動，以及 (2) 檢索生活記錄者生活經驗中出現過的特定事件。在日常活動辨識的部分，我們結合從外部資源衍伸之語意知識來增強監督式機器學習模型的訓練資料；而在生活紀錄檢索的部份，我們以預訓練的詞嵌入向量加強語意事件描述和影像視覺概念之間的語意連結。我們提出的兩個方法在影像前處理及視覺概念標記的部分都使用同樣的架構；實驗結果顯示我們提出的結合語意知識的方法，皆可以提升生活紀錄檢視系統的效能。

關鍵字

生活紀錄 ； 生活事件辨識 ； 影像生活紀錄檢索 ； 語意知識 ； 詞嵌入向量 ； NTCIR生活紀錄資料集

並列摘要

Recently, the advance in wearable technology has made lifelogging more feasible and popular. Visual lifelogs collected by wearable cameras capture different aspects of personal life experiences compared with textual records like diaries, providing new data sources for lifestyle understanding and memory recall. However, building a system for accessing and organizing visual lifelogs effectively is a challenging task due to the semantic gap between visual data and semantic descriptions of life events. In this thesis, we aim to introduce semantic knowledge for bridging such a semantic gap.
We deal with two tasks of semantic lifelog access: (1) to automatically detect and recognize daily activities for lifestyle understanding, and (2) to retrieve specific events in a lifelogger's life for memory recall support. For life event recognition, we incorporate the knowledge derived from external resources to enrich the training data for supervised learning. For lifelog retrieval, we exploit pre-trained word embeddings to enhance the semantic relatedness between event topics and visual concepts present in the visual lifelogs. The approaches proposed in these two tasks share the same image preprocessing and indexing framework, and the experimental results show that incorporating external semantic knowledge is beneficial for improving the performance of lifelog systems.

並列關鍵字

Lifelog ； Lifelog Activity Recognition ； Visual Lifelog Retrieval ； Semantic Knowledge ； Word Embedding ； NTCIR Lifelog Dataset

參考文獻

Fatma Ben Abdallah, Ghada Feki, Mohamed Ezzarka, Anis Ben Ammar, and Chokri Ben Amar. 2018. Regim Lab Team at ImageCLEF Lifelog Moment Retrieval Task 2018. In CLEF (Working Notes).

Khalid EL Asnaoui, Aksasse Hamid, Aksasse Brahim, and Ouanan Mohammed. 2017. A survey of activity recognition in egocentric lifelogging datasets. In 2017 International Conference on Wireless Technologies, Embedded and Intelligent Systems (WITS), pages 1–8. IEEE.

Raghav Bansal, Gaurav Raj, and Tanupriya Choudhury. 2016. Blur image detection using Laplacian operator and Open-CV. In 2016 International Conference System Modeling & Advancement in Research Trends (SMART), pages 63–67. IEEE.

Marc Bolanos, Mariella Dimiccoli, and Petia Radeva. 2016. Toward storytelling from visual lifelogging: An overview. IEEE Transactions on Human-Machine Systems, 47(1):77–90.

Gary Bradski. 2000. The opencv library. Dr Dobb’s J. Software Tools, 25:120–125.