基於時序多模態資料用於疼痛强度辨識的混合深度神經網路

Hybrid Deep Neural Networks for Pain Intensity Estimation via Temporal Multimodalities

周靈風(Lingfeng Zhou)

指導教授 ： 傅立成

國立臺灣大學/電機資訊學院/電機工程學研究所/碩士(2021年)

https://doi.org/10.6342/NTU202100747

摘要

疼痛表情是衡量急診患者當下情況的一項重要指標。準確且有效的辨識疼痛強度，使醫護人員能夠以適當且準確得對待不同緊急程度的病患。自動疼痛識別系統可以降低成本及減輕人力短缺的問題，隨著深度神經網路發展，這些基於深度學習與電腦視覺的方法提供了自動化疼痛判斷的潛在解決方案在傳統的識別方法中傾向於基於單個幀來提取特徵，因爲沒有考慮整段影片中幀與幀之間的關聯，這可能導致不夠準確的預測。此外也與其他使用傳統常規的循環神經網絡的疼痛強度識別方法不同，在此篇論文中，我們建議不可以在訓練神經網路的過程中僅僅使用從臉部提取的特徵，由於不同程度的疼痛之間差距比較細微，因此更需要設計一個能夠專注於面部細節的神經網路來辨識各種強度不一的疼痛。我們的研究首先提取基於主動外觀模型演算法搜尋到的患者臉部的關鍵點，然後將預處理過的面部依次輸入到定制的帶有專注功能的殘差捲積神經網絡(CNN)中，將每個臉部的動作單元(Action Units)輸入另一個密集連接的捲積網路學習對應的特徵。我們將從捲積網路中提取到的特徵和先前的臉部關鍵點相連結作爲一個組合，並將整個一組時序影片對應的這些組合連接到Transformer序列神經網路，以預測不同等級的疼痛的強度水準。我們提供的方法在UNBC-McMaster肩膀疼痛資料數據集上達到了86.5% 準確率的表現，優於目前其他的同領域的相關方法。本文首先提出了一種結合關鍵點和動作單元與原始圖像特徵的作爲輸入序列，並使用兩個不同的特製的捲積神經網路及一個序列神經網路用於不同疼痛強度判斷的方法。

關鍵字

疼痛辨識 ； 深度學習 ； 注意力機制 ； 多模態

並列摘要

Pain expression is an indicator for patients’ current condition. Accurate and effective recognition of pain intensity is significant for medical personnel to treat and care patients properly and carefully. The viewing cost and shortage of human labors call for the need of automatic pain recognition. With the development of the deep neural networks, these techniques offer great potential for automatic pain intensity recognition. Unlike other state-of-the-art pain intensity recognition methods using conventional and regular recurrent neural network, in this thesis, we suggest that not only key landmarks of the face can be used in training but also the network should focus more on facial details to enhance the performance. This study first uses Point Distribution Model to extract key landmarks of the patient’s face, and then feed the sequence of preprocessed facial images into a customized Convolutional Neural Network (CNN) with attention mechanism to extract features and related action units to another densely connected network to learn features. Concatenation of the features from neural networks and landmarks are linked to the Transformer network to predict the pain intensity levels. Our proposed method is trained and tested on the UNBC-McMaster Shoulder Pain Expression Archive Database and reaches promising performance. This thesis first proposes a methodology in combining landmarks and action units with raw images and using Transformer network for pain intensity prediction in facial images sequence.

並列關鍵字

Pain intensity prediction ； Deep Learning ； Attention Mechanism ； Multimodalities

參考文獻

[1] N. Hoot and D. Aronsky, "Systematic review of emergency department crowding: causes, effects, and solutions," Annals of emergency medicine, vol. 52 2, pp. 126-36, 2008.

[2] A. B. Nassif, I. Shahin, I. B. Attili, M. Azzeh, and K. Shaalan, "Speech Recognition Using Deep Neural Networks: A Systematic Review," IEEE Access, vol. 7, pp. 19143-19165, 2019.

[3] M. Leening, M. M. Vedder, J. Witteman, M. Pencina, and E. Steyerberg, "Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide," Annals of internal medicine, vol. 160 2, pp. 122-31, 2014.

[4] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, "A Semi-automatic Methodology for Facial Landmark Annotation," 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 896-903, 2013.

[5] L.-C. Fu , "以人工智慧改善急診病人流動及解決擁塞之全面性策略 - 快速精確的電子化驗傷," ed. 臺北國際醫療科技展: 台大醫院, 2020.