開發計算與量化人類行為於個人文化維度與疼痛情緒之應用

Toward Development of Computational Modeling and Quantifying Behaviors in Culture Dimension and Pain-Affect

蔡福昇(Fu-Sheng Tsai)

指導教授 ： 李祈均

國立清華大學/電機資訊學院/電機工程學系所/博士(2021年)

摘要

人類行為是複雜的相互作用，其表達在本質上是多模態的。語音、語言、視覺、生理訊號等行為線索提供量測模擬人類行為的方法。以人為中心的應用提供了提取行為信息與量化成數位訊號的計算方法。在本論文中，我們發表行為信號處理 (BSP) 中的量化建模應用於理解文化維度、疼痛和情感。文化是在現實生活互動中支配人類思想和社會行為的社會建構，使用多模態聲學韻律和大腦連結自動評估文化權力距離。我們提出的社會條件增強網絡達到 96.2% 的二元分類準確率，分析揭示了重要的韻律特徵和大腦區域，證明了量化個人對權力狀態的作用。另一方面，疼痛為身體或精神損傷引起的情緒體驗和不愉快的感覺。我們提出了屬性增強多頭注意力網絡，其中包括情緒、個人特徵和臨床參數，透過病患的聲音特徵和臉部表情進行建模來測量疼痛與情緒表達。在病患自述的疼痛指數辨識中，我們在二元和三元分類中分別獲得了 70.1% 和 52.1%，並證明了情緒的正負性與自述疼痛指數呈高度負相關，且比直接觀察疼痛更相關於自述疼痛指數。此外，我們提出了一個與情緒狀態、臨床參數和個人特徵共同學習的架構，達到了 58.64%（輕度疼痛、年齡、性別和疼痛部位的組合）和 46.88%（輕度疼痛、性別和疼痛部位的組合）於三元正負性(Valence)和喚醒度(Arousal)分類任務中。視覺化分析顯示多頭注意力機制對患者的情感狀態有分佈式專注化的效應。此外，臨床結果的分析顯示屬性對於用藥與住院處置有單變量和多變量顯著性。

關鍵字

行為訊號處理 ； 疼痛 ； 情緒 ； 文化權力距離 ； 功能性磁振造影 ； 韻律學 ； 中心損失 ； 多頭注意力 ； 多任務學習

並列摘要

Human behaviors are the complex interplay, and the expression are inherently multimodal. Behavioral cues such as speech, language, visual, physiological signals offer the means to measure and model human behavior. Computational approaches bring the way to extract behavior informatics and quantify as digital signals for human-centered applications. In this dissertation, we describe applications in behavioral signal processing (BSP) for quantitatively modeling and understanding cultural dimension, pain, and emotion. Culture is the social construct that dictates human thoughts and social behaviors during real life interaction. The first application is to automatically assess cultural power distance using multimodal acoustic prosody and brain connectivity. Our proposed social condition-enhanced network achieves 96.2% binary recognition accuracy, and the analyses reveal the significant prosodic features and brain region, demonstrating the role in quantifying individual belief about power status. On the other hand, pain is defined as an emotional experience and unpleasant sensation that results from physical or mental damages. We propose attribute-enhanced multi-head attention network which includes emotion, personal traits, and clinical parameters for measuring the pain-related emotion expressions by modeling patient’s vocal characteristics and facial expressions. In the experiment of patient’s self-reported pain level recognition, we obtain 70.1% and 52.1 in binary and ternary classification, and demonstrates that rated valence state is highly negative to self-reported pain level and is more correlated to pain level than to rate directly on the pain intensity. In addition, we present a computational framework that jointly learned with emotional states, clinical parameters, and personal traits, and achieves 58.64% (for the combination of mild pain, age, gender, and the pain site) and 46.88% (for the combination of mild pain, gender, and the pain site) in ternary arousal and valence classification tasks. The analysis of visualization indicates the multi-head attention models patient’s affect states with distributed and diverse attentive effects. In addition, the analyses of clinical outcomes demonstrate that attributes have both univariate and multivariate significance for analgesic prescription and patient disposition.

並列關鍵字

Behavioral Signal Processing (BSP) ； Pain ； Emotion ； Culture power distance ； fMRI ； prosody ； center-loss embedding ； Multi-Head Attention ； Multi-Task Learning

參考文獻

[1] O. P. John, E. M. Donahue, and R. L. Kentle, "Big five inventory," Journal of Personality and Social Psychology, 1991.

[2] L. Y. Mano et al., "Exploiting IoT technologies for enhancing Health Smart Homes through patient identification and emotion recognition," Computer Communications, vol. 89, pp. 178-190, 2016.

[3] S. Narayanan and P. G. Georgiou, "Behavioral signal processing: Deriving human behavioral informatics from speech and language," Proceedings of the IEEE, vol. 101, no. 5, pp. 1203-1233, 2013.

[4] E. B. Tylor, Primitive culture: Researches into the development of mythology, philosophy, religion, art and custom. J. Murray, 1871.

[5] F. M. Keesing, Cultural anthropology. Mittal Publications, 1965.