透過序列到序列模型翻譯多模態情感

Translating Multimodal Emotion through Sequence-to-sequence Model

王廉花(Devina Ekawati)

指導教授 ： 許永真

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2019年)

https://doi.org/10.6342/NTU201902572

摘要

無資料

關鍵字

Emotion Recognition ； Emotion Translation ； Sequence-to-sequence ； Attention Mechanism ； Residual Connection

並列摘要

Emotion plays a big role in our daily life. When we try to perceive emotion, we do not only rely on one modality, but rely on several modalities. Psychology studies show that our human sensories perceives several signals from our environment and translate them to codes that are similar across people.

In our work, we formulate the emotion recognition as emotion translation task using Sequence-to-sequence (Seq2seq) models which are widely used in neural machine translation task. Additionally, we add attention mechanism as this mechanism can help the model to remember long sequences. Motivated by Google Neural Machine Translation (GNMT), we also try to add residual connection to resolve the decreasing performance when the models have several stacks of hidden layers.

We use CMU-MOSEI dataset to train and evaluate our models. Experiment shows that our proposed Seq2seq architecture outperforms the baseline model on emotion translation task. Moreover, the models that use several modalities achieve better performance than the models that only use one modality. This observation proves that multimodal representation escalates the performance of emotion translation or emotion recognition.

並列關鍵字

Emotion Recognition ； Emotion Translation ； Sequence-to-sequence ； Attention Mechanism ； Residual Connection

參考文獻

[1] D. Acharya, Z. Huang, D. Pani Paudel, and L. Van Gool. Covariance pooling for facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 367–374, 2018.

[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.

[3] T. Baltrusaitis, C. Ahuja, and L. Morency. Multimodal machine learning: A survey and taxonomy. CoRR, abs/1705.09406, 2017.

[4] C. Baziotis, N. Athanasiou, A. Chronopoulou, A. Kolovou, G. Paraskevopoulos, N. Ellinas, S. Narayanan, and A. Potamianos. Ntua-slp at semeval-2018 task 1: Predicting affective content in tweets with deep attentive rnns and transfer learning. arXiv preprint arXiv:1804.06658, 2018.

[5] S. Buechel and U. Hahn. Emotion analysis as a regression problem - dimensional models and their implications on emotion representation and metrical evaluation. In Proceedings of the Twenty-second European Conference on Artificial Intelligence, pages 1114–1122. IOS Press, 2016.