同步生成影片和估計遠程光體積變化描計圖的多任務學習

Multi-Task Learning for Simultaneous Video Generation and Remote Photoplethysmography Estimation

鄒昀芸(Yun-Yun Tsou)

指導教授 ： 許秋婷

國立清華大學/電機資訊學院/資訊工程學系所/碩士(2020年)

摘要

遠程光體積變化描計圖（rPPG）是一種非接觸式方法，用於臉部影片計算生理信號。如果沒有大量的監督數據集，那麼學習一個可靠的rPPG預估模型會變得非常具有挑戰性。因此，我們認為把數據集增大以讓模型學習的更好這件事對於計算rPPG信號至關重要。在本文中，我們提出了一種新穎的多任務學習方式，在學習rPPG估計模型的同時增加訓練數據集。我們設計了三個聯合學習網絡：(1) rPPG估計網絡：從臉部影片估計rPPG信號。 (2) 圖像到影片網絡：根據原始圖片和指定的rPPG信號生成影片。 (3) 影片到影片網絡：根據原始影片和指定的rPPG信號生成影片。我們測試在三個數據集：COHFACE，UBFC-RPPG和PURE上，其實驗結果表明我們的方法成功生成了與原始影片外表相似度極高但不同rPPG信號的影片，並且預測rPPG信號的效果大大優於現有方法。

關鍵字

計算遠程光體積變化描記圖 ； 影片生成 ； 多任務學習

並列摘要

Remote photoplethysmography (rPPG) is a contactless method for estimating physiological signals from facial videos. Without large supervised datasets, learning a robust rPPG estimation model is extremely challenging. Instead of merely focusing on model learning, we believe data augmentation may be of greater importance for this task. In this thesis, we propose a novel multi-task learning framework to simultaneously augment training data while learning the rPPG estimation model. We design three joint-learning networks: rPPG estimation network, Image-to-Video network, and Video-to-Video network, to estimate rPPG signals from face videos, to generate synthetic videos from a source image and a specified rPPG signal, and to generate synthetic videos from a source video and a specified rPPG signal, respectively. Experimental results on three benchmark datasets, COHFACE, UBFC, and PURE, show that our method successfully generates photo-realistic videos and significantly outperforms existing methods with a large margin.

並列關鍵字

Remote photoplethysmography estimation ； Video generation ； Multi-task learning

參考文獻

[1] Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in International Conference on Computer Vision (ICCV), 2019.

[2] X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585, Aug 2018.

[3] X. Li, I. Alikhani, J. Shi, T. Seppanen, J. Junttila, K. Majamaa-Voltti, M. Tulppo, and G. Zhao, “The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection,” in 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 242– 249, May 2018.

[4] Z. Yu, X. Li, and G. Zhao, “Recovering remote photoplethysmograph signal from facial videos using spatio-temporal convolutional networks,” CoRR, vol. abs/1905.02419, 2019.

[5] W.ChenandD.McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in The European Conference on Computer Vision (ECCV), pp. 356–373, Springer International Publishing, 2018.