藉由三維可變形人臉模型及UV貼圖將二維StyleGAN2轉變為三維人臉生成器

Converting 2D StyleGAN2 as a 3D Face Generator via 3DMM and UV Map

鐘偉傑(Wei-Chieh Chung)

指導教授 ： 莊永裕

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2021年)

https://doi.org/10.6342/NTU202103898

摘要

近年來，StyleGAN2因為有著優良的生成品質以及和風格高度相關的潛在空間而備受關注。許多文獻已經使用了不同的方法來操控三維屬性，比如轉動姿勢。這些文獻都一再證明，StyleGAN2的潛在空間中蘊含著豐富的三維資訊，也揭露了StyleGAN2擁有著三維生成的潛力。在這篇論文中，我們將StyleGAN2轉換為三維人臉生成器，透過將三維的紋理及形狀用二維圖片表示，我們可以利用原本的StyleGAN2生成出高品質的三維紋理及形狀。我們也在這篇論文中提出了一個架構將StyleGAN2和三維可變形人臉模型結合來實現三維人臉生成。我們訓練了一個模組能從StyleGAN2的潛在代碼中預測出三維參數來作為我們初步的結果。以這個初步結果為基礎，我們就能利用StyleGAN2生成高品質的UV貼圖來作為我們的三維紋理及形狀。整個訓練過程中沒有用到任何真實世界的三維資料，全都是使用StyleGAN2所生成的圖片作為訓練資料。在後續的實驗中，我們會展現出我們的方法能夠生成出比前人更好的結果。

關鍵字

深度學習 ； 三維人臉生成

並列摘要

Recently, StyleGAN2 has gained a lot of attention because of its high-quality generation and style-disentangled latent space. Many literatures have explored different ways to manipulate the 3D attributes such as pose. It reminds us that there is rich 3D information encoded in the latent space of StyleGAN2, and thus reveals the potential of StyleGAN2 to 3D generation. In this paper, we convert StyleGAN2 as a 3D face generator with the texture and shape representation expressed in 2D images, which could be generated by StyleGAN2 with high quality. This thesis proposes a framework to combine StyleGAN2 and 3D Morphable Model (3DMM) together to implement 3D face generation. We train a module to predict 3D parameters from the latent code of StyleGAN2 as the coarse result. With predicted 3DMM as backbone, we convert StyleGAN2 to generate high-quality UV maps as the texture and shape representation. The whole training process is achieved with solely stylegan2-generated images and without any real 3D data. The following visual and quantitative experiments demonstrate that our method can generate better results compared with previous works on 3D controllable GANs.

並列關鍵字

Deep Learning ； 3D Face Synthesis

參考文獻

A. Bas, P. Huber, W. A. Smith, M. Awais, and J. Kittler. 3d morphable models as spatial transformer networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 904–912, 2017.

V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.

C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.

J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.

Y. Deng, J. Yang, S. Xu, D. Chen, Y. Jia, and X. Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.