在神經輻射場中實現更全面的三維人像編輯

Elevate 3D Editing of Neural Portrait to a More Comprehensive Level

劉子傑(Tzu-Chieh Liu)

指導教授 ： 簡韶逸

國立臺灣大學/電機資訊學院/電子工程學研究所/碩士(2024年)

https://doi.org/10.6342/NTU202400881

摘要

近年來，隨著神經渲染技術的進步，三維生成對抗網路（3D-GANs）在高保真多視角一致性肖像生成方面取得了重大進展。然而，現有方法仍難以對三維肖像的幾何和紋理進行明確與細緻的操作。這一限制帶來了顯著的障礙，因人們通常希望對其虛擬化身進行更廣泛的編輯，例如配飾的穿戴，以在虛擬世界中創造迷人與時尚的替身。因此，將肖像與配飾（或廣義至各個肖像外的部件）分離，並具有獨立控制各個部件的能力就顯得極為重要。
在本篇論文中，我們提出了多視角肖像配件生成網路（FOVAD），以生成具有各自分離屬性的三維肖像配飾。具體地說，為了將現有三維模型擴展至更全面的編輯能力，我們將整個場景的建模分解為兩個部分，僅肖像和僅配件，並將它們重新組合以進行重建。我們進一步引進了「由草圖生成配件」模組（Sketch2Accessory），它根據用戶隨意的筆跡或塗鴉生成對應的配件，並擁有自由選擇配件紋理的能力。藉由FOVAD網路，多視角肖像可以達到比起過去方法在此限制條件下，更好的操縱自由度與更佳的生成結果。從全局風格化、局部編輯到用戶指定的配件配戴，可能性是無窮的。

關鍵字

臉部編輯 ； 影像合成 ； 三維生成對抗網路 ； 神經輻射場

並列摘要

Recently, with the immense progress of neural rendering techniques, 3D-aware Generative Adversarial Networks (GANs) leap forward toward high-fidelity multi-view consistent portrait synthesis. However, existing approaches still struggle to provide fine-grained control over both geometry and texture of 3D portraits. This restriction presents a noteworthy obstacle given that people typically fancy diversely accessorizing their virtual avatars and creating fashion-forward looks. As such, decoupling portraits and accessories (controls over individual components) with different shapes and appearances (controls over geometry and texture space) is highly in demand.
In this work, we propose FOVAD (abbreviation for Free-View Portrait Adornment) generative network and bridge the aforementioned valley to generate a diverse set of accessories on 3D portraits with respective disentangled attributes. More specifically, to expand the existing 3D models to a more comprehensive level of editability, we decompose the modeling of the entire scene into two distinct parts: portrait only and decorative attributes, and re-compose them for reconstruction, while only employing a single scene-shared representation. Detaching accessories from portraits facilitates effortless “Sketch2Accessory”, which creates accessories based on the user’s unrestricted accessory sketch or doodle. Armed with FOVAD framework, free-view portraits can be customized in ways that were once impossible with only monocular, readily collectible training data. From global or local editing to multiple user-specified accessories wearing, the possibilities are endless.

並列關鍵字

Face editing ； Image synthesis ； semantic-mask-based interfaces ； neural radiance fields ； neural rendering ； 3D GAN

參考文獻

Personal avatar for vr-chat. [Online]. Available: https://www.fiverr.com/ holomiru/create-your-personal-avatar-for-vr-chat-from-a-photo

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communica- tions of the ACM, vol. 63, no. 11, pp. 139–144, 2020.

T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196, 2017.

X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1501–1510.

T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401–4410.