在360世界中行走：從單張全景圖合成全景視差

Moving in a 360 World: Synthesizing Panoramic Parallaxes from a Single Panorama

許瀞予(Ching-Yu Hsu)

指導教授 ： 陳煥宗

國立清華大學/電機資訊學院/資訊工程學系所/碩士(2021年)

摘要

我們提出 Omnidirectional Neural Radiance Fields (OmniNeRF)，是第一 個可使用在全景圖上的新視角合成應用。近期在新視野合成領域中的作品主 要都關注在透視影像的合成，但透視影像會受限於有限的視野和需要在特定 條件下足夠數量的影像資訊。相比之下，只要給予一張 360 影像作為訓練資 料，OmniNeRF 就可以在未知的視野下生成全景圖。因此，我們提出一種資 料擴充的方法，將單張的全景圖從 3D 世界投影到在不同場景位置的 2D 平面 座標系。利用這樣的方式，我們可以用在固定相機位置往 360 度全方向可視 的像素去最佳化 Omnidirectional Neural Radiance Field，以估算在任意相機 地點所看到的場景。總結而言，我們提出的 OmniNeRF 產生全新視野的全景 圖時，可以輸出具說服力的影像，並且能夠產生雙眼視差的效果。我們在合 成與真實世界的資料集中驗證我們的結果 。

關鍵字

全景圖 ； 360影像 ； 新視野合成

並列摘要

We present Omnidirectional Neural Radiance Fields (OmniNeRF), the first method to the application of parallax-enabled novel panoramic view synthesis. Recent works for novel view synthesis focus on perspective images with limited field-of-view and require sufficient pictures captured in a specific condition. Conversely, OmniNeRF can generate panorama images for unknown viewpoints given a single equirectangular image as training data. To this end, we propose to augment the single RGB-D panorama by projecting back and forth between a 3D world and different 2D panoramic coordinates at different virtual camera positions. By doing so, we are able to optimize an Omnidirectional Neural Radiance Field with visible pixels collecting from omnidirec tional viewing angles at a fixed center for the estimation of new viewing angles from varying camera positions. As a result, the proposed OmniNeRF achieves convincing renderings of novel panoramic views that exhibit the parallax effect. We showcase the effectiveness of each of our proposals on both synthetic and real-world datasets.

並列關鍵字

panorama ； 360 ； novel ； view ； synthesis

參考文獻

[1] D.Anguelov,C.Dulong,D.Filip,C.Früh,S.Lafon,R.Lyon,A.S.Ogale,L.Vincent, and J. Weaver. Google street view: Capturing the world at street level. Computer, 43(6):32–38, 2010.

[2] A. Badki, O. Gallo, J. Kautz, and P. Sen. Meshlet priors for 3d mesh reconstruction. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2846–2855. IEEE, 2020.

[3] A. X. Chang, A. Dai, T. A. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pages 667–676. IEEE Computer Society, 2017.

[4] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry. Atlasnet: A papier- mâché approach to learning 3d surface generation. CoRR, abs/1802.05384, 2018.

[5] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cam- bridge University Press, 2004.