自由視角生成引擎演算法與硬體架構設計

Algorithm and Architecture Design of Free-viewpoint View Synthesis Engine

魏谷竹(Ku-Chu Wei)

指導教授 ： 簡韶逸

國立臺灣大學/電機資訊學院/電子工程學研究所/碩士(2013年)

https://doi.org/10.6342/NTU.2013.03101

摘要

自由視角電視被認為視下一代電視發展的趨勢，相較於傳統的三維電視，自由視角電視能提供無限多個可觀賞角度，讓使用者的觀賞角度有著完全的自由。然而受限於影片在拍攝時只能透過有限多個相機得到，意即要在顯示端提供直接顯示無限多個視角是不可行的。自由視角生成技術便是解決此問題的重要關鍵：將輸入的影片作任意觀賞角度的生成，有限視角變成無限視角，而此論文便提出一自由視角電視中自由視角生成引擎的設計，從軟體層面至硬體層面。
在文獻中，自由視角生成技術主要是由深度影像繪圖法與破洞填補構成，然而要將此技術使用在實際電視的系統上，效率乃是最重要的議題，需要考量影像品質以及運算複雜度之間的權衡。因此，在此論文提出一高效率的深度影像繪圖演算法：由光線投影與推拉演算法構成，只需要一個階段就能完成，但影像品質與目前最效果最好的流程(兩階段的向後捲繞) 十分類似。其中光線投影可以視為在繪圖的過程中套用三維濾波器，而推拉演算法可以進一步解決因取樣所造成的問題。然因輸入的深度圖並非完美(時間上並不保證不一致)，導致生成的新視角影片會出現時間閃爍的現象，為了消除影片中閃爍的現象，我們另將一時間濾波器加入到此自由視角生成引擎，生成結果也因此獲得相當的改善。
因為我們在演算法層面就考慮適合硬體實作的演算法：相較於一般向後捲繞的流程需要兩個階段，我們將提出的深度影像會圖法只要一個階段。單純一階段管現化的演算法，對自由視角生成引擎的硬體架構設計便有相當大的助益。此外，為了能讓演算法符合硬體架構，我們改變了既有的破洞填補演算法：使用加權平均的方式，權重為位置
與深度資訊的結合，此方法的好是乃是能免除傳統破洞填補演算法所需的不定遞迴過程，確保在固定的次數內將破洞填補完成。硬體的架構設計上，最主要的挑戰為大量的頻寬需求以及深度繪圖法的對外部記憶體的不規則讀取。為解決此問題，我們將畫面分成區塊各自分開運算；同時，我們在深度影像繪圖法前加上一個額外的模組，預捲繞。深度影像繪圖法便能利用預捲繞得到的資訊來預測來源影片中的哪些區塊是需要讀取的，避免讀取不必要的區塊，減少大量的頻寬的需求，也因為每次都是以區塊為單位讀進晶片中，外部記憶體隨機存取的問題也能同時得到解決。藉由預捲繞與將畫面分成小區塊，我們提出的硬體架構便能在外部頻寬、內部記憶體大小、與運算資源中取得一個好的平衡點。

關鍵字

自由視角生成 ； 深度影像繪圖法 ； 自由視角電視

並列摘要

Free-viewpoint TV (FTV) is considered to be the next-generation TV. Compared to conventional 3D TV, free-viewpoint TV can provide infinite viewpoints, a fully 6DoF transformation. However, only limited views can be captured by real-world camera, and free-viewpoint view synthesis is the key technology for such application: synthesize the video with arbitrary viewpoint. The target of this thesis is to propose a free-viewpoint view synthesis engine for free-viewpoint TV. The design space is from algorithm level to architecture level.
In the literature, free-viewpoint view synthesis is composed of depth-image-based rendering followed by hole-filling. To realize view synthesis in practical systems, the efficiency of view synthesis must be considered to
achieve a good balance between the image quality and the computational complexity. We propose an efficient DIBR implementation that, when compared to state-of-the-art backward warping, requires only half of the computation complexity with comparable quality. Specifically, the proposed scheme uses ray casting and pull-push processing to render in one pass, which can be regarded as applying 3D filters in the depth-image-based rendering. Due to the imperfect input depth maps, noticeable temporal fluctuation can be perceived when playing back the synthesized video. To cope with the problem, a robust temporal filtering technique is also integrated into the view synthesis scheme to eliminate such a temporal fluctuation.
With the proposed one-pass depth-image-based rendering flow, the following hardware architecture design can be benefited, since the one-pass flow
is suitable for pipeline design. Moreover, a simple depth-aware convolution-based inpainting, where the weight is the combination of the distance and the depth value, is employed to achieve smoother data flow for hardware design with lower cost. The main design challenge of free-viewpoint view synthesis engine is the huge bandwidth requirement and the highly random access to
DRAM in depth-image-based rendering. In our design, block-based pipeline is utilized to reduce the huge bandwidth with small on-chip SRAM. To reduce the random access problem, an extra module, pre-warping, is added to pre-fetch the blocks that DIBR requires only. With the combination of pre-warping and carefully considered block-based design, the free-viewpoint
view synthesis engine achieves a nice trade-off among bandwidth, on-chip memory, and computational resources.

並列關鍵字

Free-viewpoint view synthesis ； DIBR ； Free-viewpoint TV

參考文獻

Conference on, July 2011.

(CVPR), 2001.

image based rendering with edge dependent depth filter and interpolation. In

nitnslMultimedia and Expo (ICME), IEEE International Conference on, July 2005.

[4] I. Daribo and H. Saito. A novel inpainting-based layered depth video for 3DTV.