適用於光場視點合成之視差圖合成演算法與硬體架構研究

Algorithm and Architecture Study on Disparity Map Synthesis for Light-Field Viewpoint Synthesis

程韋翰(Cheng, Wei Han)

指導教授 ： 黃朝宗

國立清華大學/電機資訊學院/電機工程學系/碩士(2016年)

摘要

光場相機的應用已經逐漸成為計算攝影領域的主流研究，自由視點合成更是其研究的重點之一。自由視點合成可以在光場相機所擷取的視點影像中任意產生欲觀看的視點影像。近年來，在depth-image-based rendering(DIBR)的自由視點合成相關研究中，在光場架構下需要與視點相同數量的disparity map來完成視點合成，如果這些disparity map皆由disparity search演算法或sensor來產生，無論是在成本與計算複雜度上都過於高昂。所以如何使用較少的disparity map來完成自由視點合成且達到良好的品質則顯得相當重要。
本論文探討的是，在光場架構中僅使用一張disparity map完成自由視點合成。我們研究發現在DIBR演算法中進行視點合成時，仍然需要欲合成方向的另一視點disparity map，所以我們將已有的disparity map根據disparity值進行forward warping至欲使用的視點，但是經過forward warping後的圖會有破洞產生，破洞會影響視點合成的品質，必須修補破洞後才能進行後續的視點合成，因此，如何修補破洞並得到良好的品質是非常重要的。此外，破洞修補也為我們演算法的運算瓶頸，需要硬體才能即時運算，為此我們也對此進行硬體架構的探討。以下將破洞修補分為演算法與架構部份做說明:在演算法部分，我們使用加權眾數濾波器(weighted mode filter, WMF)修補破洞，但是常見的WMF會造成前景資訊向外擴散且運算複雜度大，因此，我們提出優化方法，同時提升修補品質與降低運算複雜度；在硬體部份，傳統加權眾數濾波器的硬體架構於SRAM的使用數量與視差標籤(disparity label)範圍成正比，使得面積與成本隨著範圍增加而增加，因此，我們提出cacheable histogram的演算法與硬體架構用於WMF，達到減少SRAM數量與面積的目的。
本論文所提出的演算法，可以在光場架構下僅使用一張disparity map完成2D自由視點合成。此外，我們提出的硬體架構與演算法，使硬體面積皆小於傳統WMF架構；當應用於具有256個視差標籤之高畫素光場時，我們的硬體面積僅是其14 %的面積，在品質方面仍可以得到相近的disparity map與視點合成結果。

關鍵字

自由視點合成 ； 破洞修補 ； 視差圖 ； 加權眾數濾波器 ； 硬體架構

並列摘要

The applications of light field camera have gradually become the mainstream research in computational photography, and free viewpoint synthesis has become a hot spot in the research. Free viewpoint synthesis can synthesize any viewpoints which are required from light field camera. In recent years, the related research of free viewpoint synthesis of DIBR (depth-image-based rendering) has revealed that the number of disparity maps is as same as the number of viewpoints in the light field configuration. If these disparity maps are all from disparity search algorithm or sensor, the algorithm complexity and the hardware costs are too high. Therefore, it is important to synthesize good quality of viewpoint by using less disparity maps.
This thesis emphasizes the use of only one disparity map to complete free viewpoint synthesis in light field configuration. We find out that DIBR algorithm still needs another corresponding disparity map, so we forward warp the disparity map we already have to the corresponding viewpoint. However, holes appear after forward warping, and they affect the quality of viewpoint synthesis. Holes must be filled in order to synthesize viewpoint afterwards. Therefore, how to fill holes and achieve good quality are critical. In addition, hole filling algorithm is also the bottleneck in our system, so we design hardware architecture to reach real-time computation. We divide hole filling into two aspects: algorithm and architecture. In algorithm, we use weighted mode filter(WMF) to fill holes, but general WMF makes foreground information spread and requires high complexity. Our proposed method optimizes the quality of hole filling and reduce computational complexity. In hardware, the size of SRAM is proportional to the number of disparity labels in traditional hardware architecture of WMF. Therefore, we propose the algorithm and architecture of cacheable histogram to WMF. Proposed method can reduce the size of SRAM and hardware area.
Our proposed algorithm can complete 2D free viewpoint synthesis by using only one disparity map. We also propose hardware architecture and algorithm to make hardware area smaller than traditional WMF architecture. In the high resolution light field which has 256 disparity labels, our hardware area is only 14% of traditional hardware of WMF, but our method can still reach similar quality of disparity map and synthesized viewpoint.

並列關鍵字

free viewpoint synthesis ； hole filling ； disparity map ； weighted mode filter ； hardware architecture

參考文獻

[1]. M. Tanimoto, M.P. Tehrani, T. Fujii, and T. Yendo, “FreeViewpoint TV,” in IEEE Signal Processing Magazine, pp. 67-76, 2011.

[3]. M. Schaffner, P. Greisen, S. Heinzle, F. K. Gurkaynak, H. Kaeslin, and A. Smolic, “MADmax: A 1080p Stereo-to-Multiview Rendering ASIC in 65 nm CMOS based on Image Domain Warping,” in Proceedings of ESSCIRC, pp. 61–64, 2013.

[4]. N. Dodgson, “Autostereoscopic 3-D displays,” in IEEE Computer, pp. 31–36, 2005.

[6]. M. Levoy, “Light fields and computational imaging,” in IEEE Computer, pp. 46-55, 2006

[7]. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The Lumigraph,” in ACM SIGGRAPH 1996 Conference Proceedings, pp. 43–54, 1996.