In this paper, we propose a novel algorithm to generate multiple virtual views from a video-plus-depth sequence for modern autostereoscopic displays. To synthesize realistic content in the disocclusion regions from the virtual views is the main challenging problem in this task. In order to produce perceptually satisfactory images, our proposed algorithm takes advantage of spatial coherence and temporal consistency to handle the uncertain pixels in the disocclusion regions. On the one hand, regarding the spatial coherence, we incorporate the intensity gradient strength with the depth information to determine the filling priority for inpainting the disocclusion regions, so that the continuity of image structures can be preserved. On the other hand, the temporal consistency is enforced by considering the intensities in the disocclusion regions across the adjacent frames through an optimization process. We propose an iterative re-weighted framework to jointly consider intensity and depth consistency in the adjacent frames, which not only imposes temporal consistency but also reduces noise disturbance. Finally, for accelerating the multi-view synthesis process, we apply the proposed view synthesis algorithm to generate the images plus depth at the leftmost and rightmost viewpoints, so that the intermediate views are efficiently interpolated through image warping according to the associated depth maps between the two views.

在本文中，我們提出了一個新演算法可以從一個視訊加深度的影像序列去產生多個虛擬視角的視訊，並可應用於現代的自動立體顯示器上。在這個任務中，要合成原本被遮蔽的區域裡的實際內容是最主要的挑戰。為了產生令人感覺滿意的影像，我們提出的方法利用了空間的連貫性和時間上的一致性去處理在原本被遮蔽的區域裡那些不確定的點。關於空間的連貫性，我們會整合色彩的強度梯度與深度資訊來決定填補的優先順序以維持影像裡的連續性結構。至於時間一致性方面，我們會將時間上相鄰影像的色彩強度加入考慮。我們提出了一個反覆進行的方法，這個方法不僅有加入了時間一致性也降低了雜訊干擾。最後，為了加速多視角的合成過程，我們提出的影像合成演算法會先產生最左端視角點的視訊加深度和最右端視角點的視訊加深度，然後就可以透過影像轉換有效率地內插出中間的視角點影像。

Contents
List of Figures ii
List of Tables v
List of Algorithms v

1. Introduction 1
2. Previous Works 4
2.1 Video Plus Depth 4
2.2 Layered Depth Video (LDV) 7
2.3 Depth Enhanced Stereo (DES) 9
2.4 Multiple Video Plus Depth (MVD) 10
3. Proposed Method 13
3.1 View Synthesis 14
3.1.1 Preprocessing of Depth Images 14
3.1.2 Image Warping 16
3.1.3 Preprocessing of Disocclusion Region 19
3.1.4 Recovering Disocclusion Region 22
3.1.5 Synthesis Refinement 28
3.2 View Interpolation 32
4. Experimental Results 35
4.1 Dataset 35
4.2 View Synthesis Results 36
4.3 View Synthesis Quantity 37
4.4 Execution Time 39
5. Conclusion 51

References 53

[1] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” IEEE International Conference on Computer Vision, pp. 839-846, 1998.
[2] C.M. Cheng, S.J. Lin, S.H. Lai and J.C Yang, “Improved novel view synthesis from depth image with large baseline,” IEEE International Conference on Pattern Recognition, pp.1-4, 2008.
[3] A. Criminisi, P. P´erez and K. Toyama, “Object removal by exemplar-based inpainting,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, No. 2, pp.721-728, 2003.
[4] Y. Mori, N. Fukushima, T. Fujii and M. Tanimoto, “View generation with 3D warping using depth information for FTV,” 3DTV Conference on The True Vision - Capture, Transmission and Display of 3D Video, pp. 229-232, 2008.
[5] M.M. Oliveira, B. Bowen, R. McKenna and Y.S. Chang, “Fast digital image inpainting,” International Conference on Visualization, Imaging and Image Processing, pp. 261-266, 2001.
[6] R-P.M. Berretty, F.J. Peters and G..T.G. Volleberg, “Real time rendering for multiview autostereoscopic,” SPIE on Stereoscopic Displays and Virtual Reality Systems XIII, Vol. 6055, pp.208-219, 2006.
[7] W. Mark, L. Mcmillan and G. Bishop, “Post-rendering 3D warping,” Symposium on Interactive 3D Graphics, pp. 7–16, 1997.
[8] F. Christoph, “A 3D-TV system based on video plus depth information,” Asilomar Conference on Signal, System and Computers, vol. 2, pp. 1529-1533, 2003.
[9] F. Christoph, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” SPIE on Stereoscopic Displays and Virtual Reality Systems XI, Vol.5291, pp.93-104, 2004.
[10] L. Zhang and W.J. Tam, “Stereoscopic image generation based on depth images for 3D TV,” IEEE Transactions on Broadcasting, Vol. 51, No. 2, pp. 191-199, 2005.
[11] K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, “Reliability-based generation and view synthesis in layered depth video,” IEEE International Workshop on Multimedia Signal Processing, pp.34-39, 2008.
[12] L. McMillan, “An image-based approach on three-dimensional computer graphics,” Ph.D. Dissertation, University of North Carolina at Chapel Hill, 1997.
[13] A. Smolic, K. Müller, K. Dix, P. Merkle, P. Kauff and T. Wiegand, “Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems,” IEEE International Conference on Image Processing, pp. 2448-2451, 2008.
[14] R. Tanger, “3D4U,” Workshops of the ITU-R, the SMPTE (Society of Motion Picture and Television Engineers) and the European Broadcasting Union on 3DTV Broadcasting, 2009.
[15] A. Telea, “An image inpainting technique based on the fast marching method,” Journal of Graphics, GPU and Game Tools, vol. 9, No. 1, pp. 23-34, 2004.
[16] M. Bertalmio, G. Sapiro, C. Ballester and V. Caselles, “Image inpainting,” ACM. SIGGRAPH Conference on Computer Graphics, pp. 417–424, 2000.
[17] G. Zhang, J. Jia, T.T. Wong and H. Bao, “Consistent depth maps recovery from a video sequence,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 31, No. 6, pp. 974-988, 2009.
[18] G. Zhang, J. Jia, T.T. Wong and H. Bao, “Recovering consistent video depth maps via bundle optimization,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[19] http://www.nue.tu-berlin.de/people/knorr/stereoscopic.html
[20] http://sp.cs.tut.fi/mobile3dtv/stereo-video/
[21] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, Vol. 47, pp. 7-42, 2002.
[22] Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600-612, 2004.
[23] N. Atzpadin, P. Kauff and O. Schreer, “Stereo analysis by hybrid recursive matching for real-time immersive video conferencing,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 14, No. 3, pp. 321-334, 2004.
[24] L. Wang, H. Jin, R. Yang and M. Gong, “Stereoscopic inpainting: Joint color and depth completion from stereo images,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.1-8, 2008.
[25] K.A. Patwardhan, G. Sapiro and M. Bertalmio, “Video Inpainting Under Constrained Camera Motion,” IEEE Transactions on Image Processing, Vol. 16, pp. 545-553, 2007.
[26] K.A. Patwardhan, G. Sapiro and M. Bertalmio, “Video inpainting of occluding and occluded objects,” IEEE International Conference on Image Processing, Vol. 2, pp. 69-72, 2005.