運用相機陣列之高解析度攝影與場景深度估測

High-Resolution Imaging and Depth Acquisition Using a Camera Array

施光祖(Kuang-Tsu Shih)

指導教授 ： 陳宏銘

國立臺灣大學/電機資訊學院/電信工程學研究所/博士(2017年)

https://doi.org/10.6342/NTU201701839

摘要

隨著攝影成本的大幅下降，這是一個只要有智慧型手機，人人都可能做攝影師的時代。也正因如此，研究者們花費極大的心力與成本追求更高品質的影像。在各種衡量影像品質的指標中，解析度可能是最具有代表性也最重要的一項。一般認為，藉由優化光學設計與硬體構造以追求更高解析度的手段已到達一個瓶頸。因此，人們轉而研究以計算為手段來提高影像解析度的方法。而在這篇論文中，我們探討的主題即為針對多相機系統設計的計算性高解析度攝影術。
本篇論文可分為二部分。第一部分集中於探討與分析現存的高解析攝影方法。我們特別深入研究兩類方法：「次相素重新對焦法」與「影像重建型光場超解像術」。在次相素重新對焦法的部分，我們由數學推導證明現存的方法普遍缺少了反卷積的步驟，並指出在次相素重新對焦的過程中加入反卷積演算法能夠有效提高輸出影像的銳利度。同時，我們亦設計實驗探討校正誤差對次相素重新對焦法的影響，並定量地分析在給定的影像品質之下最大可容忍的校正誤差。而在影像重建型光場超解像術的部分，我們以實驗證明了此類方法所能帶來的解像度增益並無法隨著相機數量的增加而無止盡地提高，並且指出解像度增益的上限是由系統的點擴散函數所決定的。我們所設計的系列實驗證明了輸出影像的解像度與影像對齊的準度無法兩全，而這也是此類型方法之基本限制。
相對於第一部分的分析，在第二部分，我們提出原創的高解析度計算攝影學系統。我們的系統是一混合了不同焦距的相機所組成的相機陣列。與傳統的多攝影機系統相比，我們所設計的相機陣列具有極高的像素利用率，並且能夠產生具有同樣高解析度的場景深度圖。我們的相機陣列由兩部分所構成。其一是優化過的相機布局，其二是一原創的影像融合演算法。在硬體部分，我們提出一優化相機布局的方法，並根據該方法提出非平行光軸與非均一焦距的設計。在軟體部分，我們的影像融合方法可以整合由各相機所拍攝之低解析度影像以產出高解析度影像，並且可以避免之前的影像融合方法所造成的影像模糊。

關鍵字

計算攝影學 ； 相機陣列 ； 高解析度 ； 深度估測

並列摘要

In this age where everyone can be a photographer with his or her smart phone, the pursuit of higher imaging quality has become more important and profitable than ever before. Among the quality metrics of images, resolution is often the top one that people care the most. Being one of the conventional approaches to increasing the image resolution, optics optimization is believed to have reached its bottleneck. As a consequence, researchers are turning to computational photography to seek breakthrough. In this dissertation, we study the computational approach to high-resolution imaging based on multi-aperture systems such as a camera array or a lenslet array.
This dissertation can be divided into two parts. The first part is dedicated to the analysis of existing approaches. Particularly, two approaches are inspected in depth: subpixel refocusing and reconstruction-based light field super-resolution. For subpixel refocusing, we show that a deconvolution step is missing in previous work and incorporating a deconvolution in the loop significantly enhances the sharpness of the results. We also conduct experiments to quantitatively analyze the effect of calibration error on subpixel refocusing and analyze the upper bound of the error for a targeted image quality. On the other hand, for reconstruction-based light field super-resolution, we show through experiments that the resolution gain obtainable by super-resolution does not increase boundlessly to the number of cameras and is ultimately limited by the size of the point spread function. In addition, we point out through experiment that there is a tradeoff between the obtainable resolution and the registration accuracy. The tradeoff is a fundamental limit of reconstruction-based approaches.
In contrast to the analysis work in the first part, the second part of the dissertation describes our original solution: a computational photography system based on a camera array with mixed focal lengths. Our solution has two distinguished features: it can generate an output image whose resolution is higher than 80% of the total captured pixels and a disparity map of the same resolution that contains the depth information about the scene. Our solution consists of optimized hardware and an image fusion algorithm. On the hardware size, we propose an approach to optimize the configuration of a camera array for high-resolution imaging using cameras with mixed focal lengths and non-parallel optical axes. On the software side, an algorithm is developed to integrate the low-resolution images captured by the proposed camera array into a high-resolution image without the blurry appearance problem of previous methods.

並列關鍵字

computational photography ； camera array ； high resolution ； depth estimation

參考文獻

[4] A. Levin, R. Fergus, F. Durand, and W. T. Freeman, "Image and depth from a conventional camera with a coded aperture," ACM Trans. Graph., vol. 26, no. 3, pp. 70:1–70:9, 2007.

[6] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, "Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing," ACM Trans. Graph., vol. 26, no. 3, pp. 69:1–69:12, 2007.

[12] C.-C. Chen, S.-C. Fan Chiang, X.-X. Huang, M.-S. Su, and Y.-C. Lu, "Depth estimation of light field data from pinhole-masked DSLR cameras," IEEE Int. Conf. Image Processing (ICIP), pp. 1769–1772, Sept. 2010.

[13] C. Liu, “Beyond pixels: exploring new representations and applications for motion analysis,” PhD dissertation, Massachusetts Institute of Technology, Cambridge, MA, 2009.

[15] C.-K. Liang and R. Ramamoorthi, "A light transport framework for lenslet light field cameras," ACM Trans. Graph., vol. 34, no. 2, pp. 16:1–16:19, 2015.