應用於計算式攝影之光場分析,擷取,及處理

Analysis, acquisition, and processing of light field for computational photography

梁家愷(Chia-Kai Liang)

指導教授 ： 陳宏銘

國立臺灣大學/電機資訊學院/電信工程學研究所/博士(2009年)

https://doi.org/10.6342/NTU.2009.02719

摘要

在這篇論文中，我們提出了新的理論，硬體設計，以及演算法來改變分析影像成像之方法和後續的處理方式，進而大幅提升攝影時使用者的自由度和方便性，以及影像之品質和可用性。

我們首先利用光傳輸之概念，把影像成像之過程描述成一四維光場信號之轉換。這個全新的分析方式可以同時描述所有成像之現象而不需要許多獨立之數學模型。這個分析方式同時也可推廣到許多新的應用上，例如合成全對焦圖片，分析光場相機之參數設定，推測場景深度等。

接下來，我們提出了一個新的硬體裝置，稱為可程式光圈。利用此一裝置，我們可以在不影響圖片解析度下，利用單一一台相機，擷取四維之光場信號。利用四維的光場，我們可以達到許多傳統攝影所無法達成之效果，例如拍照後，利用數位信號處理之方式改變對焦平面，利用特徵點比對之互動式數位變焦系統，以及高畫質的圖片合成系統。另外，我們也提出新的演算法，來修正擷取道之光場中的光度失真，以及還原出場景之立體模型。

最後，我們提出兩個可以應用於光場信號以及其他視覺信號之演算法。第一個是平行化之訊息傳遞演算法。我們提出一個不影響結果的新的訊息傳遞模式，可以大幅減少演算法所需之記憶體與頻寬。我們把這技術同時實作在VLSI電路以及GPU程式中而得到大幅的加速。另外，我們提出了一個考慮信號雜訊特性之分解多工信號之演算法。我們考慮了視覺信號中特有的雜訊特性，進而把分解多工信號描述為一個L1正規化的最佳化問題。這個新的演算法可以大幅提升所得訊號之信號雜訊比。

關鍵字

光場 ； 計算式攝影 ； 多視角立體視覺 ； 影像處理 ； 電腦視覺

並列摘要

Photography is an abstruse skill. Taking a perfect photo needs a great deal of knowledge in aesthetics, physics, optics, and even electronics and also requires a lot of patience. In this dissertation, we examine the process of photography using 4D light field representation. This new approach leads to a novel framework to describe the image formulation, a new device to capture high dimensional visual data, and many new applications.

In the first part of the dissertation, we analyze the light transport of the light rays in the picture-capturing process and show that several photographic effects, including magnification, focusing, and vignetting, can better be explained in the 4D light field domain or the dual 4D frequency domain than in the 2D image domain. We also show this framework can be applied to many new applications, such as digital refocusing, all-focused fusion, light field camera parameter setting, depth detection without correspondence matching, and so forth.
In the second part of the dissertation, we present a new compact device, called programmable aperture, to capture the light field without moving the camera or losing the image resolution. The device is more flexible, inexpensive, easier to calibrate than the previous light field cameras. It also enables the multiplexing of the light field to improve the data quality. We show several different implementations of the programmable aperture and compare the performance of different imaging devices. We then remove the inherent defects of the captured light field by two novel postprocessing algorithms. The photometric calibration algorithm can automatically remove the vignetting-alike effects without any reference object. The multi-view depth estimation generates per-view depth maps from the light field. It utilizes accurate occlusion model and cross-bilateral filtering to efficiently achieve high quality results. The combination of the device and the algorithms produce a distortion free, high spatial- and
high angular- resolutions light field with auxiliary depth information of the scene. We demonstrate several applications using the captured light field, including view interpolation, digital refocusing, and a novel feature-based refocusing.

In the third part of the dissertation, we describe two spinoff topics. These two topics are not only related to the practical light field acquisition and applications, but are also very important to other computer vision and signal processing problems. The first topic is the computational and storage bottleneck of the global optimization algorithms. We make a complete analysis of the bandwidth and memory cost of the belief propagation and then propose a new tile-based belief propagation algorithm. While its performance is very close to the original state belief propagation algorithm, the memory and bandwidth requirements are reduced by orders of magnitude. We also propose a new message construction method which can be applied to all robust smoothness functions. These two algorithms enable efficient VLSI or GPU implementations and make the global optimization more affordable to the resource-limited platforms for real-time mobile applications. The second topic is about the signal demultiplexing. Traditional demultiplexing method assumes that the noises are independent, identically distributed, which is invalid in the image formulation. We reformulate the demultiplexing process in a probabilistic framework and show that when the noise is dependent to the sensed signal, the MAP estimation is equivalent to a L1 regularized least-square problem. The simulation results show that the signal recovered by the proposed algorithm has a higher signal-to-noise ratio than that recovered by the traditional demultiplexing method. We believe this new formulation can also be applied to other applications that require demultiplexing.

In summary, this dissertation presents several novel and important solutions to analyze, acquire, and efficiently utilize the high dimension, high quality, and high resolution visual data. We show that by using the light field representation, the photographic effects can be described in a way more close to the real functionality of the camera. We present a novel backward-compatible device that can capture high resolution light field data without moving the device itself or using complex optics, and demonstrate several novel applications of the light field data. Finally, we propose a modified belief propagation algorithm which removes the fundamental memory, bandwidth, and computation bottlenecks of the original algorithm, and a new noise-aware demultiplexing algorithm which has a better performance than the traditional one.

並列關鍵字

light field ； computational photography ； image-based rendering ； multi-view stereo ； light transport analysis ； image processing ； computer vision

參考文獻

[2] Parry Moon and Domina Eberle Spencer, The photic field, MIT Press, 1981.

[6] Shree K. Nayar, Masahiro Watanabe, and Minori Noguchi,“Real-time focus range sensor,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 12, pp. 1186–1198, Dec 1996.

[8] James T. Kajiya, “The rendering equation,” SIGGRAPH Comput. Graph., vol. 20, no. 4, pp. 143–150, 1986.

[15] Ravi Ramamoorthi and Pat Hanrahan, “A signal-processing framework for reflection,”ACM Transactions on Graphics, vol. 23, no. 4, pp. 1004–1042, 2004.

[16] Ravi Ramamoorthi and Pat Hanrahan, “Frequency space environment map rendering,”ACM Trans. Graph., vol. 21, no. 3, pp. 517–526, 2002.