應用於全高清鬆散光場之基於區塊切割快速擬真重新對焦硬體架構設計

VLSI Architecture and System Design of Fast Realistic Block-based Refocusing for Full-HD Sparse Light Fields

楊舒雯(Shu-Wen Yang)

指導教授 ： 黃朝宗

國立清華大學/電機資訊學院/電機工程學系所/碩士(2018年)

摘要

近年來，多鏡頭的應用已經是智慧型手機拍照的新趨勢，拍完照再重新對焦的功能，儼然成為一種取得含有淺景深影像的新方式。但目前市面上未有能同時保有高畫質且擬真的影像與即時生成的技術，所以在此方面的研究與應用的推廣越來越加重要，並為多種應用開闢新的可能性，例如：動態追焦、虛擬實境顯示。因此在本論文中，我們設計一個適用於全高清鬆散光場的重新對焦引擎，此引擎以區塊為單位來合成大量新視點，進而完成重新對焦的影像。

我們將光場資訊儲存在晶片外部的記憶體，使用鬆散光場可以有效減少了相對於緊密光場所需的資料儲存量，但需要內插的新視點數量非常龐大，而內插的新視點所需的大量記憶體成本，我們利用多層級的排程，達到不論內插的新視點數量，也能固定晶面內部的記憶體用量。此外，為了能即時產生重新對焦的影像，我們使用了兩種不同平行度的內插新視點的引擎：四個像素-四條線的單視點產生引擎（Single-view Renderer）、四個像素-四條線-四個視點的多視點產生引擎（Multi-view Renderer），以達到該設計的高吞吐量。

我們基於台積電四零奈米製程實作了一個支援全高清即時擬真重新對焦的超大型積體電路，此電路使用了271KB的晶片內部記憶體以及2.3百萬的邏輯閘，而此晶片的面積為1.9x1.9 mm2。運作在200MHz時，它可以每秒提供5.4G新視點畫素的吞吐量來即時產生出全高清的重新對焦的影像。

關鍵字

事後重新對焦 ； 全高清 ； 鬆散光場 ； 硬體架構設計

並列摘要

In recent years, multi-lenses cameras becomes the mainstream of smart-phones. The ability of changing focal planes after capturing the light field enables users to get the shallow-depth-of-field images. However, the fast realistic refocusing technique with high resolution is still a challenge. Therefore, this research topic becomes more and more important, and it will open more possibilities for many applications, such as auto-panning video and virtual reality display. In the thesis, we propose a Full-HD refocusing engine, which uses block-partitioned sparse light fields to synthesize numerous viewpoints and get a refocused image.

We store the sparse light fields in off-chip memory, which have less storage demand than dense light fields. But the number of rendered viewpoints is tremendous, and it results in high requirement of on-chip memory. The schedule of hierarchical pipeline is designed to synthesize numerous viewpoints using fixed number of on-chip memories. Besides, we apply two types of view-renderers with different parallelism to support the high throughput, including single-view renders in four-pixel/four-line parallelism and multi-view renders in four-pixel/four-line/four-view parallelism.

We implemented a VLSI circuit to support real-time Full-HD refocusing using TSMC 40nm technology process with 271-KB on-chip memory and 2.3M logic gates. The core area is 1.9x1.9 mm2. When synthesized at 200 MHz, it can deliver 5.4G rendered pixels per second and 40M refocused pixels per second to support Full-HD refocusing in real-time.

並列關鍵字

VLSI Architecture ； Fast Realistic Refocusing ； Full-HD Sparse Light Fields

參考文獻

[1] Lytro Camera, “https://www.lytro.com/,” .

[2] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy, “High performance imaging using large camera arrays,” in SIGGRAPH, 2005, pp.765–776.

[3] Mathieu Bredif Gene Duval Mark Horowitz Ren Ng, Marc Levoy and Pat Hanrahan, “Light field photography with a hand-held plenoptic camera,” 2005.

[4] Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar, “Picam: An ultra-thin high performance monolithic camera arrays,” in SIGGRAPH Asia, 2013, pp. 166:1–166:13.

[5] M. Harris, “Focusing on everything,” IEEE Spectrum, vol. 49, no. 5, pp.44–50, 2012.