適用於智慧影像系統應用之全景視訊記錄SoC設計

A Panoramic Video Recording SoC Design for Intelligent Video System Applications

簡呈安(Cheng-An Chien)

指導教授 ： 郭峻因

國立中正大學/工學院/資訊工程學系/博士(2013年)

摘要

全景影像是現在智慧影像系統當中非常熱門的應用，包含了像是智慧型手機、街景系統、監視系統和行車紀錄器等。在現今這些應用皆因為全景影像接合的計算複雜度太高所以只能作到靜態影像接合出單張的全景影像而無法處理連續即時影像。同時現有的接合演算法大多只有使用軟體實現且不適用於連續影像的接合。
因此本篇博士論文提出了一個全景影像記錄系統晶片的架構來應用於智慧影像系統。本系統晶片提出了一個低複雜度的多鏡頭影像接合演算法來產生全景視角的影像，包含了360度全景影像或是寬景影像，並同時支援一個優化魚眼廣角鏡頭的影像校正。而在影像錄製方面，本系統晶片提出了一個低位元速率且有效節省頻寬之H.264視訊編碼器來壓縮所產生出來的全景影像，並可適用於大資料量錄製或是遠端即時網路傳輸。使用所提出的視訊接合演算法，使用者可以自動接合數個來源影像成一個全景影像，即使是來源影像有魚眼變形、不同色調、不同旋轉拍攝角度或是不同焦距，皆可以處理。而在整個系統晶片中我們提出了一些優化演算法使其可以即時接合視訊並透過H.264視訊編碼器來壓縮。首先，我們提出了基於特徵點的影像前調色作法來消除從不同鏡頭進來的來源影像色差。由於使用基於特徵點的演算法，我們可以相較於整張影像作直方圖統計來調色有效節省80%的計算複雜度。再來，我們提出一個在全景影像視訊中的動態接縫調整演算法。使用此調整演算法可以有效在連續影像接合中避開移動物件。接著在H.264視訊壓縮中，我們提出了一個雙階層快速MB skip模式壓縮演算法來節省原計算skip cost之SATD值。如果將所提出的MB skip演算法用於監視攝影系統上可以在固定QP以及相同PSNR下節省60%的位元速率。而在整個系統晶片架構上，我們提出一個多重矩陣轉換架構來同時支援魚眼鏡頭校正、影像接合以及360度全景投影，並可以藉由此架構來簡化接合流程並降低資料存取頻寬。再來，我們也提出了一個預先讀取的機制以及一個額外的暫存寫出空間並搭配MB-based的外部記憶體擺放方式來同時降低影像接合以及H.264視訊壓縮存取資料的頻寬來達到高解析度的即時全景影像應用。
本博士論文的系統晶片是用矽製程實作，在TSMC 40奈米製程下可擁有1180K gates以及27.06KB的SRAM。最後總而言之，本設計可以達到4-channel HD1080的即時全景接合以及H.264 HD1080的即時視訊壓縮。因此本系統晶片可以啟發未來多種全景影像的應用在各種智慧影像系統當中。

關鍵字

系統晶片 ； H.264視訊壓縮 ； 全景影像

並列摘要

Panoramic images are the cutting edge applications in the intelligent video system, such as smart phones, surveillance systems, vehicle recorders, and street view systems. Today these applications are only adopting the panoramic image stitching since the computational complexity is too high to be realized in real-time. Those image stitching algorithms are implemented by software and not suitable for video stitching.
This dissertation presents a panoramic video recording SoC architecture for intelligent video systems. The proposed SoC adopts a low-complexity multiple video stitching algorithm for generating panoramic view videos which includes 360 degree panoramic video (or wide range stitched video), supports optimized fisheye correction, and equips a low bit-rate bandwidth-efficient H.264 video encoder to encode the generated videos for remote real-time internet transferring and recording. Using the proposed video stitching algorithm, users can merge several videos into a panoramic one automatically, even though these source videos have fisheye distortion, different color bases, different rotation angles and zooming actions. By combining various videos from multiple cameras, the proposed design can stitch these videos into a panoramic view video and encode it through the encapsulated H.264 video encoder. In algorithm optimization, we first proposed a feature-based image pre-blending algorithm that can remove different color bases from multiple cameras. It is able to reduce 80% of the complexity by using the proposed feature based algorithm as compared to doing the histogram of whole image. Second, we proposed a dynamic seam adjustment in the panoramic image/video. By using this proposed algorithm, it is able to bypass the moving objects in the panoramic video stitching. Third, in designing the proposed H.264 video encoder for the panoramic video encoding, we proposed a two-stage fast MB skip mode algorithm without recalculating SATD value of skip cost. The algorithm can be applied to static surveillance panoramic video applications and save 60% of bit-rate in the same PSNR value as compared to those without using it. In the proposed SoC architecture, to simplify stitching flow and reduce the data bandwidth, we adopt a multiple matrixes transform architecture for fisheye correction, video stitching, and 360 degree projection. In addition, we design a pre-load scheduler and an extended output frame buffer module for accessing source YUV, reference data and reconstructed data to achieve high efficiency data access with a MB-based scan order in both video stitching and H.264 encoder when targeting at high-definition panoramic video applications.
The proposed SoC has been implemented in silicon, which comprises 1180Kgates and 27.06Kbytes SRAM according to TSMC 40nm CMOS technology. In summary, the proposed design is able to achieve real-time 4-channel video stitching on HD1080@30fps (per channel) and H.264 video encoding on HD1080@30fps. The proposed system is able to bring enormous advantages in various intelligent video applications.

並列關鍵字

H.264 video encoding ； Panoramic video ； SoC

參考文獻

[3] Y. W. Huang, T. C. Chen, C. H. Tsai, C. Y. Chen, T. W. Chen, C. S. Chen, C. Fu. Shen, S. Y. Ma, T. C. Wang, B. Yu. Hsieh, H. C. Fang, L. G. Chen, "A 1.3TOPS H.264/AVC single-chip encoder for HDTV applications," Proc. IEEE International Solid-State Circuits Conference, pp. 128-588, Febuary 2005.

[4] C. P. Lin, P. C. Tseng, Y. T. Chiu, S. S. Lin, C. C. Cheng, H. C. Fang, W. M. Chao, L. G. Chen, "A 5mW MPEG4 SP encoder with 2D bandwidth-sharing motion estimation for mobile applications," Proc. IEEE International Solid-State Circuits Conference, pp. 412-413, Febuary 2006.

[8] Y. K. Lin, D. W. Li, C. C. Lin, T. Y. Kuo, S. J. Wu, W. C. Tai, W. C. Chang, and T. S. Chang , " A 242mW 10mm2 1080p H.264/AVC High-Profile Encoder Chip," Proc. IEEE International Solid-State Circuits Conference, pp. 314-315, Feb. 2008.

[10] S. Mochizuki, T. Shibayama, M. Hase, F. Izuhara, K. Akie, M. Nobori, R. Imaoka, H. Ueda, K. Ishikawa, and H. Watanabe, "A 64 mW High Picture Quality H.264/MPEG-4 Video Codec IP for HD Mobile Applicationsin 90 nm CMOS," IEEE Journal of Solid-State Circuits, vol. 43, no. 11, pp. 2354-2362, November 2008.

[16] M. Kim, I. Hwang, and S. I. Chae, ”A fast VLSI Architecture for Full-Search Variable Block Size Motion Estimation in MPEG-4/H.264,” Proc. Asia and South Pacific Design Automation Conference, vol. 1, pp. 631-634, January 2005.