基於圖像修補技術的錯誤補償演算法及架構設計

Image Inpainting Based Algorithm and Architecture Design for Error Concealment

陳慶懿(Ching-Yi Chen)

指導教授 ： 簡韶逸

國立臺灣大學/電機資訊學院/電子工程學研究所/碩士(2008年)

https://doi.org/10.6342/NTU.2008.01011

摘要

錯誤補償演算法(error concealment)是用來提升在通過無線網路後毀損視訊的品質。 其主要的原理是利用周圍正確接收到或是前一張影像的區域，來針對毀損區域做補償。在第一章中，會先介紹一些曾經被提出的空間(spatial)或時間(temporal)錯誤補償演算法。發現現有的空間錯誤補償演算法在毀損區域太大時容易產生模糊缺陷(blurring artifact)，所以我們引進了在電腦視覺領域中相當有用的圖像修補技術(image inpainting)，做為改良的基礎。
在第二章中，我們提出了一種基於圖像修補技術的空間錯誤補償演算法，來產生出令人眼視覺較舒服的補償結果。首先，毀損區域的結構會先從正確接收到的區域延伸出來，並且為了保持區塊(MB)間的邊緣連續性，區塊的補償順序將動態被決定。除此之外，當前一張影像存在時，混合了空間和時間錯誤補償法的演算法會被採用。而且毀損區域是一個個區塊補償，這樣在硬體實作上會較好控制，成本也比較低。總計在SRAM的實用上減少了90%，且每個區塊所需要的迴圈數可以被控制在25個之內。實驗結果證明所提出的演算法較H.264所定義的錯誤補償演算法有較好的效果並可以處理各種錯誤型態。
為了要達成更低的硬體成本並可以支援HDTV(1920x1080)的及時運算需求，在第三章中，我們更進一步的降低演算法的複雜度。空間補償演算法的運算基礎單元從一個像素變成4x4的小區塊，在加上一個設計好的選擇機制，每個區塊就可以用4個迴圈完成補償。在時間錯誤補償法上，我們採用一般常見的邊界比對演算法(boundary matching algorithm), 支援4個動態向量和一個4x5的搜索範圍。場景變換偵測(scene change detection)也被加入演算法，用來選擇目前此區塊要使用何種模式來補償。藉由這樣的修改，SRAM的使用會在之前的基礎上再下降50%，同時每個區塊所需要的迴圈數從25下降到4，大大降低了硬體控制機制的難度。.
在第四章中，我們嘗試將視訊解碼器和所提出錯誤補償演算法整合成一個系統。有的模組共用視訊解碼器現有的硬體來進行運算，有的模組則成為新的管線化架構(pipelining architecture)的一部分。在這一章的最後，所提出各模組會依據其特性做硬體/軟體切割(hardware/software partition). 低複雜度或需要彈性的函式就交由ARM處理器來運算，方便使用者做更改。其他一些比較固定或運算量較大的函式則以硬體來實現。以此為基礎，來提出一個理想中可行的整合架構。
之後，在第五章中，我們分析所提出之空間圖像修補和時間錯誤補償演算法所需要的運算量，並提出一個區塊管線化架構來平衡此兩種模式間運算量的不平衡。然後介紹一個可以達成及時運算的硬體架構。此架構有以下特點。首先，其記憶體的架構是類似於快取模組，來達成在兩個補償區塊間資料的共用，以減低頻寬的需求。另外，空間和時間兩種模式分享了同一套的運算元，以降低硬體成本。最後，此硬體符合AHB-AMBA標準，所以很容易為支援此標準的系統採用。
關於此硬體的數據可以在第六章找到。此硬體利用UMC 90nm 1P9M 製程技術實做錯誤補償晶片。根據合成與佈局繞線結果，這顆原型晶片輯閘總數大約為12 萬，核心大小為1.3×1.279mm2，最大的操作頻率可達200MHz；每秒可處理24 萬個巨集方塊；可即時解碼1920x1080 每秒30 張的視訊資料。它總共需要約5.6KBytes的內建記憶體。當操作於200MHz，1.1 伏特時，功率的消耗為131.4mW。跟目前現有的硬體相比，有較佳的視覺補償結果。是在一個容易產生傳輸錯誤的環境下又想要看到舒服視訊的一個選擇。此外，我們在SoC Designer 7.0這個平台上架設了一個系統來驗證此硬體。來證明我們的IP真的可以在SoC的系統中正確的運算，並且頻寬32bit的bus也足以支持及時運算的需求。在這篇論文的最後，結論和未來方向會被討論。

關鍵字

圖像修補 ； 錯誤補償 ； 硬體設計 ； 系統模擬 ； 晶片設計

並列摘要

Error concealment(EC) technique can improve subjective video quality in decoder when the video bitstream is corrupted during transmission. In Chapter 2, a new error concealment algorithm for I-frames is proposed. To achieve perceptually comfortable results, image inpainting algorithm is adopted with structure information generated from edge information. In addition, mixed spatial-temporal inpainting scheme is also proposed for I-frames when the previous frame is available. Moreover, a block-based scheduling is also developed for the proposed algorithm to make it simpler for hardware implementation. Experimental results show that the proposed algorithm outperforms the official H.264 decoder and can deal with various error patterns. It also shows that it is suitable for hardware implementation in terms of on-chip memory requirement and predictable processing cycles. The reduction of on-chip memory is 90%, and the iteration number of each MB is limited to 25 at most.
In Chapter 3, for low hardware cost design 1920x1080 real-time error concealment engine, the complexity of proposed algorithm is further reduced. The basic element becomes 4x4 block instead of pixel in spatial error concealment. And the BMA algorithm with four MV candidates and small search range are adopted in temporal error concealment. Scene change detection engine is added to select the concealed mode. With this algorithm, the reduction of on-chip memory is 95% comparing to original algorithm, and the iteration number of each MB is limited to 4 at most.
The integrated system of video decoder and proposed EC engine is introduced in Chapter 4. Some engines in proposed EC algorithm share the hardware of video decoder, or add to the pipelining architecture of video decoder in [17]. In the end of chapter, hardware/software partition is considered. The low complexity functions which require flexibility are processed by ARM processor. On the other hand, the fixed or complex functions will be implemented as the dedicated hardware. Based on this principle, the integrated system is proposed.
In Chapter 5, we analysis the work of spatial image inpainting and temporal error concealment. The new MB pipelining architecture is proposed to balance the required cycles of two concealed mode. Then the hardware architecture is proposed to support real-time decoding based on proposed MB pipelining architecture. This hardware engine is built on AHB-AMBA environment. Its characteristics are, two concealed mode share the same computational core to reduce hardware cost. And the cache based memory architecture is adopted to fulfill data reuse strategies between two processing MBs.
The statistics of hardware implement can be seen in Chapter 6. A prototype chip is implemented using UMC 90NM Logic & Mixed-Mode 1P9M Process Low-K technology. The total gate count is about 121K synthesized at 200 MHz. The maximum processing capability is 244.8K macroblocks per second or 1920×1080 4:2:0 30Hz video. Totally about 5.6 Kbytes on-chip memory is required. The core size is 1.3×1.279 mm2. The average power dissipation is 131.4 mW when operating at 200 MHz with 1.1 V power supply. Compared to other error concealment works, the proposed design can achieve better perceptual quality. Therefore it is a good choice to be integrated into the video decoding applications which transmission error occurs frequently and require good quality. Furthermore, system level simulation environment is built in SoC designer 7.0. It proves that our hardware IP can run on kind of system successfully can co-work with other IP without violation. In addition, required bandwidth can be support by 32 bit AHB-AMBA bus. In the end of this paper, conclusion and future work are given.

並列關鍵字

image inpainting ； error cocealment ； hardware design ； system level simulation ； HDTV ； SOC

參考文獻

[1] W. M. Lam, A. R. Reibmant, and B. Liu, “Recovery of Lost or Erroneously Received Motion Vectors,” in Proc. of IEEE Int’l Conf. Acoustic, Speech and Signal Processing, vol. 5, Apr. 1993, pp. 417–420.

[2] X. Qian, G. Liu, and H. Wang, “Texture Based Selective Block Matching Algorithm for Error Concealment,” in IEEE ICME, 2007, pp. 739–742.

[3] Y. K. Wang, M. M. Hannuksela, V. Varsa, A. Hourunranta, and M. Gabbouj, “The Error Concealment Feature in The H.26L Test Model,” in IEEE ICIP, vol. 5, Apr. 2002, pp. 729–732

[4] D. Agrafiotis, D. R. Bull, and C. N. Canagarajah, “Enhanced Error Concealment with Mode Selection,” in IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 8, pp. 960–973, Aug. 2006.

[5] M.H. Jo, H.N. Kim, and W.J. Song, “Hybrid Error Concealment Based on Block Content,” in IET Image Process., vol. 1, no. 2, pp. 141–148, Aug. 2007.