基於SIFT特徵點引導修補方向之視訊修補演算法

Video Inpainting Based on SIFT Feature Point to Modify Repairing Direction

管雲平(Yun-Ping Kuan)

指導教授 ： 郭天穎

國立臺北科技大學/電資學院/電機工程系研究所/碩士(2012年)

摘要

視訊修補(Video Inpainting)是針對視訊畫面受損或是不要的人工特效進行修復和移除。經過此程序視訊影片看起來會恢復成完整或沒有被後製的樣子。視訊修補演算法基本上可以視為將單張畫面的影像修補(Image Inpainting)演算法延伸到視訊中多張畫面的應用。基於影像修補可以分為傳統的直接對畫面進行修補以及先考慮修補優先權再進行修補的兩種技術，視訊修補技術也有這樣的差別。傳統直接取鄰近畫面資訊進行修補的方法通常為了維持畫面在時間上的連續性而導致空間一致性不夠產生畫面模糊的現象，然而先將畫面內容進行分類再對不同類型的畫面內容分先後做修補處理，雖然可以維持修補畫面空間中的一致性，但是分類機制很難適應於各種視訊畫面，所以一旦分類錯誤就會嚴重影響修補效能。
本論文提出以簡單的分類機制將畫面內容分成不同的類型，但是我們的修補順序不同於相關文獻是依照畫面內容的類型來決定，而是以一個排程機制決定不同畫面類型的修補優先權。此外加入向量場(Motion Field)的亂度(Entropy)作為優先權決策機制在時間域的參考，同時利用不同尺度空間的特徵資訊來引導修補方向，使得我們的修補結果不但同時維持良好的時間連續性(Continuity)以及空間一致性(Consistency)，而且可以適應於各種不同的畫面。我們透過實際的電視節目與傳統文獻中的視訊影片進行測試，證實本論文提出的視訊修補演算法的確較傳統文獻方法優異。

關鍵字

Haar小波轉換 ； 影像修補 ； 尺度不變特徵轉換 ； 動作向量 ； 熵

並列摘要

The video inpainting is a technique to repair the damaged part of video clips or to remove unwanted artificial post-production effects on them. The video after inpainting should be restored to a good state with a look without being ever altered. Video inpainting is basically regarded as the extended version of image inpainting. Image inpainting can be divided into two technologies, direct repair and priority-based repair, and video inpainting is the same. The direct repair of video inpainting takes the information of the adjacent frame as the source to maintain the temporal continuity of the repaired frame, but it could result in spatial inconsistencies. Priority-based approach is to classify the contents of video frames into two types and make them corresponding to different repair priorities. Although the priority-based approach could maintain the consistency of the most frames, the classification mechanism is difficult to adapt to different types of videos, because the classification error would affect the repair performance.
In this paper, we propose a video inpainting technique with a simple mechanism to divide the contents of a video frame into different types. Unlike that the literature works have to process each type in different order, we do not make the type strictly corresponding to the repair order as we also implement a scheduling mechanism for the priority. Furthermore, we use the entropy of the motion field as a reference to the priority mechanism to consider time domain information to improve the consistency of the repaired frame. We also use the robust SIFT features points as a guide to the repair direction in the spatial domain. Therefore, our repair results can maintain good continuity in both temporal and spatial domains, and can adapt to a variety of different videos. Through the experiment on the video sequences used in television broadcast and in the existing works, our proposed method is proved to be superior to other methods.

並列關鍵字

Haar wavelet transform ； image inpainting ； scale-invariant feature transform ； motion vector ； entropy

參考文獻

[2] A. Efros, T. Leung, “Texture synthesis by non-parametric sampling,” Proc. IEEE International Conference Computer Vision, pp. 1033-1038, 1999.

[3] LIANG, L., LIU, C., XU, Y.-Q., GUO, B., and SHUM, H.-Y., “Real-time texture synthesis by patch-based sampling,” ACM Transactions on Graphics Vol. 20, pp. 127–150, 2001.

[4] A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Proc. Conf. Computer Vision and Pattern Recognition, pp.1-8, 2003.

[5] A. Criminisi, P. Perez and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting,” IEEE Transactions on Image Processing, Vol.13, pp.1200-1212, 2004.

[7] Y. Wexler, E. Shechtman, and M. Irani, “Space-Time Completion of Video,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 463-476, Mar. 2007.