Title

以影格中的不變特徵為基礎的全景修補演算法

Translated Titles

Panoramic Inpainting based on Invariant Features of Video Frames

DOI

10.6846/TKU.2014.00464

Authors

張仕明

Key Words

全景影像 ; 動態修補 ; 自動化物件切割 ; Panorama ; Motion Inpainting ; Automatic GrabCut

PublicationName

淡江大學資訊工程學系博士班學位論文

Volume or Term/Year and Month of Publication

2014年

Academic Degree Category

博士

Advisor

顏淑惠

Content Language

英文

Chinese Abstract

全景成像技術是近年來十分重要的議題,到目前為止已有許多全景成像的技術發展在各式各樣的產品上。從最早期開發在個人電腦中以及近年來開發在數位相機與可攜式裝置中都可以看到這類技術的運用。在傳統的全景成像技術中,使用者可以使用一連串的影像或是一小段影片來建立同一場景的全景圖。但在傳統的全景成像技術中,如果影像或影片內有移動的物件時,所建立出來的全景圖背景會保留其物件,同時當全景圖的素材中有較為明顯的人物時也容易造成人物結構上的模糊。因此本論文針對此問題提出一個運用人臉辨識、物件切割、動量背景修補以及有效的全景合成技術的演算法。針對背景有移動人物的場景,將其人物與背景先進行分離,在此部份本論文先使用著名的物件切割方法-GrabCut演算法來切割物件並且將其改良為自動化程序,讓使用者可以不用經過手工的標記物件即可達到物件切割的效果。接著使用動量背景修補方法將分離出來的區域填補成原先可能的背景,在此部分本論文根據傳統的影像修補方法並加入動量的參考依據讓背景也跟著在移動的情況下也可以順利地得到正確的修補結構。最後再將所有的素材合成為一張全景影像,本論文所提出的全景圖製作方法加入了影像能量圖與影像縫線的概念,讓接合的縫線落在重疊區域內較不明顯的結構中,如此一來可以有效地解決人物鬼影和因為重疊位置的關係所形成的結構模糊的問題。而人臉辨識的運用除了定義出人物的大概區域外,也可以用來判斷針測到的人物是否是使用者認識的人,如果認定是陌生人時即可利用上述的方法進行移除的動作,進一步達到過濾陌生人的功能。本論文所提出的方法除了人物區域定義是可以進行手動設定,其他的步驟皆為自動化程序,如此可有效減少使用者操作上的負擔。

English Abstract

Panoramic photography is becoming very popular within the general users, skilled photographers and in many useful computer and Internet based application domains like 3D virtual reality. With the introduction of panoramic photography support in the general purpose digital cameras and smart phones, users and applications that use the panoramic photos are also increasing. In traditional panoramic photography, moving objects or as referred in this thesis - the strangers, in the background should be eliminated since those strangers obscure the scenery that we want to retain in our photograph. This thesis discusses a novel method to remove the strangers (moving objects) whose face data is not available in the face database of camera) from the background of the focused area and compose a panoramic image. In the proposed system the object segmentation is automation and based on GrabCut algorithm. The method of motion inpainting of background can be repair background on moving background, effectively. The method of panorama creation is using concept of energy map and image seam that avoided ghost problem in panorama and maintained the structure of human. The proposed of panorama creation system is fully automatic except that the user required marking the unidentified moving objects in the object segmentation phase.

Topic Category 基礎與應用科學 > 資訊科學
工學院 > 資訊工程學系博士班
Reference
  1. [1]. M. Brown, D. G. Lowe (2007) Automatic Panoramic Image Stitching using Invariant Features. International Journal of Computer Vision. 74(1):59-73.
    連結:
  2. [2]. Saeid Fazli, Hamed Moradi Pour, Hamed Bouzari, (2009) Particle Filter based Object Tracking with Sift and Color Feature. International Conference on Machine Vision. pp: 89-93.
    連結:
  3. [4]. H. Bay, A. Ess, T. Tuytelarrs, L. J. V. Gool, (2008) Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding. 110:346-359.
    連結:
  4. [8]. Song, Baosen, Yongqing Fu, Jinlin Wang (2011) Automatic panorama creation using multi-row images. Information Technology Journal 10:1977-1982.
    連結:
  5. [11]. M. Brown, D. G. Lowe (2003) Recognising Panoramas. In Proceedings of the 9th International Conference on Computer Vision (ICCV2003). pp: 1218-1225.
    連結:
  6. [12]. I. Laurent, K. Christof, and N. Ernstr (1998) A Model of Saliency Based Visual Attention for Rapid Scene Analysis, IEEE transactions on pattern analysis and machine intelligence. 20(11): 1254-1259.
    連結:
  7. [14]. Shai Avidan, Ariel Shamir (2007) Seam Carving for Content-Aware Image Resizing. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH. 26(3):Article 10.
    連結:
  8. [16]. S. Battiato (2012). Content-based image resizing on mobile devices. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP). pp: 87-90.
    連結:
  9. [17]. Chenjun Tao, Jiaya Jia and Hanqiu Sun (2007). Active window oriented dynamic video retargeting. In Proceedings of the Workshop on Dynamical Vision. pp: 1-12.
    連結:
  10. [19]. Li-Qun Chen, Xing Xie, Xin Fan, Wei-Ying Ma, Hong-Jiang Zhang and He-Qin Zhou (2003). A visual attention model for adapting images on small displays. Multimedia System. 9(4):353-364.
    連結:
  11. [20]. Xin Fan, Xing Xie, He-Qin Zhou and Wei-Ying Ma (2003). Looking into video frames on small displays. In Proc. Eleventh ACM Int. Conf. Multimedia. pp: 247-250.
    連結:
  12. [22]. Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin and Michael Cohen (2006). Gaze-based interaction for semi-automatic photo cropping. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp: 771-780.
    連結:
  13. [23]. Yunfeng Zhang, Shanshan Gao, Caiming Zhang and Jing Chi (2009). Application of a bivariate rational interpolation in image zooming. , International Journal of Innovative Computing, Information and Control. 5(11):4299-4307.
    連結:
  14. [25]. Yu-Ting Chen, Chu-Song Chen (2008) Fast Human Detection Using a Novel Boosted Cascading Structure With Meta Stages. IEEE Transactions on Image Processing. 17(8):1452-1464.
    連結:
  15. [26]. C. Rother, V. Kolmogorov, A. Blake (2004) "GrabCut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23:309-314.
    連結:
  16. [29]. Yuri Y. Boykov, Marie-Pierre Jolly (2001) Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images. Eighth International Conference on Computer Vision (ICCV'01). 1:105-112.
    連結:
  17. [30]. Patwardhan KA, Sapiro G, Bertalmio M. (2007) Video Inpainting Under Constrained Camera Motion. IEEE Transactions on Image Processing. pp: 545-553.
    連結:
  18. [31]. Y. Wexler, E. Shechtman, M. Irani (2007) Space-Time Completion of Video. IEEE Trans. on Pattern Analysis and Machine Intelligence. 29:463-476.
    連結:
  19. [32]. Yu-Ting Chen, Chu-Song Chen (2008) Fast Human Detection Using a Novel Boosted Cascading Structure With Meta Stages. IEEE Transactions on Image Processing. 17(8):1452-1464.
    連結:
  20. [33]. C. Kokaram, S. J. Godsill (1997) Joint Detection, Interpolation, Motion and Parameter Estimation for Image Sequences with Missing Data. In International Conference on Image Processing. pp: 191-194.
    連結:
  21. [34]. L. -M. Po, W. C. Ma (1996) A Novel Four-Step Search Algorithm for Fast Block Motion Estimation. IEEE Trans. Circuits Syst. Video Technol. 6(3):313-317.
    連結:
  22. [35]. R. Li, B. Zeng, M. L. Liou, (1994) A New Three-Step Search Algorithm for Block Motion Estimation. IEEE Trans. Circuits Syst. Video Technol. 4(4):438-442.
    連結:
  23. [36]. C. H. Cheung, L. M. Po (2005) Novel cross-diamond-hexagonal search algorithms for fast block motion estimation. IEEE Transactions on Multimedia. 7(1):16-22.
    連結:
  24. [37]. S. Vassiliadis, E. A. Hakkennes, J. S. S. M. Wong, G. G. Pechanek (1998) The Sum-Absolute-Difference Motion Estimation Accelerator. In the 24th. EUROMICRO Conference. pp: 559-566.
    連結:
  25. [39]. Wan-Lei Zhao, Chong-Wah Ngo (2009) Scale-Rotation Invariant Pattern Entropy for Keypoint-Based Near-Duplicate Detection. Image Processing of IEEE Transactions. 18(2):412-423.
    連結:
  26. [40]. Yifan Lu, Lei Wang, Hartley, R.,Hongdong Li, Chunhua Shen (2008) Multi-view Human Motion Capture with an Improved Deformation Skin Model. Computing: Techniques and Applications (DICTA) Digital Image. pp: 420-427.
    連結:
  27. [41]. J. M. Morel, G. Yu (2009) ASIFT: A New Framework for Fully Affine Invariant Image Comparison. SIAM Journal on Imaging Sciences. 2(2):438-469.
    連結:
  28. [42]. A. Criminisi, P. Perez, K. Toyama (2004) Region Filling and Object Removal by Exemplar-Based Image Inpainting. IEEE Trans. On Image Processing. 13:1200-1212.
    連結:
  29. [43]. A Criminisi, I Reid, A Zisserman (1999) A plane measuring device. Image and Vision Computing. 17(8):625-634.
    連結:
  30. [44]. B. Heigl, R. Koch, M. Pollefeys, J. Denzler, L. Van Gool (1999) Plenoptic modeling and rendering from image sequences taken by hand-held camera. In Mustererkennung. pp: 94-101.
    連結:
  31. [3]. H. Bay, T. Tuytelarrs, L. J. V. Gool, (2006) SURF: Speeded Up Robust Features. European Conference on Computer Vision. 3951:404-417.
  32. [5]. Kekre, Hemant B., Sudeep D. Thepade. (2008) Rotation Invariant Fusion of Partial Image Parts in Vista Creation using Missing View Regeneration. WASET International Journal of Electrical Computer and Systems Engineering (IJECSE) 47: 660.
  33. [6]. Helmut Dersch (2007) Panorama Tools. Open Source Software for Immersive Imaging International VR Photography Conference, 2007.
  34. [7]. Wang Meng (2009) Panorama Painting: With a Bare Digital Camera. In Image and Graphics, 2009. ICIG'09. Fifth International Conference on. pp: 82-86.
  35. [9]. Yingen Xiong, Kari Pulli (2010) Fast image stitching and editing for panorama painting on mobile phones. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp: 47-52.
  36. [10]. Wen-Yan Lin, Siying Liu, Yasuyuki Matsushita, Tian-Tsong Ng, Loong-Fah Cheong. (2011) Smoothly varying affine stitching. Computer Vision and Pattern Recognition (CVPR) 2011 IEEE Conference on. pp: 345-352.
  37. [13]. F. Liu and M. Gleicher (2005) Automatic image retargeting with fisheye-view warping. In Proc. ACM Symposium on User Interface Software and Technology. pp: 153-162.
  38. [15]. Y. S. Wang, C. L. Tai, O. Sorkine and T. Y. Lee (2008). Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAP Asia 2008. 27(5): Article 118.
  39. [18]. Sunghyun Cho, Hanul Choi, Matsushita, Y. and Seungyong Lee (2009). Image retargeting using importance diffusion. Image Processing (ICIP), 2009 16th IEEE International Conference on. pp: 977-980.
  40. [21]. Jianping Xiao, Xuecheng Zou, Zhenglin Liu and Xu Guo (2007). A novel adaptive interpolation algorithm for image resizing. International Journal of Innovative Computing, Information and Control. 3(6):1335-1345.
  41. [24]. Jian Yao, Jean-Marc Odobez (2008) Fast human detection from videos using covariance features. Presented at: The Eighth International Workshop on Visual Surveillance (VS2008). http://hal.inria.fr/inria-00325628/. Accessed 29 Septembre 2008.
  42. [27]. S. Vicente, V. Kolmogorov, C. Rother (2008) Graph cut based image segmentation with connectivity priors. In IEEE Conference on Computer Vision and Pattern Recognition. pp: 23-28.
  43. [28]. V. Vezhnevets, V. Konouchine (2005) Grow-Cut - Interactive Multi-Label N-D Image Segmentation. Proc. Graphicon. pp: 150-156.
  44. [38]. Bruno Postle (2011) Panorama Tools. Open Source Software for Immersive Imaging. http://panotools.sourceforge.net/. Accessed 22 February 2011.