利用高解析度彩色影片提升深度圖的品質

Depth Map Enhancement based on Its Associated High-Resolution RGB Video

翁司頻(Si-Pin Weng)

指導教授 ： 杭學鳴

國立交通大學/電機學院/電子工程系所/碩士(2015年)

https://doi.org/10.6842/NCTU.2015.00661

摘要

隨著科技的發展，如何利用深度圖突破現有的科技已經成為趨勢。Microsoft 在2010年發表的Kinect for Xbox 360 可視為深度感測器應用在人們日常生活中的重大突破。玩家只需要在Xbox 360前做簡單的動作，不需任何遙控器或控制器便可以操縱遊戲的進行。而這一切都要歸功於深度感應器Kinect，它會偵測前方物體距離感應器的距離，進而產生深度圖。雖然Kinect產生的深度圖對於辨識站在Xbox 360前方玩家的肢體動作已經足夠讓遊戲順利進行，但相對於彩色影像的解析度而言，深度圖的解析度較小且品質較差，這將會影響電腦圖形辨識的發展。因此，如何提升深度圖的品質已經成為未來多媒體應用的重要議題。
在本實驗中，我們採用Microsoft 在2014年發表的Kinect for Xbox Onex來擷取深度資訊與彩色資訊，雖然相對於其他深度感測器，Kinect v2 產生的深度圖擁有較高解析度且可靠性較佳，但跟彩色影像相比，還是會限制許多深度圖應用的發展。因此，我們提出一套演算法，藉由高解析度彩色照片改善深度圖的品質(高解析度、低雜訊) 。
Kinect v2 產生的深度圖解析度較低，且有雜訊及誤差區塊，因此為了要改善深度圖的品質，我們利用Kinect v2可以同時錄製彩色影片及深度影片的特性，將原本又小且又有許多誤差區塊的深度圖，轉變為高解析度且無破洞的深度圖。本篇論文提出的演算法可藉由影片的資訊建立出深度圖背景，並且藉由彩色影像的資訊，修正深度圖上的誤差區塊。最後拿我們提出的方法產生的深度圖與其他人的方法結果相比，實驗結果顯示，我們的方法可以產生較自然的高解析度深度圖。

關鍵字

高解析度深度圖 ； 深度圖 ； 深度圖背景重建 ； 深度圖破洞填補

並列摘要

Owing to the advance of science and technology, the depth map application is getting more and more popular. Kinect for Xbox 360 made by Microsoft Company in 2010 marked a big step in the consumer market for the Human-Machine Interaction (HMI) devices. Kinect measures the distance using the reflected infrared light to generate the depth map. But the depth map generated by Kinect has low resolution and holes. For many depth map applications, the high quality depth maps are necessary. Consequently, how to enhance the quality of depth maps is becoming a popular research topic for many multimedia applications.
In our experiments, we use Kinect for Xbox One (Kinect v2) made by Microsoft Company in 2014 to capture the color sequences and the depth sequences. The depth map captured by Kinect v2 has higher resolution than many other popular depth sensors. But it is not sufficient to use in practical applications. Hence, we propose an algorithm to improve the depth maps based on the associated high resolution color images.
Comparing to the regular color cameras, the depth map captured by Kinect v2 has low resolution, noises, occlusion regions and broken holes. To enhance the quality of depth maps, we propose two techniques. Large occlusion regions cannot be well compensated by using only the current frame. We thus construct the background depth map using multiple depth and color frames. The object edges of a depth map often contain noises. We apply a popular object segmentation technique to the color image and then correct the depth map using the object segmentation outputs. In this process, we need to separate the occlusion pixels from the other types of error pixels. At the end, we compare our high resolution depth maps with those produced by a previous method. The experimental results show that the quality of our high resolution depth maps is better.

並列關鍵字

high resolution depth map ； depth map ； depth background reconstruction ； hole filling ； Kinect v2

參考文獻

[12] The specification of SR4000, http://www.mesa-imaging.ch/products/sr4000/

[3] Y.-S. Kang and Y.-S. Ho, "Upsampling of Low-resolution Depth Map with Enhancing Depth Discontinuity Regions," Asia-Pacific Signal and Information Processing Association, Dec 2014.

[5] K. Xu, J. Zhou, and Z. Wang, "A Method of Hole-filling for the Depth Map Generated by Kinect with Moving Objects Detection," presented at the 2012 Ieee International Symposium on Broadband Multimedia Systems and Broadcasting (Bmsb), June 2012.

[7] N.-E. Yang, Y.-G. Kim, and R.-H. Park, "Depth hole filling using the depth distribution of neighboring regions of depth holes in the Kinect sensor," presented at 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC), Aug 2012.

[8] J. H. Cho, K. H. Lee, and K. Aizawa, "Enhancement of Depth Maps With Alpha Channel Estimation for 3-D Video," Ieee Journal of Selected Topics in Signal Processing, vol. 6, pp. 483-494, Sept 2012.