基於影像之深度資訊的手勢辨識方法研究

A Study on Image Depth-Based Hand Gesture Recognition

蘇俊麟(Jun-Lin Su)

指導教授 ： 丁英智

國立虎尾科技大學/電機資訊學院/電機工程系碩士班/碩士(2016年)

摘要

本論文提出運用KINECT 深度影像進行手勢辨識，採用由微軟公司所開發的KINECT感測器，利用上面所搭載的紅外線感測器進行手勢偵測識別，發展出簡單手勢便能有效地與系統產生互動。
在過去的手勢辨識當中，常見的方法是使用於RGB判別膚色在對手部進行切割與擷取，此方法容易受到光線的影響造成誤判。本論文採用KINECT上的紅外線感測器所讀取到的深度值來做辨識，深度圖有別以往的RGB影像，其原理是從紅外線發射器發射出紅外線，再利用紅外線接收器去接收打在物體上所反射的紅外線就能得到深度值，此方法不必考慮環境光線的問題，就能夠擷取出物體的輪廓。
在本論文所提出的手勢辨識方法中手部背景切割的部分，主要是透過KINECT提供的深度資訊以及PrimeSense NiTE的SDK，在空間中追蹤人體手部位置，同時與環境背景分離出手部，並且透過影像處理方法取得清楚的手部輪廓，就能根據手勢辨識方法來做特徵值的提取。
本研究辨識手勢的方法分成兩種，第一種為靜態的非連續時間性之手勢辨識，是利用Hu(Hu Moment, Hu)特徵之手勢辨識方法，其方法所提取的手勢特徵值具有平移、旋轉、大小尺度等的不變性，因此在靜態手勢的辨識中擁有不錯的穩定性，第二種方法是連續時間性之手勢辨識，我們將記錄手部在空間中的狀態，包括手部形狀及紀錄單位時間內手掌面積的變化狀況作為特徵值，並且結合隱藏馬可夫模型(Hidden Markov Model, HMM)演算法應用於手勢辨識，利用其隱藏式的馬可夫鏈來模擬出如語音訊號隨著時間改變的特性，套用於手勢連續動作的特徵值，但由於距離KINECT遠近對手部面積的變化會有影響，因此本論文提出用Type-2 Fuzzy來調整同樣手勢不同深度距離所造成不同面積特徵，利用Type-2 Fuzzy來減少特徵值之間誤差，讓KINECT在有效距離內，同樣手勢都能夠在不同距離讀取出相近的面積特徵值。
本論文所提出兩種手勢辨識方法，第一種運用Hu之靜態的非連續時間性手勢辨識，我們設定7種手勢動作，這7種手勢平均辨識率約為95%，第二種是運用HMM之動態的連續時間性手勢辨識，我們設定5種手勢動作，並且在5種不同距離進行手勢辨識，在辨識率上，近距離時的辨識率為100%，中距離則約為80%，但在遠距離只剩41.5%。本研究提出利用Type-2 Fuzzy改良深度特徵之手勢辨識，在辨識率上近距離約為99%，不過在中距離辨識率來到91%，而在遠距離更是來到81.2%，也因此證明本研究所提出的方法是有效的。

關鍵字

KINECT ； 手勢辨識 ； 手部追蹤 ； 手語辨識 ； 影像辨識 ； 影像處理

並列摘要

This thesis has proposed image depth-based hand gesture recognition. In this study using KINECT device by Microsoft is adopted for infrared sensor and using it develop hand gesture recognition system.
In the past, the skin color detection was one of the most common recognition methods for hand gesture. But, there were some situations which was difficult to recognize objects if only apply skin-color information, for example, when the hands and face overlapping, if there are some skin-color liked objects in the background, if there are other people behind the user , and so on. However, the depth map is using IR sensor by the KINECT. The emitter emits infrared light beams and the depth sensor reads the IR beams reflected back to the sensor. The reflected beams are converted into depth information measuring the distance between an object and the sensor. Therefore, we capture the depth map against lighting and complex backgrounds from KINECT.
In this paper, we had developed hand gesture recognition system using Microsoft KINECT. Through accessing the depth information was provided by KINECT and using PrimeSense NiTE API, we can easily track user's hand and segmentation the hand gesture on background.
In this study, we have proposed two methods of the hand gesture recognition. The first method was to use the Hu moments of non-continuous temporality for hand gesture recognition. The Hu moments method calculates the seven features. The first six descriptors encode a shape with invariance to translation, scale and rotation. The seventh descriptor ensures skew invariance, which enables to distinguish between mirrored images. The second method is to use Hidden Markov Model (HMM) of continuous temporality for hand gesture recognition. From the training model, we observed the probability of observable state and the probability of hidden state then we identified the optimal sequence of state with viterbi algorithm. In the second method, we add Type-2 fuzzy systems to improve distances influences that the hand has same area at different distances.
The experimental result shows that the proposed system can recognize the static state of hand gestures with an average recognition rate of 95%. The dynamic hand gestures on the 0.7m to 0.9m have average recognition rate of 100%. On the 0.9m to 1.0m, the dynamic hand gestures have average recognition rate of 80%, but on the 1.0m to 1.2m, the dynamic hand gestures remain average recognition rate of 40%. We propose using Type-2 fuzzy system of dynamic hand gestures recognition method. On the 0.7m to 0.9m, the method has average recognition rate of 99%. On the 0.9m to 1.0m, the method has average recognition rate of 91%. On the 1.0m to 1.2m, the method has average recognition rate of 80%. Experimental result shows that we propose the methods effectively.

並列關鍵字

KINECT ； Gesture recognition ； Sign language recognition ； Hand tracking ； Video recognition ； Video processing

參考文獻

[1] S. Zhao, L. Shen and W. Tan, 2010, “A method of dynamic hand gesture detection based on local background updates and skin color model”, 2010 International Conference on Computer Application and System Modeling (ICCASM), pp.657-660.

[2] R. S. Medeiros, J. Scharcanski and A. Wong, 2013, “Multi-scale stochastic color texture models for skin region segmentation and gesture detection”, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1-4.

[4] R. M. Gurav and P. K. Kadbe, 2015, “Real time finger tracking and contour detection for gesture recognition using OpenCV”, 2015 International Conference on Industrial Instrumentation and Control (ICIC), pp. 974-977.

[5] Q. Ding, J. Han, X. Zhao and Y. Chen, 2015, “Missing-Data Classification With the Extended Full-Dimensional Gaussian Mixture Model: Applications to EMG-Based Motion Recognition”, IEEE Transactions on Industrial Electronics, pp. 4994-5005.

[6] B. C. Wang, C. G. Yang and Q. Xie, 2012, “Human-machine interfaces based on EMG and KINECT applied to teleoperation of a mobile humanoid robot”, Intelligent Control and Automation (WCICA), pp. 3903-3908.