基於彩色及深度影像之台灣手語互動學習系統

手語是聽障朋友互相進行溝通的工具，但一般社會大眾對於手語的認知不多，造成雙方面溝通無法有效進行。傳統的手語學習媒介包括書籍、影片、實際上課等、但這些學習方式都有其限制，無法滿足每一個人的需求；本論文基於此動機，希望開發一套手語互動學習系統，讓學習手語的人可以透過輔助裝置，演練手語動作以加深學習效果。本論文使用 Microsoft 之 Kinect 深度攝影機進行手部資訊擷取，再進行手部位置、手型、手的移動軌跡等手語重要辨別資訊的判讀。我們透過 Kinect 骨架點之相對位置來判斷手部當前位置。手型辨識則透過深度影像進行彩色影像的前景背景分離，鎖定手部 ROI 區塊進行膚色偵測並進行平滑及型態學處理去除雜訊，得到手部影像後透過傅立葉描述子進行手型特徵描述，再交由 SVM 分類器進行手型特徵分類。手部軌跡則是透過移動角度進行八方向量化，將軌跡量化序列使用HMM 模型訓練，完成手部移動軌跡辨識。實驗結果顯示，本系統在手語單字動作之辨識準確率達 88.9%；我們也設計了手語互動學習系統的操作介面，透過手語動作的單字辨識達成手語互動學習的目的。

關鍵字

手語；手勢辨識； Kinect ；隱馬可夫模型；支持量化機；傅立葉描述子

並列摘要

Sign language is the major communication medium for hearing-impaired people.However, most people are unfamiliar with sign language. Traditionally, sign language can be learned using books, videos, or by taking a course, but all of the mentioned approaches have some restrictions on fulfilling everyone's requirement.This study intends to develop an interactive learning system for sign language. Sign language learners can master the sign language by practice with the aided equipments.In this thesis the hand information is retrieved by Microsoft Kinect sensors.The hand location, hand shape, and hand trajectory, which are essential to sign language recognition, are then identified.The hand location is recognized from the relative positions of Kinect skeleton points. The depth image is used to remove the background of color image and the hand ROI (region of interest) is obtained using skin color detection.Smooth filtering and morphological techniques are adopted to reduce image noise and the Fourier descriptors are used to describe the hand shape features. Consequently, the hand shape is recognized using SVM classification. On the other hand, the hand trajectory is quantized into eight orientations and recognized using HMM. Experimental results show that the proposed method reaches 88.9% recognition rate for single words expressed by gestures.A user-friendly interface is also designed for the proposed interactive sign language learning system. Interactive sign language learning can be efficiently achieved by gesture recognition through this system.

並列關鍵字

Sign Language ； Gesture Recognition ； Kinect ； Hidden Markov Model ； Support Vector Machine ； Fourier Descriptor

參考文獻

[2] C. Shan, Y. Wei, T. Tan, and F. Ojardias,“Real time hand tracking by combining particle filtering and mean shift,”

Proceedings.Sixth IEEE International Conference on Automatic Face and Gesture Recognition,pp. 669-674, 2004.

[3] C. Manresa, J. Varona, R. Mas, and F. Perales,“Hand tracking and gesture recognition for human-computer interaction,”Electronic Letters on Computer Vision and Image Analysis,vol. 5, pp. 96-104,2005.

[4] R. Y. Wang and J. Popović,“Real-time hand-tracking with a color glove,”ACM Transactions on Graphics (TOG), vol. 28, p. 63, 2009.

[6] Z. Ren, J. Yuan, J. Meng, and Z. Zhang,“Robust part-based hand gesture recognition using kinect sensor,”

國際替代計量

基於彩色及深度影像之台灣手語互動學習系統

全文下載

主題瀏覽