基於視覺利用乘積隱藏式馬可夫模型手語辨識

Vision-based Continuous Sign Language Recognition using Product Hidden Markov Models

余信翰(Shin-Han Yu)

指導教授 ： 黃仲陵

國立清華大學/電機資訊學院/電機工程學系/碩士(2010年)

https://doi.org/10.6843/NTHU.2010.00630

摘要

手語是聽障人士日常溝通的基本工具之一，基於此動機我們在此設計了一套手語辨識系統。在這篇研究論文中，我們以視覺為基礎下利用乘積隱藏式馬可夫模型來達到手語詞彙的辨識。由語言學構音(articulation)的研究，手語中的手勢是由：手的位置、手的型狀手的位置、手的型狀以及手的移動方向三種音素所組成的。這個系統分成四大部分；特徵擷取、模型訓練、句子切割以及辨識。首先在特徵擷取的部分是手語者佩戴不同顏色手套並且利用連續可適性平均值移 動(CamShift)演算法來達到手的追蹤，我們對雙手取7Hu 和型與主軸的夾角來描述手型的特徵，接著針對每一個手語詞彙都去訓練一組乘積隱藏是馬可夫模型。那在句子的切割，本篇論文提出兩層的連續手語句子切割，第一層利用手的位置對句子做粗略的切割，對於切割後每段區段都有其相關資訊；第二層則是利用手型的變化來針對每段區段作精細的手語詞彙邊界切割。最後在辨識方面，利用上述切割方法所得到的序列並將其對應的觀察值序列對以訓練好的乘積隱藏式馬可夫模型去計算其機率值，挑選出最高的機率模型，則將其選擇為被辨識的手勢。 在這個實驗中，我們挑選了40 個台灣手語詞彙來當作是我們的語料庫，相關影片由每個受測者拍攝並作為我們的實驗樣本。經過測試平均後，我們的系統可以獲得94.04%的字彙辨識率。在另一個實驗，我們收集三句台灣手語句子，每句平均由18~23 個手語字彙組成，平均偵測ME 的recall 為74.5%，precision 為89%.

關鍵字

手語辨識 ； 乘積馬可夫模型

並列摘要

In this thesis, we introduce a vision-based continuous sign language recognition to recognize sign language sentences in a simple background. The system consists of four modules: feature extraction, product hidden Markov model (PHMM) training, sign words recognition using the PHMMs. To allow real-time moving hand tracking and hand shape extraction, the signer wears gloves with different colors. CamShift algorithm is used to track the moving hands gesture in a video. We apply the 7Hu moment and orientation of major axis to characterize the hand shape. After having
extracted the feature, we train our system using PHMM. Then, we use the hand location to roughly segment the continuous sign. After rough segmentation, we apply the hand-shape-based segmentation to divide the CSR image sequence into image sub-sequences, and then use the trained PHMM to recognize the isolated sign word.
In the experiments, we choose 40 Taiwan Sign Language (TSL) sign words, and collect the sign language videos made by different signers. The experimental results demonstrate that our system achieves a good performance of sign-word recognition accuracy of 94.04%. In another experiments, we collect 3 TSL sentences which consist of 18~23 sign-words. The experimental results show that the average sign
spotting recall rate is 74.5% and precision rate is 89%.

並列關鍵字

sign language recognition ； product hidden markov model

參考文獻

[11] Richard O. Duda, Peter E. Hart, David G. Stork, “Pattern Classification,’

[23] William C. Stokoe. Sign Language Structure: An Outline of the visual

Large Vocabulary Continuous Sign Language Recognition,” Proc. Sixth

May 2004.

Architecture Classifier for Chinese Sign Language Recognition with Large