一套使用視覺通透式頭戴顯示器為輔助的擴增實境鋼琴學習系統

PLSOH: An Augmented Reality Based Piano Learning System Using Optical See-Through Head Mounted Display

陳年麒(Nien-Chi Chen)

指導教授 ： 歐陽明

國立臺灣大學/電機資訊學院/資訊工程學研究所/碩士(2015年)

https://doi.org/10.6342/NTU.2015.00124

摘要

音樂學習對初學者來說是具有挑戰性的，尤其當學習的樂器較為複雜的時候，例如:鋼琴。對初學者來說，初期最困難的部分之一就是學習如何看懂樂譜。當樂譜上的記號複雜且讓人混淆時，初學者容易失去信心且容易忘記在課堂上學到的正確知識。
因此，我們提出另一種學習方式來克服這些令人沮喪的問題。由於擴增實境技術的進步，我們可以幫初學者創造一個易認知的學習環境。透過視覺通透式頭戴顯示器，初學者可以看到鋼琴上附加的額外虛擬輔助訊息。在系統中，包含了練習模式跟教學影片兩種模式可以選擇。
更重要的是，這個系統可以被應用在電子鋼琴和傳統鋼琴上。因為我們利用了深度攝影機來偵測琴鍵深度的變化，來判斷使用者是否正確地按下琴鍵。在系統中被特殊設計的輔助標記，不僅是拿來偵測，同時也可以增進使用者音樂的記憶性。
最後，小規模的使用者調查顯示這套系統可以讓沒有音樂背景的人，提高22%的進步。這個系統中的概念，希望在未來的音樂教育和娛樂上，能夠被使用到。

關鍵字

擴增實境 ； 虛擬實境 ； 視覺通透式頭戴顯示器 ； 深度影像 ； 音樂教育

並列摘要

Music learning for beginners is challenging, especially when learning an intricate instrument such as a piano. The most difficult part for beginners at its preliminary stage should be developing the reading ability of sheet music. Once the music score is complicated and confused, beginners may lose self-confidence easily and simply for-get the instructor’s comments from lessons that they have learned before.
Therefore, we propose an alternative way to overcome these frustrating problems. Thanks to the well-developed technology of augmented reality, we can create a perceptive learning environment for beginners. By using an optical see through head-mounted display, beginners can see the additional helpful virtual information superimposed on the piano keys. Moreover, there are two modes, such as practice and tutorial videos mode, for users to select by their choice.
Most important of all, our system can be applied to not only electronic piano but also traditional piano. The reason is that we use a depth camera to detect whether user presses the correct piano keys by discriminating the depth change of piano keys. The markers in the system is purposely designed for not only feature detection but also improve the memorability of user.
Overall, the small-scale user test shows that the PLSOH system can successfully assist beginners who have no prior musical knowledge with 22% accuracy improvement in average. The concept of the system is expected it will have future usage in music education and entertainment in the future.

並列關鍵字

Augmented Reality ； Virtual Reality ； Depth Image ； Optical See Through Head-mounted Display ； Music Education

參考文獻

[5] L. Housley, T. Lynch, R. Ramnath, and J. Ramanathan, “Implementation Con-siderations in Enabling Visually Impaired Musicians to Read Sheet Music Using a Tablet,” Computer Software and Applications Conference, Washington, DC, USA, pp. 678-683, 2013.

[8] M. Rahman, K. Mitobe, M. Suzuki, C. Takano, and N. Yoshimura, “Analysis of dexterous finger movement for piano education using motion capture system,” International Journal of Science and Technology Education Research Vol. 2(2), pp. 22-31, 2011.

[9] L. Raymaekers, K. Luyten, J. Vermeulen, K. Coninx, “Game of Tones: Learning to Play Songs on a Piano Using Projected Instructions and Games,” CHI 2014, Toronto, ON, Canada, pp. 411-414, 2014.

[1] I. Barakonyi, and D. Schmalstieg, “Augmented Reality Agents in the Develop-ment Pipeline of Computer Entertainment,” International conference on Enter-tainment Computing, Springer-Verlag Berlin, Heidelberg, pp. 345-356, 2005.

[2] J. Barzdins, R. Freivalds, and C. H. Smith, “Learning with Confidence,” Annual Symposium on Theoretical Aspects of Computer Science, Grenoble, France, pp. 22-24, 1996.