基於成對比較之多向度與多目標同儕評量系統

Peer evaluation system for multi-degree and multi-pair based on pairwise comparisons

周逢琦(Feng-Chi Chou)

指導教授 ： 葉丙成

國立臺灣大學/電機資訊學院/電信工程學研究所/碩士(2018年)

https://doi.org/10.6342/NTU201802968

摘要

近年來隨著科技的進步，行動裝置與網際網路的發達，以及數位學習的時代來臨，線上同儕回饋系統在課堂上的應用已逐漸成為趨勢，而傳統的教學模式，老師在台上教，學生在台下學，這樣單方面的學習模式，缺少批判性思考以及相互觀摩學習的機會，已逐漸無法滿足現代學生的需求，加入互評機制除了可以讓傳統的課堂學習提高豐富性，提高學生學習效率外，亦可以降低老師批改作業的負擔，在課後老師可以把心力擺在優化教學上，也可以幫助老師了解學生的學習狀態。
本論文致力於開發完整的線上成對比較互評系統，本研究訪問了許多在第一線教學的老師，根據這些回饋設計出新的成對比較演算法，不同於以往的給分制度互評系統，本系統採用成對比較的方式互評，互評時不要求互評者打一個準確的分數，而是以比較的方式決定何者在某個向度中表現較佳，經過互評者多次評比後，建立一個勝敗關係表，根據每個作品的勝敗紀錄轉換為實質分數，而在許多文獻中證實比較方式是比給分方式有著更客觀的評分結果，除此之外本系統在與許多老師訪談之後，本研究發現了成對比較時的盲點，缺乏向度比較以及互評時間過長，因此新增了向度系統，不再只是單純比較誰的作品較佳，增加了向度系統後學生互評成績與老師成績依然保持著高度正相關。在互評時間過長的問題中，提出了多目標互評方式，也就是一次讓學生觀看多組作品，並決定這些作品的名次，根據實驗結果，新的互評方式可以節省學生2.15倍的互評時間，且經過假設檢定驗證後，多目標互評方式的學生互評時間較短，顯著性達到非常顯著p=0.001，且與老師評定成績相比依然可以得到高度正相關，相關係數為0.723。
在論文的後半部分將本系統實際運用於課堂中，利用實驗設計與大量使用者的數據分析之後，證明本研究所提出的演算法符合現代老師們的需求，且可以用互評結果輔助老師找出評分時的盲點，並在之後以問卷形式訪問老師使用本系統後的想法。本研究的系統開發成果皆為開源程式碼，研究結果可作為未來成對比較系統的考量，提升成對比較的效率以及準確度。

關鍵字

同儕互評 ； 線上互評系統 ； 成對比較 ； 教育科技

並列摘要

Thanks to new technological advances, mobile device and network technology has gained popularity. Online peer feedback system has come to be one of the most recent trends. However traditional teaching model lack of critical thinking and learn from each other’s work. To solve this problem, peer grading is the one of best solution. It can enhance student’s willingness to learn, and it can reduce teacher’s burden of checking student’s homework. Teacher can have more time to optimize their teaching. It also allows teachers to better understand their students and know whether they really get the concept in the classroom.
In our work, we develop an online pairwise comparison peer grading system. We seek advice from lots of teachers who are teaching in the elementary school, junior high school, high school and the university to improve our peer grading system. Our system is different from the other peer grading system. We use Peer-Evaluations replace with Card Evaluations to peer grading. In the peer grading student will not be asked for giving an accurate score. They should choose the assessment who works better. After few times of comparison we can create a win-lose form. After that the system give all student a score according to this form. In many papers , it is confirmed that Peer-Evaluations has more objective result than Card Evaluations. In addition to this, we find a way to optimize pairwise comparison. It can reduce 2.15 times students spend in the peer grading. Besides, after significance testing we get p value equals 0.001. The peer grading results can get highly positive correlation with teacher’s rating. The correlation coefficient is 0.723.
We run our system in the classroom to collect user data and analyze the data. The result proves that our way actually improved the pairwise comparison performance. At the end we use questionnaire to conduct case in some teachers who uses our system. We find out that our system can be used to help the teacher evaluate the assignment more accurately and our works are open source on github.

並列關鍵字

Peer grading ； Online peer feedback system ； Pairwise comparison ； Education Technology

參考文獻

[1] N. B. Shah, J. K. Bradley, A. Parekh, M. Wainwright, and K. Ramchandran, "A case for ordinal peer-evaluation in MOOCs," in NIPS Workshop on Data Driven Education, 2013.

[2] P. M. Sadler and E. Good, "The impact of self-and peer-grading on student learning," Educational assessment, vol. 11, no. 1, pp. 1-31, 2006.

[3] C. Piech, J. Huang, Z. Chen, C. Do, A. Ng, and D. Koller, "Tuned models of peer assessment in MOOCs," arXiv preprint arXiv:1307.2579, 2013.

[4] 吕鹏, "基于神经网络的中国水墨画艺术风格学习," 天津: 天津大学, 2009.

[5] B. M. Moskal and J. A. Leydens, "Scoring rubric development: Validity and reliability," Practical assessment, research & evaluation, vol. 7, no. 10, pp. 71-81, 2000.