可處理隨機輸出的自動程式作業批改系統

An Automatic Assessment System for Marking Programming Exercises with Random Output

吳育姿(Yu-Tzu Wu)

指導教授 ： 謝尚賢

國立臺灣大學/工學院/土木工程學研究所/碩士(2017年)

https://doi.org/10.6342/NTU201701661

摘要

在算程式程用計機課採自動化批改作業工具既給予學生即時的反饋也降低教學人員評分的負擔。然而，當題目的範例輸出包含隨機資訊時，批改系統的制式流程常無法判定諸如此類的題目，使得佈署一道新題目超越了圖形介面的操作，導致修改系統原始碼變成週期性活動。

本研究在此實作一個線上程式批改系統，有別於演算法競賽型解題系統的架構，將每道題目的批改邏輯抽離系統，獨立實作於批改腳本，使得系統成為能夠容納任意批改規範的平台，以應付各種要求隨機輸出的練習題。批改腳本主要功能為：設定此題是否有測試資料(2個選項)、調整程式輸出要如何正規化(2個選項)、選擇此題的呈現方式(3個選項)，共12個面向以涵蓋所有需要文字輸出的練習題。

針對程式初學課程常見的經典練習題，本研究提供教學人員一套簡便的批改腳本客製化流程包含：首先，分辨題目是否需要測試資料；接著，依據輸出內容是否具有圖形意義(例如：印出兩個矩陣相加、印出金字塔)調整正規化程度；最後，將僅須透過比對學生程式輸出與解答程式輸出是否一致就能達成批改的題目歸納為「標準輸出比對題型」，而將輸出具有隨機內容(例如：當下時間、隨機數字、機率)的題目歸納為「分支輸出比對題型」與「後處理輸出比對題型」。歷經一個學期的測試，透過此流程所設計的批改腳本確實能夠批改所有程式作業，其中著名競賽式解題系統無法佈署而只有本研究能夠處理的題目佔45.95%。

受益於系統保留了所有學生繳交的程式碼，批改系統除了確實大幅降低了評分的工作量，也處理了偵測作業抄襲的工作，使得教學人員有更多的時間解答學生們的疑惑。而以一個課程輔助方案的角度，批改系統除了能即時地給予學生反饋與允許重複提交修正後的程式碼，亦提供教學人員一個平台在課後檢視學生們的程式碼品質與繳交情形，因而更能夠適時地補充或增強教學內容。

最後，對於系統後續的發展潛力，本研究分析了學生們留在系統的紀錄資料與編程能力的相關性，發現大量的編譯錯誤次數為後段學生的共通點(Max. |r|=0.52; p<0.01)，未來可以將每道練習題的編譯錯誤次數當作特徵值，以排定早期需要加強輔導的學生名單。

關鍵字

線上批改系統 ； 客製化 ； 腳本 ； 程式練習題

並列摘要

Implementing automatic assessment tools on laboratory or homework exercises in programming courses is a way to provide timely feedback for students as well as lower the instructors’ marking burden. However, if the exercises involve random outputs, which are not judged against a gold standard of comparison, system-level source code modification is often required on a ubiquitous online judging system.

This research develops an online judging system as a platform, aiming at adopting a variety of assessment specifications. This research also introduces a simple procedure for customizing judge scripts as the core of the online judging system and deals with all textual output problems in 12 aspects: two options for determining whether a problem requires input of test data; three options for classifying a problem with or without random outputs; and two options for matching special output symbols. By designing the judge scripts based on theses 12 aspects, this approach was an excellent solution that dealt with almost half (45.95%) of the exercises that other online judging systems could not handle.

With the advantage of preserving submitted source codes, the online judging system not only replaces instructors’ grading work but also enables plagiarism detection for each exercise. Thus, the online judging system frees instructors’ from tedious routine tasks, enabling them to offer further assistance to students in need. During the experiment in a programming course, students could successively correct and re-submit their source codes until their codes passed the test, while the instructors could inspect quality of students’ source codes after class for improving pedagogy.

To investigate the future potential of the online judging system, this research analyzed correlations between the students’ performance and data collected from the system. It was found that making a large number of repeated compiling errors is one of the main behaviors of the weak students (Max. |r|=0.52; p<0.01). Therefore, this research proposes that the instructor may set a compiling
error threshold to identify the weak students at an early stage so that extra tutoring can be arranged.

並列關鍵字

Online Judge ； Automatic Assessment ； Script ； Customization ； Programming Assignments

參考文獻

[2] Ahadi, A., R. Lister, & A. Vihavainen (2016). “On the Number of Attempts Students Made on Some Online Programming Exercises During Semester and their Subsequent Performance on Final Exam Questions.” In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16), Arequipa, Peru, 218-223.

[4] Ala-Mutka, K. M. (2005). “A Survey of Automated Assessment Approaches for Programming Assignments.“ Computer Science Education, 15(2), 83-102.

[5] Albuquerque, D., B. Cafeo, A. Garcia, S. Barbosa, S. Abrahão, & A. Ribeiro (2015), “Quantifying Usability of Domain-Specific Languages: an Empirical Study on Software Maintenance.” Journal of Systems and Software, 101, 245-259.

[11] Constable, R. L. (1980). “The Role of Finite Automata in the Development of Modern Computing Theory.” In Proceedings of The Kleene Symposium, North-Holland Publishing Co., 61-83.

[15] Friedl, J. E. F. (2006). Mastering Regular Expressions (3rd ed.). O'Reilly Media, Inc.