Analyzing the Grammar Feedback of Two Automated Writing Evaluation Systems: My Access and Criterion

分析兩套全自動英文文章批改系統的文法回饋

陳浩然(Howard Hao-Jan Chen) ； 邱作麟(Stan Tsuo-Lin Chiu) ； 廖珮如(Pearl Liao)

《英語教學期刊》 33卷2期 (2009/09) Pp. 1-43

https://doi.org/10.6330/ETL.2009.33.2.01

摘要

英語為第二語言的學生須要加強寫作能力以通過各式語言能力測驗。為了減輕老師的負擔與提供學生立即的回饋，市場上已有幾個商業化的寫作自動評量系統可供選擇。雖然這些創新系統很受歡迎，但針對系統提供的回饋進行檢測的實證研究卻很少。由於越來越多人使用這些系統，瞭解系統優缺點的需求也與日俱增。本文將檢驗兩個著名的自動寫作評量系統Vantage My Access與ETS Criterion所提供的文法回饋。本文主要目的在於發現此二系統文法回饋的優點與缺點。首先，我們先收集此二系統針對269篇學生文章所提供的文法錯誤回饋並加以分析。結果顯示此二系統提供各約30種不同的回饋訊息。然而，My Access提供了過多的錯誤回饋。Criterion的表現較好，但它仍然無法偵測一些學生的常見錯誤，像是詞序、語氣、時態、搭配詞、連接詞、選字、與代名詞等。很明顯的此二系統對於英語為第二語言的學生無法提供令人滿意的協助。欲使用此類寫作自動評量系統的老師與學生須留意它們的不足之處與可能遇到的問題。

關鍵字

英語為第二語言 ； 文法錯誤 ； 自動回饋 ； 軟體評鑑

並列摘要

ESL/EFL students need to enhance writing skills to pass various language tests. To help teachers reduce their teaching load and to give students faster feedback, several commercial AWE (automated writing evaluation) tools are available. Although these innovative tools have become popular, few empirical studies have examined the quality of feedback generated by these systems. There is a pressing need to better understand the strengths and weaknesses of these widely used AWE systems. In this paper, we examined the grammar feedback provided by two well-known systems, Vantage My Access and ETS Criterion. The purpose of this paper is to uncover the strengths and weaknesses of the feedback provided by these two systems. Automated feedback messages on 269 student essays were collected and analyzed. The results showed that these two systems can provide about 30 different types of feedback messages. However, My Access provided many false alarm messages. Criterion performed much better than My Access, but it still failed to deal with several common errors related to word order, modals, tenses, collocations, conjuncts, word choice, and pronouns. It is clear that the two AWE systems do not provide satisfactory help to ESL students. Teachers and students who use these new tools should be aware of these problems and limitations.

並列關鍵字

ESL writing ； grammar errors ； automated feedback ； software evaluation

參考文獻

Burstein, J.,Chodorow, M.,Leacock, C.(2003).Criterion^SM Online essay evaluation: An application for automated evaluation of student essays.Proceedings of the Fifteenth Annual Conference on Innovative Application of Artificial Intelligence.(Proceedings of the Fifteenth Annual Conference on Innovative Application of Artificial Intelligence).:

Burstein, J.,Marcu, D.(2003).A machine learning approach for identification of thesis and conclusion statements in student essays.Computers and the Humanities.37(4),455-467.

Chen, C-F.,Cheng, W-Y.(2006).The use of a computer-based writing program: Facilitation or frustration?.Paper presented at the 23rd International Conference on English Teaching and Learning in the R.O.C. May 27, 2006.(Paper presented at the 23rd International Conference on English Teaching and Learning in the R.O.C. May 27, 2006).:

Chen, H. J.(2006).Proceedings of Tamkang University Conference on Second Language Writing.Taipei:Tamkang University.

Ferris, D.(1999).The case for grammar correction in L2 writing classes: A response to Truscott.Journal of Second Language Writing.8(1),1-11.