多層面Rasch模式於學術研討會論文評分之應用

A Multi-Facet Rasch Analysis on Rating the Academic Scientific Papers

張新立(Hsin-Li Chang) ； 吳舜丞(Shun-Cheng Wu)

《測驗學刊》 55卷1期 (2008/04) Pp. 105-128

https://doi.org/10.7108/PT.200804.0105

摘要

論文評審之表面得分實受論文之表現、評分項目之難度及評審之嚴厲度等因素所左右。本研究嘗試利用多層面Rasch之量測分析，透過已設計好之論文評審項目的考驗，且分離評審者嚴厲度之影響效果後，能從論文評審之原始順位尺度得分中，將論文表現之優劣以等距尺度量測出，且符合後續比較分析所需之統計推論特性。本研究利用年中華民國運輸學會第二十屆學術論文研討會之論文評審資料進行實證研究，其結果顯示「具有理論與實務貢獻」以及「結論是否正確且充分獲得研究結果之支持」兩項為受評論文最感難以達成之評審項目，而進一步之評審嚴厲度分析更發現不同學術分組評審委員之平均評審嚴厲度確實存在差異，更加突顯以原始評審得分作為評定論文表現優劣之潛在缺失。透過多層面Rasch模型之分析，除了可以公平估算研究論文之表現外，更可透過所估計得之評審項目難度，指引相關領域人員在學術創作上之努力方向，建議學術論文研討會之主辦單位應固定辦理並公布資訊以供各界參考。

關鍵字

多層面Rasch模型 ； 論文評分 ； 評審項目難度 ； 評審嚴厲度

並列摘要

Rating scores for the academic papers submitted to the conference are determined by three facets; namely, paper performance, evaluation item difficulty, and rater severity. For fairly rating the performance of papers, this study applies the Multi-Facets Rasch Model (MFRM) to disclose the information behind the original ordinal scores. The MFRM helps to identify the influence coming from each facet. For the estimates of all facets are mapped into an interval scale measures from the original ordinal scores, these values can thus be compared scientifically. An empirical study was hold by examining the original scores rated via 8 items on the 1 93 papers submitted to the 20th Conference of Chinese Institute of Transportation Annual Meetings, 2005. It is showed that ”Providing the academic or practical contribution” and ”Sufficiency and accuracy of the conclusion” are the two items with the highest difficulties. The papers belonging to some transportation fields are found to have better performance than the others. The well-fit in rater severity indicates the rating work of these 131 invited raters is reliable and convincible. For the MFRM can provide a scientific assessing approach which unveils the information of these three facets behind the original ordinal rating scores, the MFRM is suggested as a reliable approach to assess the performance of submitted academic papers.

並列關鍵字

Item difficulty ； Multi-Facet Rasch Model ； Rating scores ； Rater severity

參考文獻

王文中(2004)。Rasch測量模式與其在教育與心理之應用。教育與心理研究。27(4)，637-694。

余民寧(1992)。試題反應理論的介紹（二）－基本概念和假設。研習資訊。9(1)，5-9。

姚漢禱()。,未出版。

曾盟堡(2002)。是誰評判不公。測驗統計年刊。10，121-133。

Andrich, D.(1978).A rating formulation for ordered response categories.Psychometrika.43(4),561-573.