監控評分者效果的Yes/No Angoff標準設定法之效度檢核：以國中教育會考數學科為例

本研究為先前研究「監控評分者效果的Yes/No Angoff 標準設定法」之延伸，目的在於蒐集各種效度來源的資料，驗證前述標準設定法結果之效度，據以檢核該標準設定法是否適用於國中教育會考。先前的研究將評分者效果的監控整合在標準設定流程裡，運用試題反應理論的部分計分模式來分析學科專家在標準設定流程中對試題難度的判定資料，提出了前述的標準設定法。本研究則依據Kane（1994）以及Pitoniak（2003）建議的程序、內部、外部等多元效度證據的觀點，以2013年國中教育會考數學科測驗資料為例，蒐集不同來源的效度資料來評估此標準設定結果之效度。結果發現：(1)專家們同意本標準設定法之執行過程是合理的；(2)本標準設定法在方法內、專家內、專家間的結果具有高度一致性；(3)本標準設定法與群聚分析取向標準設定法之分類結果具有89.82%的一致性；(4)本標準設定法分類結果與考生的國中三年在校成績分類之相關達.75。以上這些結果指出本標準設定法的程序及產出具有理想的效度，可供相關測驗機構進行標準設定時的參考。

關鍵字

效度；國中教育會考；評分者效果／評分者效應；標準設定法

並列摘要

This study is an extension of the newly proposed ＂rater-effects-monitored Yes/No Angoff standard-setting method,＂ and the purpose of it is to collect validity evidence from various sources to verify the standard-setting results and to examine whether this method is applicable to the Taiwan Comprehensive Assessment Program for Junior High School Students (CAPJHSS). This method uses the partial credit model of the item response theory to analyze content experts' rating data generated in the standard-setting process to monitor the rater effects. Based on suggested procedure proposed by Kane (1994) and Pitoniak (2003), this study combines validity evidence collected from procedural, internal, and external sources to examine the 2013 CAPJHSS Math test data. The results reveal that: (a) content experts agree that the implementation process of this standard-setting method is reasonable, (b) the standard-setting results generated within-method, within-expert, and between-experts are consistent, (c) the consistency between the results from this method and the cluster-analysis-approach method is 89.82%, and (d) the correlation between the classification outcomes from this method and students school grades is .75. These results indicate that this standard-setting method possesses adequate validity, and could be considered by other testing organizations. Suggestions for future research are provided.