評分者信心與評分者內變異之相關研究

許純瑜

指導教授 ： 陳柏熹

國立臺灣師範大學/教育學院/教育心理與輔導學系/碩士(2011年)

摘要

本研究的目的是在探討評分者在評分過程中的評分信心程度與其評分結果變異程度的關係。研究方法的部分，本研究共分為兩個子研究，研究一是藉由模擬的方式以了解影響隨機效果多面向模式參數與評分者內變異數估計準確度的因素，作為研究二中資料分析的參考。研究二則是透過隨機效果多面向模式進行自編電腦化創造力測驗的實徵資料分析，並進一步探討評分者信心程度與評分者內變異數、受試者能力估計之間的相關。
研究一的結果顯示，每位評分者進行評分的作品份數、評分規準數以及評分者內變異分配情形均會影響隨機效果多面向模式中固定效果參數與隨機效果參數的估計準確性。當評分作品份數越多、評分規準個數增加，或評分者內變異較小的情況下，對於參數估計會較為準確；相反的，若評分作品份數太少、評分標準個數較少，或評分者變異較大的情況下，則參數估計的準確度會較差。每份作品評分人數的多寡則不會影響兩種參數估計的準確性。研究二的結果顯示，在創新性評分規準中，評分者自評信心分數與評分者內變異數兩者之間的相關未達顯著，然而將其中一位評分者的結果排除後，可以發現其餘六位評分者的信心分數與評分者內變異數大小呈正相關；實用性評分規準的部分，評分者的自評信心分數與評分者內變異數兩者的相關則未達顯著。針對以上結果，作者最後提出若干未來研究與實務建議。

關鍵字

評分者信心 ； 評分者內變異 ； 隨機效果多面向模式

並列摘要

The goal of the research was to explore the correlation of rater confidence and intra-rater variation. There are two studies in this research. In Study 1, simulations were conducted to examine the variables that might be the factors which influence precision of parameters estimation under random-effects facet model. The result of Study 1 was as reference to Study 2. Study 2 was an empirical study. The real data was analyzed with random-effect facets model and was examined the correlation between rater confidence and intra-rater variation.
The results of Study 1 indicated that rating numbers per rater, numbers of rating criteria and magnitudes of intra-rater variation affected the precision of parameters estimation through random-effects facet model. The parameters estimation was higher precision for the situation of more rating numbers per rater, more rating criteria numbers and small intra-rater variation. There was no difference on precision of parameters estimation between 2 and 4 raters. The results of Study 2 indicated that there was no significant correlation between rater confidence and intra-rater variation on creativity criteria. However, when we excluded the data of one rater, we found that there was positive correlation between rater confidence and intra-rater variation. There was also no significant correlation between rater confidence and intra-rater variation on utility criteria. According to the results of this research, the researcher proposed some opinions for future study and practice.

並列關鍵字

rater confidence ； intra-rater variation ； random-effects facet model

參考文獻

黃國禎、朱蕙君、王榕榆 (2008)。以答題信心度為基礎之線上診斷評量系統。師大學報：教育類，53，1-24。

Andrich, D. (1978). A rating formulation for ordered response categories. psychometrika, 43, 561-573.

Bands, C., & Murphy, K. (1985). Toward narrowing the research-practice gap in performance appraisal. Personnel Psychology, 38, 335-345.

Bernardin, H. J. & Orban, J. (1985). Leniency effect as a function of rating format, purpose for appraisal and individual differences. Presented at Annual meeting of the Academy of Management, Boston.

Bradlow, E. T., Wainer, H., & Wang, X. (1999). A bayesian random effects model for testlets. Psychometrika, 64(2), 153-168.