科學探究能力評量之標準設定與其效度檢核

Validating the Standard Setting on Multimedia-based Assessment of Scientific Inquiry Abilities

林小慧(Hsiao-Hui Lin) ； 吳心楷(Hsin-Kai Wu)

《教育心理學報》 50卷3期 (2019/03) Pp. 473-502

https://doi.org/10.6251/BEP.201903_50(3).0005

摘要

本研究係以臺灣大台北地區605位11年級學生接受科學探究能力評量施測所蒐集的實徵資料，以達到兩項研究目的。其一為依據待加強、基礎、精熟三個等級之標準表現描述，設定科學探究能力評量之標準，其二則從內部、過程及外部等多元效度證據來源，檢核Bookmark法進行科學探究能力標準設定的適切性及有效性。研究結果顯示，本研究科學探究能力的標準設定可獲得過程效度證據的支持。其次，內部效度評估結果顯示，14位標準設定成員在第一輪到第二輪之各表現等級的標準誤均在可接受範圍（SE ＜ 0.12），表示成員內標準設定結果檢具可靠性。另以二輪決斷分數中位數之樣本平均數的標準誤評估標準設定方法內的一致性，結果顯示各表現等級的標準誤均在可接受範圍（SE ＜ 0.12），表示標準設定方法內的結果相當一致。再者，以獨立樣本t檢定進行標準設定成員間一致性的考驗，分析結果顯示不同群組成員所設定的決斷分數均未達顯著差異。此外，標準設定極端值的監控結果發現，僅有少數極端值出現，故而對於整體決斷分數的影響甚微。因此，本研究科學探究能力標準設定可獲得內部效度證據的支持。最後，本研究以群聚分析標準設定，透過探討Bookmark法所得決斷分數之輻合效度，結果顯示二種標準設定法將學生分為三個表現等級之相關係數達顯著水準，表示在判斷表現等級有相當程度的一致性。另利用區別分析檢核標準設定的一致性，分析結果顯示，Bookmark法在「觀察與定題」、「計畫與執行」、「分析與發現」及「推理與論證」整體分類一致性依序為79.50%、86.00%、100.00%、89.90%，可見Bookmark標準設定法所得出的決斷分數在各表現等級分類之區別力相當高，可獲得外部效度證據的支持。綜合以上證據，研究結果顯示經由Bookmark法所設定之科學探究能力標準適切而且有效。

關鍵字

科學探究能力 ； 效度驗證 ； 標準設定 ； Bookmark標準設定法

並列摘要

This study developed a standard setting for Grade 11 of the Multimedia-based Assessment of Scientific Inquiry Abilities (MASIA) based on three levels of standard performance descriptions: below basic, basic, and proficient. The study also used a bookmark to identify the cut-off scores. Furthermore, the study discussed the correct degree of the MASIA standard, which depends on multiple levels of evidence, namely procedural, internal, and external evidence for validity. First, the result of the procedural evaluation for validity showed that the standard setting of scientific inquiry abilities adopted in this study is supported by the procedural evidence for validity. Second, the result of the internal evaluation for validity showed that the standard error of each performance level that participants reached in rounds one and two were within an acceptable range (standard error [SE] ＜ 0.12), thus indicating good intra-rater consistency. The consistency within the standard-setting method was evaluated using the standard error of the sample mean based on the median of the cut-off scores from round two. The result showed that the standard error of every performance level was within an acceptable range (SE ＜ 0.12), thus denoting high consistency within the results of the standard-setting method. Third, the inter-rater consistency of the standard setting was examined using an independent sample t test, and the results showed that none the cut-off scores set by the participants of different groups reached statistical significance. Therefore, the standard setting of scientific inquiry abilities can be supported by internal evidence for procedural validity. Finally, this study treated the quasi-setting results derived from the cluster analysis as convergent validity-based evidence to assess external validity. The results showed that the correlation coefficient of the three performance levels of the students differentiated by two standard-setting methods reached statistical significance, thus indicating that those who judged the performance levels of the students had a certain degree of consistency. Moreover, a discriminant analysis was conducted to determine the consistency of the standard settings, and the results revealed that the consistency of the factors classified into ＂observing and questioning,＂ ＂planning and experimenting,＂ ＂analyzing and concluding,＂ and ＂reasoning and arguing＂ were in the following sequence: 79.50%, 86.00%, 100.00%, and 89.90%. In the present study, the cut-off scores obtained by the bookmark presented high discrimination for each performance level category and can be supported using external evidence for validity. These results suggest that the bookmark standard setting of scientific inquiry abilities are appropriate and effective.

並列關鍵字

Bookmark ； Scientific inquiry abilities ； Standard setting ； Validation

參考文獻

吳清山（2014）：標準參照測驗。教育資料與研究，113，205-206。DOI：10.6724/ERR.201405_(113). 0007。[Wu, C. S. (2014). Standard reference test. Educational Resources and Research, 113, 205-206. DOI: 10.6724/ERR.201405_(113).0007]

林小慧、林世華、吳心楷（2018）：科學能力的建構反應評量之發展與信效度分析：以自然科光學為例。教育科學研究期刊，63（1），173-205。DOI：10.6209/JORIES.2018.63(1).06。[Lin, H. H., Lin, S. H., & Wu, H. K. (2018). Developing and validating a constructed-response assessment of Sientific abilities: A case of the optics unit. Journal of Research in Education Sciences, 63(1), 173-205. DOI: 10.6209/JORIES.2018.63(1).06]

陳慧娟（2015）：「師生共同增能」與「學生增能」教學實驗方案促進偏遠地區國中學生知識信念，自我調整策略與科學學習成就之比較研究。教育科學研究期刊，60（4），21-53。DOI：10.6209/JORIES.2015.60(4).02。[Chen, H. J. (2015). Effects of empowerment programs in remote junior high schools on scientific epistemological beliefs, self-regulation strategies, and academic achievement in science. Journal of Research in Education Sciences, 60(4), 21-53. DOI: 10.6209/JORIES.2015.60(4).02]

謝名娟、謝進昌、林世華（2013）：不同方法設定英文科決斷分數之實務性研究。測驗學刊，60（3），519-544。DOI：10.7108/PT.201212.0513。[Hsieh, M. C., Hsieh, J. C., & Lin, S. H (2013). The comparison of two standard setting method on an English test. Psychological Testing, 60(3), 519-544. DOI: 10.7108/PT.201212.0513]

吳宜芳、鄒慧英、林娟如（2010）：標準設定效度驗證之探究：以大型數學學習成就評量為例。測驗學刊，57（1），1-27。DOI：10.7108/PT.201003.0001。[Wu, Y. F., Tzou, H., & Lin, J. R. (2010). Validating the performance standards for cut scores in a large-scale mathematics assessment. Psychological Testing, 57(1), 1-27. DOI: 10.7108/PT.201003.0001]