A Comparison of Pre-Knowledge Detection Procedures With Response Time Modeling

Jin Zhang

《International Journal of Intelligent Technologies and Applied Statistics》 14卷2期 (2021/06) Pp. 107-121

https://doi.org/10.6148/IJITAS.202106_14(2).0003

摘要

Purpose: This study investigated the effectiveness of two Bayesian procedures in identifying test-takers and items with response time (RT) patterns indicating item pre-knowledge. Design: Various proportions of test-takers with pre-knowledge and items affected by pre-knowledge were manipulated in simulation data to find out: (1) which of the two methods produced lower type-I error, (2) which of the two methods successfully detected items affected by pre-knowledge, and (3) which of the two methods successfully detected test-takers with pre-knowledge. Findings: The results show that the person-fit method could accurately detect items whose RTs are not lognormally distributed but had low detection rates to identify persons with item pre-knowledge. The postpredict method showed high type-I error rates but good detection power when testing for extreme RT patterns for persons. Practical implications: The analyses indicate that the postpredict method is more promising that the person-fit method in detecting test-takers with pre-knowledge of item content. The cross-validated residuals utilized by the postpredict method could effectively alleviate the influence of the biases when detecting RT patterns indicating pre-knowledge. Value: The results from this study not only give lights to which method is more effective in detecting possible item breach, but also provide information about when the detecting methods are most effective.

關鍵字

Pre-knowledge ； Bayesian ； Response time ； Person-fit