Using Generalizability Theory to Construct an Optimal Medical Student Selection Interview

Shih-Chieh Liao ； Yueh-Nu Hung ； Cheng-Chieh Lin ； Cheng-Chun Lee ； Chia-Der Lin

《Journal of Medical Education》 23卷1期 (2019/03) Pp. 62-74

https://doi.org/10.6145/jme.201903_23(1).0005

摘要

Objectives: Multiple Mini Interview (MMI) is a popular method for medical student selection, and medical schools design MMI based on their goals of medical education. Generalizability Theory (GT) is a psychometric method used to estimate the assessment reliability across sources of measurement error. This research relied on GT to investigate the relationships between the number of MMI station and measurement reliability and to find the important implication for optimal MMI design. Methods: Medical student selection interview data from the Medical School of China Medical University (MSCMU) in Taiwan from 2016 to 2017 were collected. The participants were 468 medical student candidates invited by MSCMU to do entrance interviews. Participants each completed six MMI stations and one group interview. The two reliability-like coefficients, Generalizability Coefficient (GC) and Index of Dependability (ID), were used as indications of MMI measurement reliabilities. Results: The mean scores and standard deviations (mean, S.D.) of MMI scores in 2016 and 2017 are (58.656, 9.432) and (69.193, 6.746), respectively. There are three major findings in this research. First, GC and ID can be predicted from the number of MMI station. Take 2017 for example, the GC for 6 stations is 0.614; when there are 10 MMI stations, GC is 0.726. Second, when the desired increase of GC or ID is determined, the number of additional MMI stations can be calculated. When GC is set to be equal to or higher than 0.700, then there need to be nine stations in 2016 and nine in 2017. Third, while GC and ID increase as MMI station number increases, the increase rate slows down gradually as the MMI station number continues to increase. Take 2016 for example, when the number of MMI station is increased from five to six, 11 to 12, 17 to 18 and 23 to 24, the increase of GC is 0.043, 0.016, 0.008 and 0.005, respectively. Conclusions: GT can help medical educators to decide on the required MMI station number based on the desired GC and ID, evaluate the relationship between measurement reliability increase and MMI station number increase and decide whether such increase is worthwhile. Medical schools can then plan an optimal number of MMI station in medical student selection interview.

關鍵字

generalizability theory ； assessment reliability ； multiple mini interview ； medical student selection interview

參考文獻

2. Turnwald GH, Spafford MM, Edwards JC. Veterinary school admission interviews, Part 1: literature overview. J Vet Med Educ. 2001;28(3):111-21.

3. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78(3):313-21.

4. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004;38(3):314-26.

5. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The relationship between interviewers' characteristics and ratings assigned during a multiple mini-interview. Acad Med. 2004;79(6):602-9.

6. Harris S, Owen C. Discerning quality: using the multiple mini‐interview in student selection for the Australian National University Medical School. Med Educ. 2007;41(3):234-41.