Examining whether Direct Observation of Procedural Skill (DOPS) is a Reliable Assessment Tool for Assessing the Performance of Procedural Skills

Yu-Ting Tsao ； Tien-Shang Huang ； Huey-Ling Chen ； Tzong-Shinn Chu ； Ming-Been Lee ； Yen-Yuan Chen

《Journal of Medical Education》 25卷3期 (2021/09) Pp. 103-111

https://doi.org/10.6145/jme.202109_25(3).0004

摘要

Purpose: The reliability of Direct Observation of Procedural Skills (DOPS) has been of concerns. For convincingly examining the reliabilities of DOPS, we built up three video-recorded clinical scenarios played by standardized patients/family members and standardized house officers. The objective of this study was to investigate the test-retest reliability, interrater reliability and internal consistency reliability of DOPS to assess house officers' capability of performing procedural skills. Methods: We developed Scenario I, Scenario II, and Scenario III for guaranteeing that all raters assessed exactly the same procedural skills performed by standardized house officers. We recruited the raters, using purposive sampling, with plentiful experiences of rating house officers' performance of procedural skills from one governmental university-affiliated, and three medical university-affiliated medical centers, and two Veterans Affairs Council-affiliated medical centers. Each rater was required: (1) to watch the video-recorded performance in a consecutive order starting from Scenario I in a single rating session; and (2) to rate each house officer's performance of procedural skills in each scenario. We examined the test-retest reliability, interrater reliability, and internal consistency reliability of DOPS using Pearson's correlation coefficients (PCC), intra-class correlation coefficient (ICC), and internal consistency reliability (Cronbach's alpha), respectively. Results: We found that: (1) a rater gave similar rating results in face of exactly the same procedural skill as indicated by acceptable or more than acceptable test-retest reliability (PCC = 0.75 ~ 0.81, P ＜ .01); (2) the raters gave the similar rating results to a house officer's performance of procedural skills as indicated by an excellent interrater reliability (PCC = 0.93 ~ 0.96, P ＜ .01); and (3) all the items of DOPS reflected uni-dimensionally a construct-a house officer's performance of procedural skills as supported by a much better internal consistency reliability (Cronbach's alpha = 0.84 ~ 0.91). Conclusions: Our study result convincingly reported that DOPS is a reliable assessment tool for assessing a house officer's performance of procedural skills, and can be widely used in clinical encounter to rate the performance of procedural skills.

關鍵字

Direct Observation of Procedural Skill ； reliability

參考文獻

Miller GE: The assessment of clinical skills/competence/performance. Acad Med 1990; 65(9 Suppl): S63-7.

Van Der Vleuten CPM: The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1996; 1:41-67.

Hengameh H, Afsaneh R, Morteza K, Hosein M, Marjan SM, Ebadi A: The Effect of Applying Direct Observation of Procedural Skills (DOPS) on Nursing Students' Clinical Skills: A Randomized Clinical Trial. Glob J Health Sci 2015; 7(7): 17-21.

Cobb KA, Brown G, Jaarsma D, Hammond RA: The educational impact of assessment: A comparison of DOPS and MCQs. Med Teach 2013; 35(11): e1598-e1607.

Bansal M: Introduction of Directly Observed Procedural Skills (DOPS) as a Part of Competency-Based Medical Education in Otorhinolaryngology. Indian J Otolaryngol Head Neck Surg 2019; 71(2): 161-166.