Exploring the Association Between USMLE Scores and ACGME Milestone Ratings: A Validity Study Using National Data From Emergency Medicine.
Accreditation
Adult
Clinical Competence
/ statistics & numerical data
Education, Medical, Graduate
/ statistics & numerical data
Educational Measurement
/ methods
Emergency Medicine
/ education
Female
Humans
Internship and Residency
/ statistics & numerical data
Licensure, Medical
Male
Middle Aged
Multilevel Analysis
Regression Analysis
Reproducibility of Results
United States
Young Adult
Journal
Academic medicine : journal of the Association of American Medical Colleges
ISSN: 1938-808X
Titre abrégé: Acad Med
Pays: United States
ID NLM: 8904605
Informations de publication
Date de publication:
01 09 2021
01 09 2021
Historique:
pubmed:
17
6
2021
medline:
8
9
2021
entrez:
16
6
2021
Statut:
ppublish
Résumé
The United States Medical Licensing Examination (USMLE) sequence and the Accreditation Council for Graduate Medical Education (ACGME) milestones represent 2 major components along the continuum of assessment from undergraduate through graduate medical education. This study examines associations between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME emergency medicine (EM) milestone ratings. In February 2019, subject matter experts (SMEs) provided judgments of expected associations for each combination of Step examination and EM subcompetency. The resulting sets of subcompetencies with expected strong and weak associations were selected for convergent and discriminant validity analysis, respectively. National-level data for 2013-2018 were provided; the final sample included 6,618 EM residents from 158 training programs. Empirical bivariate correlations between milestone ratings and Step scores were calculated, then those correlations were compared with the SMEs' judgments. Multilevel regression analyses were conducted on the selected subcompetencies, in which milestone ratings were the dependent variable, and Step 1 score, Step 2 CK score, and cohort year were independent variables. Regression results showed small but statistically significant positive relationships between Step 2 CK score and the subcompetencies (regression coefficients ranged from 0.02 [95% confidence interval (CI), 0.01-0.03] to 0.12 [95% CI, 0.11-0.13]; all P < .05), with the degree of association matching the SMEs' judgments for 7 of the 9 selected subcompetencies. For example, a 1 standard deviation increase in Step 2 CK score predicted a 0.12 increase in MK-01 milestone rating, when controlling for Step 1. Step 1 score showed a small statistically significant effect with only the MK-01 subcompetency (regression coefficient = 0.06 [95% CI, 0.05-0.07], P < .05). These results provide incremental validity evidence in support of Step 1 and Step 2 CK score and EM milestone rating uses.
Identifiants
pubmed: 34133345
doi: 10.1097/ACM.0000000000004207
pii: 00001888-202109000-00040
pmc: PMC8378430
doi:
Types de publication
Comparative Study
Evaluation Study
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1324-1331Informations de copyright
Copyright © 2021 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Association of American Medical Colleges.
Références
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(suppl 9):S63–S67.
ten Cate O, Carraccio C, Damodaran A, et al. Entrustment decision making: Extending Miller’s pyramid. Acad Med. 2021;96:199–204.
American Educational Research Association; American Psychological Association; National Council on Measurement in Education. The Standards for Educational and Psychological Testing. 2014.Washington, DC: American Educational Research Association;
Kane MT. An argument-based approach to validityPsychol Bull. 1992;112:527–535.
Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: A practical guide to Kane’s framework. Med Educ. 2015;49:560–575.
Messick S. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaningAm Psychol. 1995;50:741–749.
CampbelL DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull. 1959;56:81–105.
Beeson MS, Holmboe ES, Korte RC, et al. Initial validity analysis of the emergency medicine milestones. Acad Emerg Med. 2015;22:838–844.
Beeson MS, Hamstra SJ, Barton MA, et al. Straight line scoring by clinical competency committees using emergency medicine milestones. J Grad Med Educ. 2017;9:716–720.
Hamstra SJ, Yamazaki K, Barton MA, Santen SA, Beeson MS, Holmboe ES. A national study of longitudinal consistency in ACGME milestone ratings by clinical competency committees: Exploring an aspect of validity in the assessment of residents’ competence. Acad Med. 2019;94:1522–1531.
National Board of Medical Examiners. 2018 Directory: NBME & USMLE Committees & Volunteers. https://www.nbme.org/sites/default/files/2020-01/2018USMLE-NBME-Examination-Committee-Members.pdf . Accessed May 7, 2021.
Holmboe ES, Yamazaki K, Nasca TJ, Hamstra SJ. Using longitudinal milestones data and learning analytics to facilitate the professional development of residents: Early lessons from three specialties. Acad Med. 2020;95:97–103.
Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2002.2nd ed. Thousand Oaks, CA: Sage Publications;
Gelman A, Hill J. Data Analysis Using Regression and Multilevel/Hierarchical Models. 2007.New York, NY: Cambridge University Press;
United States Medical Licensing Examination. Summary Report and Preliminary Recommendations From the Invitational Conference on USMLE Scoring (InCUS), March 11-12, 2019. https://www.usmle.org/pdfs/incus/incus_summary_report.pdf . Accessed May 7, 2021.
Cuddy MM, Dillon GF, Clauser BE, et al. Assessing the validity of the USMLE Step 2 Clinical Knowledge examination through an evaluation of its clinical relevance. Acad Med. 2004;79(suppl 10):S43–S45.
Margolis MJ, Clauser BE, Winward M, Dillon GF. Validity evidence for USMLE examination cut scores: Results of a large-scale survey. Acad Med. 2010;85(suppl 10):S93–S97.
Norcini JJ, Boulet JR, Opalek A, Dauphinee WD. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89:1157–1162.
Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: A validity study of score inferences from a licensure examination. Acad Med. 2017;92:1780–1785.
Ames SE, Ponce BA, Marsh JL, Hamstra SJ. Orthopaedic surgery residency milestones: Initial formulation and future directions. J Am Acad Orthop Surg. 2020;28:e1–e8.
Conforti LN, Yaghmour NA, Hamstra SJ, et al. The effect and use of milestones in the assessment of neurological surgery residents and residency programs. J Surg Educ. 2018;75:147–155.
Edgar L, Roberts S, Yaghmour NA, et al. Competency crosswalk: A multispecialty review of the Accreditation Council for Graduate Medical Education milestones across four competency domains. Acad Med. 2018;93:1035–1041.
Edgar L, Roberts S, Holmboe E. Milestones 2.0: A step forward. J Grad Med Educ. 2018;10:367–369.
Mainous AG III, Fang B, Peterson LE. Competency assessment in family medicine residency: Observations, knowledge-based examinations, and advancement. J Grad Med Educ. 2017;9:730–734.
Holmboe ES, Yamazaki K, Edgar L, et al. Reflections on the first 2 years of milestone implementation. J Grad Med Educ. 2015;7:506–511.
Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051–1056.