A National Conversation and Exploration


The Issue
Since its introduction in 1992, the United States Medical Licensing Examination (USMLE) has produced a numeric score in support of its primary purpose as an assessment for initial medical licensure. Throughout the history of USMLE there have been, and continue to be, secondary uses of USMLE numeric scores within undergraduate and graduate medical education.

Many stakeholders value the current practice of reporting a numeric score on USMLE examinations. Over time, however, increased emphasis on USMLE numeric scores as a major factor in residency screening and selection has led others to view this as a negative consequence. Conversely, many in the graduate medical education community value the USMLE numeric score as a common, objective measure of the knowledge and skills of all prospective residents (both US and international graduates).

The Conference
March 11-12, 2019
Philadelphia, PA


The USMLE parent organizations (The Federation of State Medical Boards and The National Board of Medical Examiners), along with the American Medical Association, the Association of American Medical Colleges, and the Educational Commission for Foreign Medical Graduates are convening an invitational conference of key stakeholders to discuss the complex issue of USMLE scores and their use by primary and secondary users within undergraduate and graduate medical education, as well as to explore possible changes to or recommendations (short and/or long-term) for USMLE scoring. While it is anticipated that the conference will focus on numeric score reporting vs. Pass/Fail score reporting, other score reporting options will be considered.

Representatives from a variety of organizations will participate in the conference, including stakeholders representing a broad spectrum of medical education (undergraduate and graduate), state medical boards, examinees, the house of medicine, and the public.

Continue to: Appreciating a Complex Issue »

Appreciating a Complex Issue


The United States Medical Licensing Examination (USMLE) is co-sponsored by the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME). Licensing authorities across the United States rely upon the USMLE to help evaluate the qualifications of individuals seeking an initial license to practice medicine. Three "Step" exams make up the USMLE: Step 1, Step 2 (consisting of the Step 2 Clinical Knowledge [Step 2 CK] exam and the Step 2 Clinical Skills [Step 2 CS] exam), and Step 3. Candidates who have passed all components of the USMLE may apply for a license to practice medicine in the United States and its territories. USMLE delivers approximately 140,000 test administrations annually. International medical students/graduates account for nearly 40% of these administrations.

Students enrolled in an LCME-accredited program typically take Step 1 after completing some of their medical school curricula, Step 2 prior to matching with a residency program, and Step 3 at some point during residency. Examinees receive numeric scores for Steps 1, 2 CK, and 3, which are computer-based examinations. Scores on these exams are considered passing if they are equal to or above the minimum passing scores set by the USMLE Management Committee. Step 2 CS uses standardized patients to test students and graduates on their ability to gather information from patients, perform physical examinations, and communicate their findings to patients and colleagues. Scores from this component are reported as Pass/Fail.

Primary Purpose
An examinee’s USMLE scores are reported to medical licensing authorities for their use in the decision to grant a provisional license to practice in a post-graduate training program and the decision to grant an initial license for the independent practice of medicine. This is the primary intended purpose of USMLE scores.

Secondary Purposes
Trends in medical education have increased examinee efforts to maximize their USMLE performance as demonstrated through the numeric score. These trends include limited residency training slots, particularly in certain specialties, as well as medical schools' adoption of pass/fail grading for foundational curricula. Because residency programs use USMLE scores as a means to screen and select applicants, examinees strive to obtain the highest possible scores.

This use of the score is considered a secondary use of USMLE scores. USMLE was not designed for these purposes and the program did not envision the markedly increased reliance on numeric scores in graduate medical education. Program directors acknowledge the limitations of using USMLE scores for residency selection, but justify the practice given USMLE’s position as a nationally standardized measure of knowledge and skills and as the only common metric by which to evaluate all applicants.

Medical Schools
This emphasis on maximizing USMLE scores has led medical school students to the perception of a "parallel curriculum" to the curriculum of their medical schools. Students engage in their school’s curriculum while simultaneously preparing for what is perceived to be "other material" important for achieving high scores on the USMLE assessments.

In addition, medical school faculty and staff have noted potentially negative effects of preparation for high-stakes testing on student well-being.

Prior Reviews of Scoring Issues
This is not the first time the USMLE program has considered the advantages and disadvantages of numeric score reporting. This topic was discussed and evaluated at the inception of the program (1992) and during a comprehensive review of USMLE (2007-2008). In both cases, the program elected to maintain numeric score reporting. The FSMB and the NBME are keenly aware of the ongoing controversy regarding numeric score reporting and see it as a symptom of the larger challenges in the transition from undergraduate medical education (UME) to graduate medical education (GME).

USMLE Numeric Scores – Pros and Cons »

USMLE Numeric Scores – Pros and Cons


PROS CONS
USMLE offers highly reliable, objective assessment of relevant competencies. A national standard. A passing score on USMLE demonstrates minimum competency. Pass/Fail reporting suffices for this.
Mitigates the reliability challenges of some medical school assessments Licensure requires only a Pass/Fail outcome
May stimulate student preparation more than a Pass/Fail outcome Focus on numeric scores negatively impacts student well-being
Incremental numeric USMLE performance correlates with other valued measures
e.g., specialty board certification, state board disciplinary actions, improved practice
Standardized test scores best predict other standardized test scores (vs. clinical performance)
Offers a "level playing field" for all examinees, including those from international and new or lesser known schools Taking focus off scores may create more diversity within various specialty programs
If no USMLE numeric score exists, and the demand for a national assessment remains, what fills the gap? Opportunity cost due to students’ heavy focus on maximizing USMLE scores (e.g., less time for research, volunteerism)

Deliverables »

Deliverables


It is the intent of the convening organizations to produce several deliverables as part of the work arising from the invitational conference. The first of these is this website, which is envisioned as a primary mechanism for providing updates and status reports on the conference and its outcomes.

Additionally, it is anticipated that the conference will culminate in a statement on the issue of USMLE numeric score reporting. This may take the form of a consensus statement on the part of attendees, a set of recommendation(s), and/or a white paper (informational or a proposal on the issue).

It is also anticipated that a formal paper intended for broader dissemination through peer-reviewed literature may develop from the conference.

Any recommendation(s) to revise existing USMLE policies on score reporting will be sent to the USMLE Composite Committee and FSMB and NBME governance for a final decision or action. We anticipate further details on deliverables and potential timelines to become available after the March conference

Timeline »

Timeline* - updated May 2019


Contact Information for Transcripts
December 2018 Invitations extended to leaders and subject matter experts from major stakeholder groups. These include but are not limited to state medical boards, graduate medical education program directors, student affairs deans, clerkship directors, medical students and residents, and the public. Representatives from organizations such as ACGME, NRMP, LCME, etc. have also been invited to participate.
February 2019 Attendees and organizations asked to share perspectives in advance of meeting
March 11-12, 2019 Invitational Conference on USMLE Scoring
June 2019 Update presented to AMA Council on Medical Education. Preliminary recommendation(s) will be posted on this webpage; 6 week period for open commentary.
July/August 2019 Development of final recommendation(s)
Fall 2019 Presentation of final recommendations pertaining to USMLE and broader system to CEOs and co-convening organizations' senior leadership, as well as USMLE governance. Completion and submission of manuscript. Post final outcome of any approved recommendations.
Winter 2019-2020 Begin executing next steps (end of 2019 and throughout 2020).

*This timeline is an approximation. The details (both in scope and complexity) of any recommendation(s) derived from InCUS may impact the projected timeline for approval.

Additional Key Literature »

Additional Key Literature


Overview

The Invitational Conference on USMLE Scoring (InCUS) will provide an opportunity for various stakeholders to review, discuss, and make recommendations about USMLE numeric scoring practices.

In order to facilitate an informed and productive dialogue, a literature review was conducted, focusing on research related to USMLE Step 1 and Step 2 CK score uses and interpretations. Here we provide summaries of pertinent studies published since 2008. The selected articles originate from diverse institutions, and employ varied datasets and methodological techniques.

In general, these summaries speak to various viewpoints and conclusions on the validity argument for Step 1 and Step 2 CK score uses and interpretations. The summaries provide an overview of the current state of available empirical knowledge about the pros and cons associated with contemporary uses of USMLE Step 1 and Step 2 CK scores that in turn will help foster a robust discussion and an empirically-grounded set of recommendations emerging from the InCUS conference.

The research summaries are organized into three thematic sections:

  • Section 1 focuses on studies that address uses and interpretations of USMLE Step 1 and Step 2 CK scores.
  • Section 2 highlights studies that examine relationships among USMLE Step 1 and Step 2 CK scores and measures of specialty-specific educational achievement.
  • Section 3 includes studies that explore uses and interpretations of USMLE Step 2 CS performance.

A fourth category lists a sampling of relevant Commentaries and Letters to the Editor related to USMLE Step 1 and Step 2 score uses and interpretations. The primary purpose for including this list is to acknowledge the perspectives of individuals impacted by Step 1 and Step 2 score uses and interpretations, and to provide examples of the types of conversations in which they are engaged. As these pieces are not empirically based, research summaries are not provided.

To view the research summaries, click on the link associated with the topic area in which you are interested. This will take you to a .pdf file that includes the summaries, as well as links to each of the study abstracts.

Links to Research Summaries and Perspective Pieces

Updates


USMLE Scoring: An Ongoing Conversation

In early March, an invitational conference of key stakeholders was convened to discuss the complex issue of USMLE scores and their primary and secondary uses within state licensing, as well as undergraduate and graduate medical education. The Invitational Conference on USMLE Scoring (InCUS) explored possible changes to USMLE score reporting as well as changes to the GME application and selection system in the United States.

The InCUS planning group is currently working through the information gathered during the conference and summarizing the conference proceedings. Draft recommendations for USMLE as well as the UME-GME transition system are planned for posting on the USMLE website in early June 2019. A period of public comment will follow and will run about six weeks. Any additional viewpoints received will then be considered, and may result in changes to the draft recommendations. The resulting recommendations will then be reviewed by the USMLE parent organizations, as well as the other relevant organizations in the house of medicine, in the ensuing months. This timeline is an approximation; the details of the scope and complexity of any recommendations will determine the final schedule.

USMLE’s parent organizations, NBME and the Federation of State Medical Boards, along with the American Medical Association, the Association of American Medical Colleges, and the Educational Commission for Foreign Medical Graduates make up the InCUS planning group.

As more information becomes available, this webpage will contain updates and opportunities to learn and participate in this important topic.

Timeline- updated May 2019

Post-conference statement

List of InCUS attendees

Episode 5: InCUS Experience and Upcoming Timeline

Transcript

In this episode of USMLE Connection, produced by NBME Communications, Mike Barone, MD, MPH, Vice President, Licensure for NBME, and Dave Johnson, Senior Vice President for Assessment Services at Federation of State Medical Boards, discuss InCUS Experience and Upcoming Timeline.

Episode 4: InCUS Learnings and Next Steps

Transcript

In this episode of USMLE Connection, produced by NBME Communications, Mike Barone, MD, MPH, Vice President, Licensure for NBME, and Dave Johnson, Senior Vice President for Assessment Services at Federation of State Medical Boards, discuss InCUS learnings and next steps.

Episode 3: Pros and Cons of Numeric Scoring Vs. Pass Fail

Transcript

In this episode of USMLE Connection, produced by NBME Communications, Mike Barone, MD, MPH, Vice President, Licensure for NBME, and Dave Johnson, Senior Vice President for Assessment Services at Federation of State Medical Boards, discuss the pros and cons of numeric scoring of the United States Medical Licensing Exam® (USMLE®), pass/fail scoring, and other options.

Episode 2: A High-Quality, Collaborative Approach to Developing the USMLE Program

Transcript

In this episode of USMLE Connection, produced by NBME Communications, Mike Barone, MD, MPH, Vice President, Licensure for NBME, and Dave Johnson, Senior Vice President for Assessment Services at Federation of State Medical Boards, discuss the collaborative approach to creating the United States Medical Licensing Exam® (USMLE®).

Episode 1: How the USMLE Can Best Serve State Licensing Boards, Students, and Other Stakeholders

Transcript

In this episode of USMLE Connection, produced by NBME Communications, Mike Barone, MD, MPH, Vice President, Licensure for NBME, and Dave Johnson, Senior Vice President for Assessment Services at Federation of State Medical Boards, discuss the United States Medical Licensing Exam® (USMLE®) program and their thoughts on the Invitational Conference on USMLE Scoring (InCUS).