Quarterly News & Notes: Winter
2024
In this edition of News & Notes …
- The Longitudinal Knowledge Assessment (LKA®): Enrollment now open in 17 specialties
- How does ABIM create assessment questions?
- Has your practice changed? You can deactivate your certificate in that discipline.
- Study finds new Milestones Rating System reduces but does not eliminate bias in residency knowledge ratings
- How the physician community informs ABIM’s programs
- Research Roundup
The Longitudinal Knowledge Assessment (LKA®): Enrollment now open in 17 specialties
Physicians are using the LKA to meet their Maintenance of Certification (MOC) assessment requirement for more than 65,000 certificates—a number that is growing every day—making it the most popular MOC assessment option at a rate of about four to one over the traditional, 10-year MOC exam. Physicians say they like the flexibility and immediate feedback the LKA provides, and spend about an hour per quarter responding to questions on average. While the traditional, 10-year exam is still available to those who prefer it, physicians using the LKA said the questions and references in the LKA have helped them refresh their medical knowledge and put what they learned from the assessment into practice. ABIM will continue to work with physicians to address some constructive feedback we have heard, and look forward to further enhancing the diplomate experience with LKA.
How the LKA works
The LKA is a 5-year cycle during which 30 questions are answered quarterly and regular feedback is provided on how physicians are performing. As long as the LKA Participation Requirement and any other MOC requirements are being met, participants will continue to be publicly reported as Certified for the entire 5-year LKA cycle.
If you are due for an assessment and have been considering the LKA, enrollment is open until 6/30/24; however, the first quarter’s questions are available now through 3/31/24, so if you’re considering enrolling, it’s a good idea to do so as soon as possible to allow for more time to answer questions.
New in 2024
Physicians who work primarily in an inpatient setting may be interested in the new Internal Medicine: Inpatient assessment that launched in 2024 in both the LKA and traditional, 10-year MOC exam formats. If you are already enrolled in the general Internal Medicine LKA you can switch to the inpatient-focused track until 6/30/24.
We encourage you to see what other physicians are saying about the LKA. You can also learn more about the Internal Medicine: Inpatient assessment, and enroll or find out what MOC requirements you may have coming up by signing in to your Physician Portal.
How does ABIM create assessment questions?
ABIM launched the Longitudinal Knowledge Assessment (LKA®) in 2022 in response to feedback from the physician community asking for more flexible and convenient ways to maintain their certification.
As the LKA includes a greater number of questions than the traditional, 10-year MOC exam, ABIM established the Item-Writing Task Forces to meet the demand for assessment content. Each year ABIM develops 200 new questions in each specialty (equating to 3,200 new items across all disciplines annually).
The Item-Writing Task Forces are composed entirely of physicians in each specialty. They develop exam content through innovative, science-based approaches to item development. Any physician can apply to be an item-writer in their specialty area. Each assessment has a blueprint that outlines the content of the exam. Item-Writing Task Force members are assigned a different blueprint topic to cover and each is responsible for developing about 20 questions annually.
For each question, the writer first develops a “testing point” that addresses what a physician should know on that topic without having to consult medical resources or references. The testing point is like a map that links the topic to an area of the blueprint so that the test will accurately cover what the blueprint requires. An example of a testing point is: “Know to include cystic fibrosis in the differential diagnosis of an adult patient who has chronic cough and bronchiectasis.”
Item-writers then carefully craft well-researched, multiple-choice questions (MCQs) with a single best answer. ABIM uses MCQs because research has shown that scores obtained with MCQs are correlated with superior clinical performance and are particularly suitable for simulating clinical decision-making. The overwhelming majority of ABIM exam questions use a clinical stem (patient-based case scenario) format that assesses the higher-order cognitive abilities required for clinical decision-making.
“The thing I like about being an item-writer is that it involves a well thought out process,” said Carlos Mena Hurtado, MD, a member of the Interventional Cardiology Item-Writing Task Force. “There is a rationale as to why things are written—from the social determinants of health to persistent symptoms in a patient—and each answer is backed by scientific evidence.”
ABIM reviews all items for compliance with item-writing best practices, then copyedits them and leads the creation of accompanying illustrations. Item-Writing Task Force members then have an opportunity to review and provide feedback on the revised questions.
All questions are then rigorously reviewed by the Approval Committees. Approval Committee members are also practicing physicians with extensive experience developing ABIM assessment content. Out of 100 questions created, approximately 70 will make it through the internal process and appear as an item on an assessment for pre-testing.
Once a question is written, there is an elegant, sophisticated peer review process, I learn a lot from the active feedback I receive in the process. It helps me to understand why a question is good or bad and how to improve it. It has also helped me to understand the value in what ABIM does.
Carlos Mena Hurtado, MD, a member of the Interventional Cardiology Item-Writing Task Force.
Joseph Ramzy, MD, a member of the Sleep Medicine Item-Writing Task Force, said: “I chose to join the ABIM Item-Writing Task Force because I find it both interesting and crucial to contribute to the development of exams that truly reflect evidence-based medicine. Being passionate about sleep medicine, this experience has deepened my understanding of the intricate process involved in crafting well-structured and comprehensive exam questions. It has not only bolstered my confidence in the exams’ quality but also provided me with valuable insights into the dedication and expertise of the diverse group of knowledgeable providers collaborating on this task force. Contributing to the development of these exams has significantly enhanced my confidence and pride in being a part of a team committed to upholding high standards in medical education.”
In addition to this rigorous item development and review process, ABIM staff reviews all comments and item performance statistics prior to scoring. During this process, items with potential issues are flagged and reviewed with the Approval Committee chair. If determined to be unfair, biased or inappropriate, the item is removed from scoring.
We invite you to consider openings on the Item-Writing Task Forces and learn more by viewing a video of item-writers sharing their experiences.
Has your practice changed? You can deactivate your certificate in that discipline
You may have found your practice has changed over time, and you no longer wish to maintain one or more of your certificates. If you would like to deactivate a certificate (which means you would no longer be listed as certified in the specialty) please let us know by 1/31/24. Deactivating a certificate also means you would not need to pay the annual MOC fee in 2024 for that certificate. If we do not hear from you by 1/31/24, we will assume you want to maintain that certificate and you will be responsible for the annual MOC fee for any certificates you have not deactivated. Visit your Physician Portal to deactivate a certificate.
Study finds new Milestones Rating System reduces but does not eliminate bias in residency knowledge ratings
Researchers from the American Board of Internal Medicine (ABIM) and Harvard University recently published a study that revealed significant bias in the knowledge ratings of Black, Latino and Asian internal medicine residents before the adoption of the Milestones ratings system in 2014. Though the new system appears to have reduced bias, it did not eliminate it against U.S.-born Black residents after being implemented.
Their findings were published in Annals of Internal Medicine in December and accompanied with an editorial commentary by authors from the George Washington School of Medicine and Health Sciences. The commentary noted that the study “is an important step in unraveling and combating the persistent nature of bias in the evaluation of internal medicine residents of color,” and made several suggestions as to how the information could be used to minimize bias further.
The Internal Medicine Milestones, developed by the Accreditation Council for Graduate Medical Education (ACGME) and ABIM, are competency-based developmental outcomes that incorporate competency committees, as well as more explicit and detailed rating scales into resident knowledge ratings than the previous model. The study compared pre- and post-Milestones ratings to scores on the ABIM Certification exam.
“The significant bias we found pre-Milestones both suggests that there exists large underlying evaluation bias against the minoritized groups we studied and provides an explanation as to why these groups are vastly underrepresented in academic medicine and other leadership positions,” said Bradley Gray, Ph.D., lead author and Senior Health Services Researcher for ABIM. “The fact that large bias persists for U.S.-born Black residents relative to their non-U.S. born counterparts is troubling, and suggests that there is a cultural history deeply seated in racism in the United States that specifically affects bias against Black Americans.”
“As a U.S.-born Black medical student and internal medicine resident, I witnessed firsthand what I thought was evaluator rater bias. It was really jarring to me when I was a medical student,” said Robert O. Roswell, MD, Associate Dean for Diversity, Equity and Inclusion, and Professor of Cardiology and Science Education at the Zucker School of Medicine at Hofstra/Northwell, and Chair-Elect of the ABIM Board of Directors. “ABIM and ACGME are to be commended for implementing a fairer resident assessment rating. Although the assessment is fairer, more research is needed to guide medical educators on how to reduce bias against U.S.-born Black residents for whom substantial bias persisted under the Milestones ratings system. At the very least, increasing awareness of implicit bias among medical educators is imperative.”
How the physician community informs ABIM’s programs
As we begin a new year, we want to thank everyone in the physician community who has so generously provided insights into their experiences with ABIM. By sharing your suggestions, you are helping ABIM better understand your needs and how we can enhance our programs so they fit within your busy life.
Through ABIM’s Community Insights Network, we receive feedback from thousands of physicians representing a wide array of career experiences, specialty areas and practice settings. We are learning what is working for you, the challenges you face in practicing medicine today and how ABIM can do its part to minimize burdens while helping you know and demonstrate that you are keeping your medical knowledge current.
Over the past several years, your feedback has led to the creation of new, more convenient and flexible assessment options, such as the Longitudinal Knowledge Assessment (LKA®). You also advocated for ABIM to advance equity in medicine by including questions on the social determinants of health on assessments. Most recently, you’ve asked for more tailored assessments, leading to the introduction of a new inpatient-focused assessment in internal medicine, as well as the Sleep Medicine: Obstructive Sleep Apnea Emphasis option.
Thank you for helping ABIM become a better organization in serving your needs. We look forward to our continued work together in 2024.
Research Roundup
Today’s physicians face a lot of demands on their time, including time spent studying and keeping their medical knowledge current. A growing body of evidence supports knowledge assessments as a way for physicians to know their medical knowledge is current:
In case you missed it:
Researchers from the University of Pittsburgh published a series of articles that focused on cognitive perspectives on maintaining physicians’ medical expertise. The articles explored how physicians maintain medical expertise across their careers and highlighted four key elements that are important in keeping medical knowledge current.
- In the absence of study, learned information is forgotten over time. Medicine is constantly changing and physicians need to be able to access new information quickly for patient care. Outdated knowledge can hinder learning and access to new information, which can harm outcomes, the authors say.
- We are not as aware of our weaknesses as we think. The authors say that poor performers tend to overestimate their ability, and learners prefer to focus on topics that they find interesting rather than topics in which they are the weakest. They also note that accessing external knowledge may be confused with actually having knowledge.
- The act of taking a test promotes knowledge, and feedback with correct responses enhances the ability to put knowledge into practice. Cognitive skills decline over time. The authors note that while crystallized intelligence remains intact until age 80, fluid knowledge (novel learning or balancing multiple tasks in working memory) declines as we age. Assessments help physicians keep up with changing standards, which ultimately improves patient care. This is especially true with assessments that provide feedback, such as ABIM’s Longitudinal Knowledge Assessment (LKA®).
- Passing an evaluation is a great motivator for staying current in medical practice. The act of testing in itself enhances learning.
A recent article in JAMA Internal Medicine found that a high number of diagnostic errors were linked to hospital deaths and ICU transfers. The study found that mistakes were driven by patient assessment and testing, and often led to harm or death. A separate study published in BMJ Open last year found that physicians who scored higher on diagnostic questions on ABIM’s MOC assessments were less likely to make errors that led to death, an ED visit or hospitalization for conditions at high risk for diagnostic errors. That study was done by researchers from ABIM, Harvard Medical School and the Mayo Clinic.
