How Does ABIM Create Assessment Questions?

ABIM launched the Longitudinal Knowledge Assessment (LKA®) in 2022 in response to feedback from the physician community asking for more flexible and convenient ways to maintain their certification.  

As the LKA includes a greater number of questions than the traditional, 10-year MOC exam, ABIM established the Item-Writing Task Forces to meet the demand for assessment content. Each year ABIM develops 200 new questions in each specialty (equating to 3,200 new items across all disciplines annually).

The Item-Writing Task Forces are entirely composed of physicians in each specialty. They develop exam content through innovative, science-based approaches to item development. Any physician can apply to be an item-writer in their specialty area. Each assessment has a blueprint that outlines the content of the exam. Item-Writing Task Force members are assigned a different blueprint topic to cover and each is responsible for developing about 20 questions annually.

For each question, the writer first develops a “testing point” that addresses what a physician should know on that topic without having to consult medical resources or references. The testing point is like a map that links the topic to an area of the blueprint so that the test will accurately cover what the blueprint requires. An example of a testing point is: “Know to include cystic fibrosis in the differential diagnosis of an adult patient who has chronic cough and bronchiectasis.”

Item-writers then carefully craft well-researched, multiple-choice questions (MCQs) with a single best answer. ABIM uses MCQs because research has shown that scores obtained with MCQs are correlated with superior clinical performance and are particularly suitable for simulating clinical decision-making. The overwhelming majority of ABIM exam questions use a clinical stem (patient-based case scenario) format that assesses the higher-order cognitive abilities required for clinical decision-making.

“The thing I like about being an item-writer is that it involves a well thought out process,” said Carlos Mena Hurtado, MD, a member of the Interventional Cardiology Item-Writing Task Force. “There is a rationale as to why things are written—from the social determinants of health to persistent symptoms in a patient—and each answer is backed by scientific evidence.”

ABIM reviews all items for compliance with item-writing best practices, then copyedits them and leads the creation of accompanying illustrations. Item-Writing Task Force members then have an opportunity to review and provide feedback on the revised questions.

All questions are then rigorously reviewed by the Approval Committees. Approval Committee members are also practicing physicians with extensive experience developing ABIM assessment content. Out of 100 questions created, approximately 70 will make it through the internal process and appear as an item on an assessment for pre-testing.

“Once a question is written, there is an elegant, sophisticated peer review process,” said Dr. Mena. “I learn a lot from the active feedback I receive in the process. It helps me to understand why a question is good or bad and how to improve it. It has also helped me to understand the value in what ABIM does.”

Joseph Ramzy, MD, a member of the Sleep Medicine Item-Writing Task Force, said: “I chose to join the ABIM Item-Writing Task Force because I find it both interesting and crucial to contribute to the development of exams that truly reflect evidence-based medicine. Being passionate about sleep medicine, this experience has deepened my understanding of the intricate process involved in crafting well-structured and comprehensive exam questions. It has not only bolstered my confidence in the exams’ quality but also provided me with valuable insights into the dedication and expertise of the diverse group of knowledgeable providers collaborating on this task force. Contributing to the development of these exams has significantly enhanced my confidence and pride in being a part of a team committed to upholding high standards in medical education.”

In addition to this rigorous item development and review process, ABIM staff reviews all comments and item performance statistics prior to scoring. During this process, items with potential issues are flagged and reviewed with the Approval Committee chair. If determined to be unfair, biased or inappropriate, the item is removed from scoring.

We invite you to consider openings on the Item-Writing Task Forces and learn more by viewing a video of item-writers sharing their experiences.