How Assessments are Approved for Physicians
ABIM’s Approval Committees, composed of practicing physicians in each discipline, play an important role in the process of building assessments: they review, revise, approve and finalize every question for every ABIM assessment. They are also responsible for maintaining the blueprints for the initial certification and traditional, 10-year Maintenance of Certification (MOC) exams and the Longitudinal Knowledge Assessment (LKA®). In part, this means they work to ensure that the appropriate percentage of questions on the assessments cover corresponding subjects in the blueprint and therefore that physicians are tested in the most relevant areas of each discipline.
The questions—or items—for assessments are written by hundreds of expert physicians who volunteer on the Item-Writing Task Forces. When they have completed a draft, the items are delivered to the appropriate Approval Committee and put through a rigorous review process.
“The thing that strikes me about the examination approval committee’s work is its rigor,” said Ryan Maves, MD, FCCM, FCCP, FIDSA, a member of the Critical Care Medicine Traditional, 10-Year MOC Exam Approval Committee. “Everyone has a commitment to making sure that the test is a fair and accurate assessment of intensivists’ medical knowledge. Our goal is to make sure that every question is relevant to modern practice, reflects current evidence and is consistent with existing guidelines, when relevant. Every member of the Critical Care Medicine Approval Committees is a practicing intensivist and takes care of patients in the ICU, and I hope that that grounding in bedside care makes a difference in the quality of the test for our colleagues.”
Approval Committees are composed entirely of practicing physicians in each specialty; most of them have served on the Item-Writing Task Forces or the former Exam Committees, while others have experience writing questions for other organizations like specialty societies. All of the committee members are approved by ABIM Specialty Boards and undergo additional intensive training on best practices in item development and how to apply their skills as clinicians to the process of assessment-writing and approval. The Approval Committees also work with the Task Forces to offer mentorship and guidance as more experienced item-writers, ensuring the development of well written, relevant and rigorous test questions.
“Approval Committees are constituted by physicians who are seasoned experts in both their medical specialties and the art and science of creating test questions,” said Jeremy Dugosh, Ph.D., Vice President of Test Development at ABIM. “In addition to leading the development of relevant, practitioner-validated blueprints for assessments in their discipline, they make the final decisions for content developed by our hardworking taskforce of volunteer item-writers before it is eligible to appear on an assessment. This synergy between Item-Writing Task Forces and Approval Committees is central to the principle of assessment cocreation as an important part of professional self-regulation in internal medicine.”
Committee members review each question for clarity and to ensure that the clinical scenario is realistic, that there is only one correct response and that the rationales are clear and referenced properly.
Approval Committee members individually review multiple items each quarter and prepare to present them at meetings. At those meetings, each question is discussed and deconstructed.
Sometimes a question is returned to an item-writer for revision with an explanation about why it needs to be altered, such as a recent guideline change that was identified by an Approval Committee member. As a result, usually around 70 out of 100 questions are approved for pre-testing.
These questions appear on an assessment but are not included as part of the score. Instead, the responses are analyzed by psychometricians who ensure that the items are fair and that their performance statistics meet or exceed our minimum standards. No item is included in a diplomate’s score until it has passed the pre-testing process to ensure that scores are reliable and based on the best evidence for the discipline.
“The committees work through scores of questions to ensure that they are relevant to practice and identify important educational gaps as our knowledge base expands,” said Morie Gertz, MD, MACP, who was a member of the Hematology Exam Committee and then chair of the Hematology LKA Approval Committee before accepting a position as Chair of the Hematology Board. “There is a sense of pride associated with Maintenance of Certification and the knowledge that I can objectively demonstrate staying current in a discipline.”
Furman McDonald, MD, MPH, ABIM’s Senior Vice President for Academic and Medical Affairs added, “by mentoring the Item-Writing Task Force members and working with the Specialty Boards, all of which are made up of practicing diplomates in the field, the Approval Committees are an important part of ABIM governance and central to the success of the LKA and long form assessments.”
