How does ABIM create its assessments?

ABIM creates and administers at least 50 initial certification and traditional 10-year MOC assessments across nearly two dozen disciplines each year, and 15 Longitudinal Knowledge Assessments (LKA^®) quarterly. This number continues to grow with the development of the LKA in additional specialty areas. For every assessment, each question is developed, reviewed, revised, approved, pre-tested and then discarded (if shown to not be working properly) or revised and tested again before finally appearing on an assessment as a scored question.

Each question is carefully considered from every angle: should every physician in this specialty area be able to answer the question without a reference? Is it relevant to the practice in that specialty? Is it a fair question? Is there more than one correct response? Is it biased? Does it require some thought and analysis? One question can take many revisions to get through the approval process and become a scored assessment question.

“It is critically important that question writers are trained to write fair and valid questions and that the process ABIM uses to develop and vet these questions is rigorous and thorough,” said Rebecca Lipner, Ph.D., Senior Vice President, Assessment and Research at ABIM. “By so doing, we can be assured that test takers are scored only on questions that have been put through the process and have been approved by experts in the field.”

The process begins with the blueprint, which is initially created by the Approval Committee for each specialty and then vetted with a larger community of diplomates through a blueprint review process. A blueprint is similar to a table of contents for an assessment. It is based on analyses of current practices and the relative importance of the clinical problems in the specialty area. Blueprints are reviewed annually and changes usually follow trends in current practice.

Check out the blueprint for your specialty to see what’s covered.

Once a blueprint is developed and approved, three groups of subject matter experts step up to create and approve exam content.

• Item-Writing Task Force members develop questions for ABIM assessments.
• Mentors and professional staff advise new item-writers throughout the item development process.
• Approval Committee members review and approve all items before they are used on an assessment. Approval Committee chairs review any question that is not performing as expected before it goes into a test taker’s score.

ABIM is always looking for qualified physicians to join one of these groups. Participating in creating assessments provides a meaningful and rewarding opportunity to join outstanding colleagues in service of the profession on behalf of patients. All item writers, mentors and Approval Committee members must hold valid Board Certification in Internal Medicine or an internal medicine subspecialty. View openings and learn what members say about their experiences. If you have any questions, feel free to contact the Test Development Task Force.

Each Item-Writing Task Force member is assigned a different blueprint topic to cover and is responsible for developing about 20 questions annually.

For each question, the writer first develops a “testing point” that addresses what a physician should know in that topic without having to consult medical resources or references. The testing point is like a map that links the topic to an area of the blueprint so the test will accurately cover what the blueprint requires. An example of a testing point is: “Know to include cystic fibrosis in the differential diagnosis of an adult patient with chronic cough and bronchiectasis.”

They then carefully craft well-researched, multiple-choice questions (MCQs) with a single best answer. ABIM uses MCQs because research has shown that scores obtained with MCQs are correlated with superior clinical performance and are particularly suitable for simulating clinical decision-making. The overwhelming majority of ABIM exam questions use a clinical stem (patient-based case scenario) format that assesses the higher-order cognitive abilities required for clinical decision-making.

“Being an item-writer has been one of the most gratifying and meaningful experiences of my career,” said Thomas Strouse, who is currently on the Hospice & Palliative Medicine LKA Approval Committee. “For me it was a great intellectual exercise and an opportunity to work alongside some remarkable people.” He added that working to create clear questions that have a single correct response is a great challenge and sharpens the skills and knowledge of item-writers.

Every item is then processed by ABIM, reviewed for compliance with item-writing best practices and copyedited, and illustrations are processed. Item-Writing Task Force members then have an opportunity to review and provide feedback on the edited questions.

All questions are then rigorously reviewed by the Approval Committee. The review process seeks to ensure that each question is fair and relevant, can be read and answered within a reasonable time by the test taker and has only one correct answer. Only questions that are approved by the Committee are eligible to move on to pretesting.

Once a question has approval, it moves on to the pretesting phase. Pretesting is a standard practice in assessment development that allows testing of new questions without risk to the test taker, since they are not counted in the test score at this point in time.

Each pretest question is assessed according to statistical performance criteria, and if it passes the criteria, it is then accepted into the pool of scorable questions.

“Pretesting is another key aspect of the process that ensures fairness in that only good questions get used in the assessment of test takers,” said Dr. Lipner.

Once the questions are finalized, vetted and approved, assessments are created using an Automated Test Assembly (ATA) program. ATA ensures a fair balance of content on each form of an assessment. ATA also ensures that exams are comparably constructed with regard to question difficulty, discrimination, relevance and other statistical constraints. This process is used for all ABIM assessments.

How does ABIM Determine the Passing Score for an Exam?

The passing score, also known as a cut score or test standard, is the score that examinees must meet or exceed to pass the assessment. Examinees who score at or above the passing score will pass; examinees who score below the passing score will not. Passing scores at ABIM are absolute, meaning there is no predetermined pass rate. Instead your pass/fail decision is driven entirely by your performance relative to the passing score.

In accordance with best practice in assessment, ABIM periodically consults with practicing physicians in your discipline to review and update the passing score. This allows ABIM to ensure that the passing scores reflect appropriate and current expectations for examinee performance in the discipline. As a result, the passing score shown on the score report may change every five to seven years.

More information on standard setting is available on the ABIM website.

“The standard setting process is at the cornerstone of certification programs since it identifies those with adequate knowledge to become certified or maintain certification,” said Dr. Lipner. “It is essential that this process is done credibly, and uses the appropriate experts and a rigorous, research-supported methodology based on informed judgment.”