Physician Driven Change: ABIM’s Open-book Assessments
Exploring an Open Book Assessment
As part of its Community Engagement efforts, thousands of physicians made it clear to ABIM through surveys, focus groups and society meetings that they wanted assessments to more closely reflect how they practice today’s medicine. Among those suggestions were implementing open-book exams with access to resources they use every day at the point of care.
To determine the feasibility and impact of incorporating electronic resources use at the point of care into MOC exams, in 2017, more than 800 doctors participated in a research study conducted by ABIM. Key findings included:
- Post-study survey results indicated that having access to resources did reduce physicians’ anxiety about the exam.
- The open-book option enhanced the exam’s capacity to differentiate between high and low performances.
- Physicians in open-book conditions used more time, but time constraints did not adversely affect exam performance and did not change the specific skill or factor that is measured by the exam.
ABIM also asked society partners and worked with its specialty boards and exam committees to help determine the most common resources that physicians use in their respective disciplines. UptoDate was a clear front-runner with over 1.5 million physicians worldwide using it as their evidenced-based clinical decision support tool. Additionally, an earlier study examining the effect of electronic resources on performance characteristics showed that 90% of survey respondents reported using UpToDate as their typical clinical resource. Read the press release announcing the decision to work with UpToDate here.
Armed with these results, ABIM worked with Pearson VUE and UpToDate to make sure that this new feature would be available to test-takers in 2018.
This spring, the first cohort of more than 4,000 board certified physicians were able to experience the open-book feature through the Internal Medicine and Nephrology Knowledge Check-Ins and traditional MOC exams.
Did it live up to expectations?
For most physicians, UpToDate performed as expected. We did receive some reports—from both those taking the Knowledge Check-In online and in a test center—that the exam software took longer to load than it should have and we are working with our vendors to improve the integration and load time.
Here’s some (first-hand, unedited) commentary from physicians who used the resource:
“Having access to external reference is more similar to my actual daily practice. Maybe a little more time should be allow for each question. Also Up-To-Date is good. There are other good references too to consider to include (such as the ACP online website).”
“Use of external resource was a mimic of actual practice and in this fashion a good proxy of critcial thinking.”
“Having the UpToDate resource made the testing experience better and a more realistic evaluation of practice ability. The test required clinical knowledge and reasoning ability but the availability of UpToDate reduced the need to memorize easily retreivable facts.”
The Future of Open Book Assessments
ABIM has worked closely with Pearson VUE and UpToDate and is already putting fixes in place to ensure improved performance for the fall. Starting in fall of 2018 UptoDate will be available in all traditional MOC and Knowledge Check-In assessments. ABIM also continues to collaborate with Pearson VUE to identify feasible strategies for supporting multiple resource formats within the exam interface.
We will be surveying diplomates by specialty on their preferred resources which could eventually lead to selecting a set of specialty-specific resources for each exam. Stay tuned for more information!
