Collaborative Cognitive Item Mapping

Ronkowski, E. Collaborative Cognitive Item Mapping Paper presented at the 2019 Conference of the American Board of Medical Specialties, Chicago, IL, September 2019.

Learning Objectives

Attendees will leave this presentation with ideas on how to innovate traditional item-writing workshops through Collaborative Cognitive Item Mapping (CCIM). They will also have an understanding of how to implement the Plan-Do-Check-Act (PDCA) model to innovate test development in a data-driven manner.

Session Summary

Collaborative Cognitive Item Mapping (CCIM) is a dynamic, new form of item development that builds on the literature in automatic item generation (AIG). In CCIM, a small group of subject matter experts (SMEs) develops items that assess essential testing objectives related to a clinical presentation, such as neck masses. The SMEs select high-frequency, high-impact diagnoses related to the topic, then map out patient findings and clinical decision-making processes. An item editor transforms the map into a set of items.

CCIM is beneficial because it is collaborative, systematic, and intentional. Independent item writing (IIW) can be challenging for physicians who are used to constant interactions and movement; CCIM allows SMEs to develop items without the intimidation of the blank page. The systematic approach of CCIM ensures that items include necessary details, such as duration of symptoms, and results in better distractors as SMEs think through plausible options for multiple diagnoses at the same time. With IIW, it is difficult to control for SMEs writing similar items on the same topics. With CCIM, a small group, rather than an individual, decides the testing objectives and diagnoses; this results in items that reflect the breadth and scope of the topic.

To develop CCIM, we implemented the Plan-Do-Check-Act (PDCA) model. At a pilot workshop, participants wrote items through IIW and CCIM. Through a collaboration of psychometricians, editors, and test developers, we fast-tracked a group of nearly 100 items for pretesting, and the results showed no significant statistical difference in item performance between the CCIM and IIW items. Our preliminary findings also suggest that CCIM can boost item production as much as 30% compared to traditional workshops.