Updates and News

A member of the NBOME team visits COMs participating for the first time in Phase 2. This provides the COMs with an onsite resource and allows the NBOME team to see in person how each COM makes the C3DO their own, nesting the standardized assessment within their local setup and protocols. The COMs do amazing work and it’s been extremely helpful to see and learn from their approaches to the C3DO.

The NBOME and participating COMs are also sharing experiences—with the osteopathic medical education community and globally. The NBOME presented on the C3DO at The International Meeting on Simulation in Healthcare (IMSH) in January and the Ottawa Conference on Clinical Competence in February. Attendees at last week’s American Association of Colleges of Osteopathic Medicine (AACOM) Educating Leaders conference had the opportunity to hear from the Phase 1 COMs about their experiences participating in the C3DO pilots and why they decided to continue with the program.

The COM perspective is also featured in the Annual Report 2023: SOARING.

If you missed those presentations, catch our upcoming presentations at the Association of Hospital Medical Education (AHME) in May and at the Association of SP Educators (ASPE) in June.

The C3DO Task Force met on March 20. Members received a brief update on Phase 2. Afterward, the Task Force engaged in discussions on quality assurance strategies for the first three goals of Phase 2.

Goals of Phase 2:

1. Can we create a standardized distributed osteopathic clinical skills assessment (added cases and questions from Phase 1)?
2. Is that assessment feasible in implementation and administration across various COMs, in various models (with significant updates from Phase 1)?
3. Does that activity produce scores that are “reasonable” given other known measures?
4. How can C3DO provide value to various stakeholder groups?

The first three goals are extensions of goals from Phase 1, focusing on changes made from the Phase 1 administrations. Ensuring we are evaluating the quality of the assessment will provide important information to the C3DO Advisory Panel and then to the NBOME Board of Directors.

The second half of the meeting was a discussion of the fourth goal—the “value” of the C3DO activity to various stakeholder groups. Members contributed their thoughts on how C3DO may provide added value to groups such as students, faculty, and administration at COMs, state licensing boards, residency program directors, patients, and others. Members asked insightful questions and provided valuable guidance regarding what this activity can and does measure and for what potential purpose.

These discussions contribute to the growing body of information to be considered by the NBOME Board of Directors regarding this pilot program of a distributed, standardized model for osteopathic clinical skills assessment.

February also saw the training of physician examiners, the rollout of new and updated training materials, and work toward process improvements for data transmission. Pilot COMs that begin testing later in the year have also started preparations, eager to learn from their early-testing colleagues as we work together to refine the C3DO assessment.

The workshop was designed to introduce the teams to the phase 2 project plan, starting with a summary of phase one and lessons learned. Important updates to the program that were a direct result of feedback from schools and student participants in 2023 included enhanced case development and item editing, simplification of training materials for standardized patients, improvements in data exchange, expanded student informational materials, and the creation of a comprehensive COM “toolkit,” as a few examples.

Sessions included how to create an exam form (the set of cases a candidate would see on a given testing day); review of Phase 2 administrative and IT elements; an introduction to the NBOME’s online Learning Center and the courses it hosts for C3DO standardized patients (SPs) and staff; discussion of OMT in the C3DO; an overview of SP training; review of resources and timelines; and a discussion about scoring considerations. Physician leads from the three COMs that had also participated in Phase 1 joined with NBOME staff to share insights from Phase 1 as well as recommendations regarding communicating with and preparing students to participate.

On January 22, each lead received their C3DO “toolkit,” allowing them and their teams to dive more deeply into the topics covered at their own pace and begin their preparations. The next step for each COM, depending on their testing dates, is a kickoff meeting for the COM’s core team. Pilots begin in March 2024 for returning colleges, and April 2024 for colleges participating for the first time, and run throughout the year.

“I enjoyed meeting others in the osteopathic profession and learning about what goes into case and testing development.”

“Great couple of days working on and editing cases. This is extremely valuable work by all involved and I can’t wait to see the final outcome.”

Additional opportunities will be available to participate through the year. If you are interested in participating, the best way to begin is by joining the NBOME National Faculty and indicating your interest in C3DO.

Proposals from COMs interested in participating in the C3DO Phase 2 pilots were due November 3. De-identified proposals were shared with members of the C3DO Task Force in advance of their November meeting.

The Task Force met on November 28. Members received an update on Phase 1 pilot scoring trends and correlations and the October Case Development Committee meeting. They also continued review of student feedback from Phase 1 to address concerns regarding value, timing, student preparation, and student and COM feedback in anticipation of Phase 2 pilots in 2024.

The Task Force reviewed the evaluation criteria for the Phase 2 proposals, discussed and debated points of difference among the proposals, and highlighted their relative strengths.

After the meeting, Task Force members individually evaluated each proposal, noted whether they would recommend the submitting COM for inclusion in the Phase 2 pilots, and ranked the submissions for inclusion. Information from the survey will be compiled and recommendations presented to the NBOME Board of Directors at their December meeting.

The RFP was sent to COM deans along with a registration link for an optional informational webinar. For information about the RFP or webinar, please reach out to your dean.

The C3DO Advisory Panel will meet on October 10. This is a panel made up of representatives of multiple stakeholder groups (AACOM [BOD, AOGME, ECOP, COSGP], AOA, AAOE, Licensing Board members, residents, students, and others). See March update for list of members. They will review the progress and reports from the C3DO Task Force and Pilots and advise the NBOME Board of Directors.

Also in October, a group of osteopathic physicians, including representatives from COMs that participated in Phase 1, will be convening in Conshohocken, PA for a C3DO case development meeting, creating and revising cases for use in the upcoming Phase 2 pilots.

In addition, they reviewed student survey feedback from participants in the Phase 1 pilots. Some examples:

798 students completed the post-activity survey, for a 98% response rate. They rated elements of the C3DO assessment and offered a total of 1,865 comments. 92% of students indicated that the C3DO provided at least a partial assessment of their skills (with additional questions asking about individual skill performance), and 89% agreed that the cases presented appropriate clinical challenges. Sample comments include:

• “I think because the cases were so diverse and many of them required you to think about their health holistically, and OMM was integrated in the case, I really did feel like I was able to integrate my osteopathic training.”
• “Overall, this was a great experience and very comfortable being in my home institution.”
• “Feel that this format was a formal and very thorough assessment of my clinical knowledge and skills up until this point in clinical training. Highly recommend”

NBOME staff is creating a brief video highlighting the student experience—watch for it coming soon!

The Task Force reviewed a draft request for proposals (RFP)—the guide for COMs thinking about submitting a proposal for their inclusion in the Phase 2 pilots. The RFP is being crafted to reflect lessons learned from Phase 1 and to make the process more user-friendly, including streamlining applications by using an online tool to apply. The 2023 RFP will be released on September 29, and proposal responses will be accepted through November 3, 2023.

Our goals were to identify opportunities to assess the feasibility of clinical skills assessment at the COMs while ensuring a national, standardized experience for fair evaluation of all osteopathic medical students. Our deep gratitude to all four schools for their hard work on the C3DO pilots and their gracious hosting of NBOME representatives during those visits.

The NBOME is now turning to analysis of the vast stores of information gathered during the pilot, from site visit reports, examination data, analytics from online training courses for pilot COM staff and standardized patients (SPs), and feedback received through surveys of COM staff, SPs, and students. Information from all of these sources will be joined with lessons learned over the past several months of pilot preparation to inform the C3DO Task Force and Advisory Panel in the coming months, as they consider next steps in C3DO development.

Additional announcements about the future of the pilot program as well as the current COM Dean clinical skills attestations from the NBOME Board of Directors are expected in late July.

• “I appreciated the preparation I made for this as it really helped push my training and understanding.”
• “After going through 2 years of clinical rotations, this was an opportunity to demonstrate what I have learned. I felt like my training as a medical student trained me well and this was my chance to show it.
• I think the assessment itself was reasonable but the preparation material could be more specific in explaining what is expected of us–will we be using aspects of different screening exams together etc.”
• “There is enough variety to touch on most fundamental osteopathic clinical skills.”
• “I would recommend giving a better understanding of the schedule and the expectations.”
• “Need OMT to be more apparent.”

Pilots continue through the late spring: MUCOM is administering their pilot sessions now and CUSOM and MWU-AZCOM are beginning their sessions in early June. The task force will meet again in early summer to review additional data.

A centralized group of OMT physician examiners chosen from across the country have been oriented and trained to provide evaluation of the performance of OMT on 3 cases per session. We believe that this model will provide an independent assessment of these skills during the patient encounters.

The C3DO Advisory panel met on April 12. Please see the last update for the names of these individuals and their represented organizations. This diverse group of individuals represents organizations from across the profession of osteopathic medicine and the learning continuum (including students and residents) as well as patient safety and advocacy. Each person had the opportunity to identify benefits, unintended consequences, and suggestions for the program. They will continue to provide advice regarding the project, the exam prototype, and feedback on reports generated from the C3DO Task Force.

NBOME leaders are meeting again with the state medical and osteopathic medical licensing community on May 3-May 6 for further input on this initiative at the national Federation of State Medical Boards (FSMB) Annual Conference and annual meeting of the American Association of Osteopathic Examiners (AAOE).

Pilot format

The pilot format will include 8 standardized patient encounters (14 minutes each); scored elements of history and physical examination (checklist); OMT (centrally scored; distributed physician examiner pool); and communication and respectfulness evaluation (global rubric). There will not be a SOAP note component, but students will instead answer two questions following their encounters: the first an open-ended differential diagnosis (or problem list) format pertinent to the SP presenting problem, and the second an extended multiple choice question aimed at key features of the diagnosis or treatment for the patient they encountered.

Pilot goals

Feasibility: Can the COMs administer a centrally developed examination? What is the experience like for them? The NBOME will use measures such as surveys of staff, faculty, Standardized Patients (SPs), and students, among other measures. As the participating COMs have selected uses for this investigation that align with their own curriculum, it will be investigated how this can integrate into their current processes.

Standardization: Can the resulting examination be standardized across COMs? The NBOME will analyze student performance evaluations, SP performance, and administration.

Advisory Panel

After an extensive application cycle with nominee solicitation from multiple stakeholder groups (AACOM [BOD, AOGME, ECOP, COSGP], AOA, AAOE, Licensing Board members, residents, students, and others), the chosen members of the C3DO Advisory Panel were notified on March 1, 2023. Their first meeting will be in April 2023, and they will review the progress and reports from the C3DO Task Force and Pilots. Based on the pilot research and the current performance assessment environment, the panel will also advise the NBOME Board of Directors. David Kuo, DO, has agreed to chair this panel.

AACOM BOD, Sharon J. Obadia, DO
AACOM BOD, Italo Subbarao, DO, MBA
AACOM BOD, Andrea P. Mann, DO
AACOM BOD, Heath A. Parker, DO
AAOE, Christine M. Khandelwal, DO
AAOE, William B. Swallow, DO
AAOE, David E. Tannehill, DO
ACOFP, Peter F. Bidey, DO, MSEd
ACOFP, Gautam J. Desai, DO
ACOI, Laura M. Rosch, DO, CS, MS
ACOI, David L. Tolentino, DO
ACOS, Mary K. Carneval, DO
ACOS, Alyssa R. Pastorino, DO
AOA BOT, Robert Dolansky, DO
AOA BOS, Carolyn Quist, DO
AOA BEL, Selena Raines, DO
AOA BEL, David O. Shumway, DO, HBS
AOA COCA, Janice A. Knebl, DO, MBA
AOSED, Brian Bowles
COSGP, Edoardo G. Frezza, OMS – through June 2023
COSGP, Nichelle Chastain, OMS – added October 2023
NF At-large, Rebecca Giusti, DO
NF At-large, Natalie A. Hyppolite, DO, MBS
Patient Safety, Maureen P. Barnes, ARM, CPHRM
Patient Safety, Bruce Anderson
SOMA, Lawsen Parker, OMS
Educator, Sherman Gorbis, DO
Educator, Amanda Cain, DO
Educator, Suzanne L. Rogers, DO
Educator, Darren Cohen, DO
Resident, Brysen Keith, DO

Next steps

Pilots begin in mid-April at the COMs’ selected times. We have been able to accommodate all COM requests for their preferred schedule to align with their existing activities.

NBOME staff and consultants have been meeting with representatives of the COMs, keeping the Physician Leads engaged with C3DO development and beginning work with SP training teams from the COMs that are administering the first pilot sessions. Training is competency-based, designed to ensure standardized performance across the COMs while letting the COMs draw on their expertise as they work with their SPs.

On Thursday, February 16, Jeanne Sandella, DO, VP for Professional Development Initiatives and Communications, and Amy Lorion, Senior Director for Professional Development Initiatives, presented at AACOM’s Community Forum Meeting. She shared updates and answered questions from the AACOM Board of Deans regarding the C3DO pilots.

The goals of this first set of pilots are to examine the feasibility of training, administration, and scoring at each site and to evaluate the standardization of the examination across the sites. The four selected COMs are excellent partners; all have clinical skills testing programs with experienced and dedicated staff and reflect different regions of the country and different sized classes.

A call for members of the C3DO Advisory panel, a larger oversight group formed of broader representation including from specialty colleges, student and patient advocacy groups, and other representation of the profession (e.g., AACOM/AACOM Groups, AOA and COCA, AAOE/Licensure Boards, other GME groups) was sent out earlier this month. Input from across the profession remains integral to the C3DO initiative.