Innovations in Assessment: Interview on CATALYST
January 9, 2020
NBOME recently sat down with Sandra Waters, MEM, Vice President for Collaborative Assessment & Initiatives, to learn more about the upcoming release of CATALYST, a new longitudinal assessment platform that will initially house COMSAE Phase 2 content when it launches this spring.
NBOME: Your team is debuting a new product this spring—COMSAE Phase 2 on CATALYST. We’re already familiar with COMSAE, but what exactly is CATALYST?
Sandra Waters: CATALYST is a longitudinal assessment platform designed to enhance learning. So, it isn’t actually content, it’s a new mechanism to deliver content to users.
NBOME: How is longitudinal assessment different from the more traditional learning approaches we’re used to?
SW: The notion of assessing someone over time—that really is the key. In a traditional class, individuals learn about a subject for a period of time, and then they learn about another subject, and then another subject. Longitudinal assessment uses something called topics interleaving, which enables an individual to gain exposure to ALL of those different components at one time. It creates these bursts of learning and knowledge acquisition.
If an individual is performing well in a certain area, they don’t need to be assessed nearly as frequently in that area. In areas where an individual isn’t performing well, CATALYST can increase the volume and frequency of content related to that trouble spot. The intent is to fine-tune the learning component, make it more targeted, and use that to increase knowledge and skills.
CATALYST was developed to combine learning with assessment. The assessments NBOME normally conducts are taken at a single point in time. Whereas, the CATALYST platform enables an individual to assess their knowledge and skills over an extended period of time. And it aids learning by providing users with immediate feedback while the material is still front-of-mind. Research has shown us that this is a much more effective way for an individual to learn—as opposed to sitting down, taking a test, and never truly understanding what you got wrong—or why.
NBOME: How did this all come about? What’s CATALYST’s origin story?
SW: When we first had the idea for CATALYST, we focused our efforts on designing the technology for board re-certification. The current approach involves a physician coming to a testing center every 6-10 years to take a closed book exam for 6-8 hours—not exactly the easiest feat when you’re running a full-time practice, seeing patients, and fulfilling all of the other responsibilities of a busy physician. CATALYST has the ability to change the whole playing field.
However, as we were developing and testing the platform, we continued to identify other ways to use it and other content we could put on there, including our own COMSAE content.
NBOME: Tell me more about why you decided to launch using COMSAE content.
SW: When we were developing CATALYST, we decided to pilot COMSAE Phase 2 on the platform. It just presented itself as an easy entry point for developing the added features that make CATALYST so special.
Each question includes a rationale essentially explaining why the correct answer is correct and why the incorrect answer is not correct. CATALYST also provides references for further learning and understanding. It’s a self-contained way to test knowledge and skills while providing additional information.
Further, COMSAE on CATALYST is built for busy schedules and maximum flexibility. It’s designed to feed questions to users at self-selected intervals. For example, you could opt to receive 10 questions each week or 30 questions all on one day. As we discussed before, everyone learns a little differently, and we all have different needs and schedules. This platform helps speak to that.
NBOME: With such strong focus on mobility and digitally nimble technology these days, what is the roll-out plan for COMSAE on CATALYST?
SW: COMSAE and other products offered on the CATALYST platform will be available on all devices, and also include a mobile app. Flexibility and convenience were extremely important to us as we developed the product.
NBOME: Who is eligible to purchase COMSAE Phase 2 on CATALYST and how does the system work? Can you walk me through the user experience?
SW: It is available to anyone who has an account with NBOME. Candidates may purchase the product through NBOME’s secure portal, at which point, they’ll be sent a welcome email along with login credentials to access the CATALYST platform. From here, they can customize the frequency of questions to suit their unique needs and learning goals. Based on those settings, they will begin to receive notifications when questions are available.
NBOME: If I was a student considering COMSAE on CATALYST, why would I want this over the traditional COMSAE format?
SW: I actually think you’d want both. The traditional COMSAE allows you get game-day-ready in an environment that closely mimics COMLEX-USA. Questions are formatted to match in style, and it’s a timed administration—just like COMLEX-USA. You also receive a final report once you complete the assessment. COMSAE on CATALYST is much more of a learning tool. It focuses on mastery of the content. You receive question-by-question and performance-by-domain feedback, rather than just a final report.
Because they serve completely different purposes, I wouldn’t necessarily see one replacing the other.
NBOME: What future enhancements can we look forward to with the CATALYST platform and longitudinal learning?
SW: We’re working on plans to expand our content offerings on the platform. COMSAE Phase 1 is being considered as an option, as well as COMAT subjects. Stay tuned!
