Scores & Transcripts: Setting Score Standards COMLEX-USA
Setting the Pass/Fail Standard
Consistent with other national high-stakes examinations for physician licensure worldwide, the NBOME follows industry standard best practices in determining minimum pass/fail standards for the COMLEX-USA examinations and in periodic resetting of the standards. One of the most important components of a high-quality, and defensible examination program is a fair, and reasonable approach to identifying a “cut score” – the score at or above which a candidate is deemed to have passed an examination.
For COMLEX-USA examinations, a passing score means the examinee has demonstrated at least a minimal level of competency by scoring at or above the level judged to be required for the safe and effective practice of osteopathic medicine at the supervised level for COMLEX-USA Level 1, Level 2-CE and Level 2-PE, or at the unsupervised, independent practice of osteopathic medicine practice level for COMLEX-USA Level 3.
Every three to five years, the NBOME reviews the standards for minimum competence required to pass COMLEX-USA examinations. The standard setting process may result in a change in the cut score needed to pass an examination.
Periodic review of the COMLEX-USA standards ensures that the tested skill and proficiency of osteopathic medical knowledge reflect current medical education and practice required for licensure. The process is consistent with the NBOME’s mission of protecting the public by assessing competencies for osteopathic medicine and related health professions.
Approach to Standard Setting
The NBOME applies criterion-referenced methods to establishing cut scores. Sometimes referred to as “absolute standards,” these are not norm-based or relative to the performance of other candidates.
A process of triangulation is used to establish cut scores. The process is widely used for high-stakes examinations for physician licensure around the world and follows industry standard best practices. Triangulation includes standard-setting surveys, standard-setting panel meetings and a comprehensive final review.
The NBOME surveys key stakeholders to help gauge perceptions about the abilities of the candidate pool and the appropriateness of the current standards. Those surveyed include osteopathic medical school deans and residency program directors who work regularly with the examinee population. The residency program directors include those from ACGME-accredited graduate medical education programs. We also directly survey students and residents.
Physicians on the standard-setting panels are carefully selected to ensure broad representation of the profession with respect to clinical expertise and specialty, geographic considerations, age, gender and ethnicity. Panelists are selected to include representation from clinical practice, colleges of osteopathic medicine, state medical licensing boards and graduate medical education programs. The NBOME regularly seeks panel nominations from deans of the colleges of osteopathic medicine, the American Association of Colleges of Osteopathic Medicine (AACOM), residency program directors from ACGME-accredited programs, the Association of Osteopathic Graduate Medical Educators (AOGME), and the regulation and licensure communities.
Consistent with best practices in standard setting used by other physician testing and regulatory organizations worldwide, the NBOME reviews all survey data, panelist judgments, and psychometric analysis to arrive at the final pass/fail standard for each examination. Consideration is also given to recent trends in candidate performance, the relationship of score precision to the pass/fail decision as well as the political, social and educational implications of the data.
For the computer-based multiple choice COMLEX-USA examinations, the NBOME uses commonly accepted standard-setting methods such as the modified Angoff method.
In the modified Angoff standard-setting procedure, panelists make judgments based on test content (e.g., the expected performance of a minimally qualified candidate on a given test question).
The Angoff method begins with a discussion of the candidate who is minimally qualified to pass the examination. Panelists then review the test items and estimate the probability that this minimally qualified candidate will answer the item correctly. A passing score is derived for each panelist by summing these item probabilities. The final passing score is calculated by taking the average passing score across panelists.
The Hofstee method does not require individual question judgments. Instead, panelists are asked to give their impressions of what the minimum and maximum failure rates should be for the examination, as well as what the minimum and maximum percent correct scores should be. These rates and percent correct scores are averaged across panelists and projected onto the actual score distribution to derive a passing score. The Hofstee method is often used to evaluate or adjust the passing score derived using Angoff methodology. When the two methods produce similar passing scores, they validate each other.
This approach is consistent with the approach used around the world by most testing organizations related to physician licensure.
“I have been associated with many standard-setting initiatives over my 35 years in the educational assessment and credentialing exam fields. My impression is that your (standard-setting) study was one of the best that I have seen. You built your plan on best practices in the field, sought technical advice and used it, implemented the plan with sophistication and care, and finally, have comprehensively analyzed the results and nicely documented the process itself.”
Ronald K. Hambleton, Ph.D., Distinguished Professor, leading researcher, author and psychometrician on NBOME’s approach to recent standard setting for COMLEX-USA Level 2-PE