• Introduction
• Methods
• Results
• Discussion

Introduction

There has been strong interest among osteopathic medical students in measurement tools at various educational stages to self-assess their understanding of osteopathic medicine. The National Board of Osteopathic Medical Examiners (NBOME) developed the Comprehensive Osteopathic Medical Self-Assessment Examination (COMSAE) series to serve this need for self-assessment for osteopathic medical students.  COMSAE Phase 1 was first launched on March 24, 2008; Phase 2 was launched on June 26, 2008; and Phase 3 was launched on October 31, 2008.  By the end of 2008, more than 3000 osteopathic students had completed COMSAE.

Many osteopathic students took COMSAE before they took COMLEX. They used COMSAE to assess readiness for COMLEX.  It is in the interest of both the NBOME and osteopathic students to investigate how well COMSAE fulfills its intended purposes.  Since the competency level of osteopathic medicine assessed by COMSAE was defined using the same framework as COMLEX, a comparison between a student’s performance on COMSAE and on COMLEX can be considered as evidence of the predictive validity effectiveness of COMSAE. This report was designed to examine the relationship between osteopathic students’ COMSAE performances and their later COMLEX performances.  It was expected that osteopathic students who performed well on COMSAE would also perform well on COMLEX.

Methods

This report uses Phase 1 of COMSAE for its analysis because the largest number of students took Phase 1.  Phases 2 and 3 were launched in the later part of 2008.  The number of students for those two examinations was not sufficient for a detailed analysis.

The subjects of this study were 1152 Phase 1 examinees who took Phase 1 between March and November 2008.  All of those students took Phase 1 for the first time before they took COMLEX Level 1. 

The analysis first examined the correlation between scores on Phase 1 and Level 1.  A high correlation implies consistency between what was measured by the two examinations.  Although COMSAE was not developed to predict future performance on COMLEX, the experience of taking COMLEX-style COMSAE should help performance on COMLEX.  Therefore, this study also compared the score changes on the two examinations.  In addition, the relationship between pass/fail status on Phase 1 and Level 1 was also investigated.

COMSAE has two timing modes: Timed and Untimed.  In the Timed Mode, examinees had, proportionally, the same amount of time for Phase 1 as for Level 1.  In the Untimed Mode, examinees could have much more time than the actual Level 1 allows.  It was expected that the timing modes would have an impact on the relationship between performance on COMSAE and COMLEX.  So, the analyses described above were carried out for both timing modes for the purpose of understanding better the relationship between the two examinations.

Results

The correlation between Phase 1 scores and COMLEX Level 1 scores was .76 for timed Phase 1 and .68 for un-timed.  Both correlation coefficients are statistically significant.  In other words, examinees who scored high on Phase 1 were likely to score high on COMLEX Level 1 later and vice-versa.  However, this relationship was slightly weaker for those who took un-timed Phase 1 than it was for timed.

As Table 1 indicates, out of 1152 examinees, 734 (64%) took untimed Phase 1 and 418 took timed Phase 1 (36%).  The scores of Phase 1 and Level 1 were similar, with Level 1 scores being slightly higher.  The COMLEX Level 1 scores of examinees who took the timed COMSAE exam were, on average, 26 points higher than their Phase 1 scores.  For examinees who took the untimed COMSAE, the mean difference was 10 points less.  It was not clear why examinees who took the untimed version scored lower.  Since it was each examinee’s own decision to choose the timing mode, it can be inferred that the reasons for taking a particular timing mode might be related to the score differences between the two timing modes.

Table 1
The Average Difference between Level 1 and Phase 1 Scores

Timing Mode	N	Difference	Std
Timed	734	26.39	50.22
Untimed	418	16.39	64.13

 

Table 2 shows how many in this examinee group passed or failed Phase 1 and Level 1 using the standard score of 400 as the cutscore for both examinations.  This table is not broken down by timing modes.  Out of 1152 examinees, 129 (11.2%) failed Phase 1, and 120 (10.4%) failed Level 1.  The failing rates on both exams were very close to the actual Level 1 failing rate in the 2008-2009 testing year.  Table 2 shows that 83% of this examinee group passed both exams, and 4.7% failed both.  Analysis shows that 6.5% failed Phase 1 but improved to pass Level 1 later.  On the other hand, 5.7% passed Phase 1 initially but failed Level 1 later.  The chi-square test of this cross-classification table was significant, suggesting that passing or failing Phase 1 was significantly correlated to the passing or failing status on Level 1 (chi-square value=153.9149 and p-value<.0001).

Table 2
Pass and Fail Status in Phase 1 and Level 1

		COMSAE
COMLEX		Pass	Fail	Total
	Pass	957	75	1032
	Fail	66	54	120
	Total	1023	129	1152

 

Discussion

COMSAE is a self-assessment tool for osteopathic students.  Its format and structure closely resemble that of COMLEX.  Its scoring and reporting are also similar to those of COMLEX.  The value of a self-assessment tool for osteopathic students that reflects the style of COMLEX is easy to understand.

The empirical data further demonstrated the value of COMSAE.  The results of the first-year administration of Phase 1 strongly suggest that the outcomes of Phase 1 and Level 1 were highly related.  Most importantly, the average COMLEX Level 1 score was higher than the average COMSAE Phase 1 score for this group of examinees.  No doubt, the benefits came from becoming more familiar with COMLEX Level 1 and from conducting a self-assessment by taking COMSAE Phase 1.

The results also suggest that the timed COMSAE probably has higher self-assessment value because the testing conditions of timed COMSAE are more similar to the real timing conditions of COMLEX.

On the other hand, like any other measurement tools, COMSAE cannot predict COMLEX in absolute terms.  There were still 5.7% of examinees who passed Phase 1 but failed COMLEX.  For any COMSAE examinee, especially those who narrowly passed COMSAE, it is important to understand that passing COMSAE does not guarantee passing COMLEX.  Without extra effort, it is not guaranteed that the same performance level on COMSAE can be maintained for COMLEX.