NBOME

Choosing Wisely: So Many Options for Assessment Administration. Which will Enhance Your Exam’s Validity and Fairness?

Browne, M, Wojnakowski M, Horber DT.  Choosing Wisely: So Many Options for Assessment Administration. Which will Enhance Your Exam’s Validity and Fairness? Paper presented at the 2019 Innovations in Testing Conference, Orlando FL, March 2019.

 

Short Description

With advances in assessment, credentialing organizations are presented with myriad options to “enhance” test format and administration. Two organizations have been conducting research and pilot testing to explore some options alone and in combination – use of resources while testing, and, high stakes testing in remote proctored conditions.

Reference availability may increase an assessment’s fidelity to real life clinical situations, but it raises many implementation questions: Which references will be useful and what is the best way to make them available? What is the effect on test time needs, outcomes, and validity?

Remote proctoring is attractive to candidates as a convenience and can offer some cost savings. In reality though, just how easy is it to test from home? What are the security implications? Copyright treats remote proctored tests differently; how can this be addressed?

The presenters will discuss obstacles encountered, comparison of outcomes, and best practices found.

 

Full Description

With advances in assessment, credentialing organizations are presented with myriad options to “enhance” test format and administration. Two organizations have been conducting research and beta testing to explore some options alone and in combination – use of resources while testing, and, high stakes testing in remote proctored conditions.

 

As certification organizations move toward nontraditional assessments, provision of reference resources during assessment is one of many areas of uncertainty. Although reference material availability likely increases an assessment’s fidelity to real life applicable clinical situations, it raises many implementation questions as well as concerns about test outcomes and validity.

 

Remote proctoring, long a hot topic, has rarely been contrasted with in-person proctoring in a high stakes examination. The differences that materialized in candidate acceptance, test administration and outcomes can inform much constructive discussion.

 

One organization is researching options for continued professional certification for its 50,000-plus certificants. A 1,500-participant research study incorporating open-book features and different proctoring conditions was completed in October 2018. The research divided the participants into six different experimental conditions – in-person proctored vs remote proctored, no resources, e-resources, and hard copy resources.  Presenters will discuss development of the research design as well as the research outcomes.

 

Presenters from the second organization will discuss aspects of the development of an innovative item format that focuses on competency domains other than clinician knowledge recall. This incorporates the use of online resources to locate clinical diagnostic and treatment information to answer questions. The item format contains a clinical case scenario with associated multiple-choice items that would require most examinees to access online resources in order to answer the questions. In current day-to-day practice. Presenters will discuss the item development process and relate quantitative and qualitative data obtained from the Proof-of-Concept study. Lessons learned from this study and planned next steps will provide insights to organizations seeking more authentic modes of assessment of clinical behavior and decision-making.

 

^