Validating the Canadian Academic English Language Assessment for Diagnostic Purposes From Three Perspectives: Scoring, Teaching, and Learning

Abstract

Large-scale assessments are increasingly being used for more than one purpose, such as admissions, placement, and diagnostic decision-making, with each additional use requiring validation regardless of previous studies investigating other purposes. Despite this increased multiplicity of test use, there is limited validation research on adding diagnostic purposes—with the intention of directly benefiting teaching and learning—to existing large-scale assessments designed for high-stakes decision-making. A challenge with validating diagnostic purposes is to adequately balance investigations into the score interpretations and the intended beneficial consequences for teachers and students. The Assessment Use Argument (AUA) makes explicit these internal and consequential validity questions through a two-stage validation argument (Bachman & Palmer, 2010). This research adopted the AUA to examine the appropriateness of the Canadian Academic English Language (CAEL) Assessment for diagnostic purposes, by forming a validity argument that asked, to what extent did the CAEL essay meet the new diagnostic scoring challenges from the rater perspective, and a utilization argument centered on teachers' and students’ uses of the diagnostic information obtained from the assessment. This study employed three research phases at an English for Academic Purposes (EAP) program in one Canadian university. Data collection strategies included interview and verbal protocol data from two raters (Phase 1), interview and classroom observation data from one EAP course instructor (Phase 2), and interview and open-ended survey data from 47 English Language Learners (Phase 3). A multifaceted perception of CAEL for diagnostic purposes was observed: raters noted the greatest diagnostic potential at higher score levels, and teacher and student perceptions were largely influenced by previous diagnostic assessment experiences. This research emphasized the necessity of including multiple perspectives across contexts to form a deeper realization of the inferences and decisions made from diagnostic results.