Understanding How Raters Communicate in the Context of Medical High-Stakes, Performance-Based Assessments

Thumbnail Image
Sebok, Stefanie Susane
Rater-Based Assessments , Competency-Based Assessment , Psychometrics , Medical Education
It is well established in the medical education literature that raters represent a significant source of unexplained variance in performance assessments. To better understand the variance associated with raters’ evaluative judgments, this dissertation investigated how raters, in various assessment contexts, communicate and articulate what they observe during performance assessments. Empirical data collected from three different performance assessments: a multiple mini-interview (MMI), an objective structured clinical examination (OSCE), and workplace-based assessments (WBAs) were analyzed to better understand raters’ behaviours. Relationships that exist between different forms of assessment were also examined. The first study established a proof of concept, showing that raters are not always consciously aware of the associations they make during assessments. The second study examined conditions under which raters find it difficult to make a seemingly uncomplicated judgment. Finally, the third study examined relationships that exist between quantitative and qualitative methods used in the evaluation of performance assessments. The empirical findings from this dissertation suggest that although raters may behave idiosyncratically, there exist patterns of behaviour that could be used to attain better information from raters about a learner’s performance and potentially allow raters to be more suitably matched according to their strengths in performance-based assessments. The findings of this research should help to overcome some of the challenges associated with rater-mediated assessments.
External DOI