Institutional assessment of student learning objectives has become a fact-of-life in American higher education and the Association of American Colleges and Universitiesâ€™ (AAC&U) VALUE Rubrics have become a widely adopted evaluation and scoring tool for student work. As faculty from a variety of disciplines, some less familiar with the psychometric literature, are drawn into assessment roles, it is important to point out two easily made but serious errors in what might appear to be one of the more straightforward assessments of measurement qualityâ€”interrater reliability. The first error which can occur when a third rater is brought in to adjudicate a discrepancy in the scores reported by an initial two raters has been well-documented in the literature but never before illustrated with AAC&U rubrics. The second error is to cease training before the raters have demonstrated a satisfactory level of interrater reliability. This research note describes an actual case study in which the interrater reliability of the AAC&U rubrics was incorrectly reported and when correctly reported found to be inadequate. The note concludes with recommendations for the correct measurement of interrater reliability. Accessed 1,575 times on https://pareonline.net from December 08, 2017 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
"The Miscalculation of Interrater Reliability: A Case Study Involving the AAC&U VALUE Rubrics,"
Practical Assessment, Research, and Evaluation: Vol. 22
, Article 11.
Available at: https://scholarworks.umass.edu/pare/vol22/iss1/11