Seeking the Real Reliability: Why the Traditional Estimators of Reliability Usually Fail in Achievement Testing and Why the Deflation-Corrected Coefficients Could Be Better Options

Authors

Jari Metsämuuronen, University of Turku, Turku Research Centre for Learning Analytics, and Finnish National Education Evaluation Centre (FINEEC)Follow

DOI

https://doi.org/10.7275/pare.1264

Abstract

Traditional estimators of reliability such as coefficients alpha, theta, omega, and rho (maximal reliability) are prone to give radical underestimates of reliability for the tests common when testing educational achievement. These tests are often structured by widely deviating item difficulties. This is a typical pattern where the traditional Pearson correlation between items and score (Rit) may be radically deflated. Because Rit is embedded in the traditional estimators of reliability, this causes deflation in the estimates of reliability, and the magnitude of deflation may be remarkable. Within achievement testing, deflation-corrected estimators of reliability (DCER) would be better options. Instead of Rit, DCERs use other estimators of correlation as the linking factor between the item and the score variable that are less prone to deflation. Selecting wisely the linking coefficient, DCERs may give significant advance in estimating the true reliability and true standard error related to the test score.

Recommended Citation

Metsämuuronen, Jari (2023) "Seeking the Real Reliability: Why the Traditional Estimators of Reliability Usually Fail in Achievement Testing and Why the Deflation-Corrected Coefficients Could Be Better Options," Practical Assessment, Research, and Evaluation: Vol. 28, Article 10.
DOI: https://doi.org/10.7275/pare.1264
Available at: https://scholarworks.umass.edu/pare/vol28/iss1/10