Loading...
Identifying Examinees Who Possess Distinct and Reliable Subscores When Added Value is Lacking for the Total Sample
Citations
Altmetric:
Abstract
Research has demonstrated that although subdomain information may provide no added value beyond the total score, in some contexts such information is of utility to particular demographic subgroups (Sinharay & Haberman, 2014). However, it is argued that the utility of reporting subscores for an individual should not be based on one’s manifest characteristics (e.g., gender or ethnicity), but rather on individual needs for diagnostic information, which is driven by multidimensionality in subdomain scores. To improve the validity of diagnostic information, this study proposed the use of Mahalanobis Distance and HT indices to assess whether an individual’s data significantly departs from unidimensionality. Those examinees that were found to differ significantly were then assessed separately for subscore added value via Haberman’s (2008) procedure. To this end, simulation analyses were conducted to evaluate Type I error, power, and recovery of subscore added value classifications for various levels of subdomain test lengths, subdomain inter-correlations, and proportions of multidimensionality in the total sample. Results demonstrated that the HT index possessed around 100% power across all conditions, while maintaining Type I error below 5%, which led to nearly perfect recovery of subscore added value classifications. In contrast, the power rates for Mahalanobis Distance were much lower ranging from 13% to 61% with Type I errors maintained at the nominal level of 5%. Although the power rates were below the desired criterion of 80%, the cases identified as aberrant using this method were found to have greater variability between subdomain scores, increased reliability, and lower observed subdomain correlations when compared to the generated data. As a result, outlier cases were found to have subscore added value for nearly 100% of cases across conditions even when the generated multidimensional data did not possess subscore added value. These results were cross-validated using a large-scale high-stakes test in which the Mahalanobis Distance measure was found to identify 6.57% of 8,803 test-takers that possessed subscores with added-value who otherwise would have been masked by the unidimensionality of the total sample. Overall, this study suggests that the Mahalanobis Distance measure shows some promise in identifying examinees with multidimensional score profiles.
Type
dissertation
Date
2016-09