Off-campus UMass Amherst users: To download campus access dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.

Non-UMass Amherst users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Dissertations that have an embargo placed on them will not be available to anyone until the embargo expires.

Author ORCID Identifier

N/A

AccessType

Open Access Dissertation

Document Type

dissertation

Degree Name

Doctor of Philosophy (PhD)

Degree Program

Education

Year Degree Awarded

2016

Month Degree Awarded

September

First Advisor

Stephen G. Sireci

Second Advisor

Craig S. Wells

Third Advisor

Michael Lavine

Subject Categories

Educational Assessment, Evaluation, and Research | Quantitative Psychology | Statistical Methodology

Abstract

Research has demonstrated that although subdomain information may provide no added value beyond the total score, in some contexts such information is of utility to particular demographic subgroups (Sinharay & Haberman, 2014). However, it is argued that the utility of reporting subscores for an individual should not be based on one’s manifest characteristics (e.g., gender or ethnicity), but rather on individual needs for diagnostic information, which is driven by multidimensionality in subdomain scores. To improve the validity of diagnostic information, this study proposed the use of Mahalanobis Distance and HT indices to assess whether an individual’s data significantly departs from unidimensionality. Those examinees that were found to differ significantly were then assessed separately for subscore added value via Haberman’s (2008) procedure. To this end, simulation analyses were conducted to evaluate Type I error, power, and recovery of subscore added value classifications for various levels of subdomain test lengths, subdomain inter-correlations, and proportions of multidimensionality in the total sample. Results demonstrated that the HT index possessed around 100% power across all conditions, while maintaining Type I error below 5%, which led to nearly perfect recovery of subscore added value classifications. In contrast, the power rates for Mahalanobis Distance were much lower ranging from 13% to 61% with Type I errors maintained at the nominal level of 5%. Although the power rates were below the desired criterion of 80%, the cases identified as aberrant using this method were found to have greater variability between subdomain scores, increased reliability, and lower observed subdomain correlations when compared to the generated data. As a result, outlier cases were found to have subscore added value for nearly 100% of cases across conditions even when the generated multidimensional data did not possess subscore added value. These results were cross-validated using a large-scale high-stakes test in which the Mahalanobis Distance measure was found to identify 6.57% of 8,803 test-takers that possessed subscores with added-value who otherwise would have been masked by the unidimensionality of the total sample. Overall, this study suggests that the Mahalanobis Distance measure shows some promise in identifying examinees with multidimensional score profiles.

DOI

https://doi.org/10.7275/8892126.0

Download

Share

COinS