Off-campus UMass Amherst users: To download dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.
Non-UMass Amherst users, please click the view more button below to purchase a copy of this dissertation from Proquest.
(Some titles may also be available free of charge in our Open Access Dissertation Collection, so please check there first.)
The effects of dimensionality and item selection methods on the validity of criterion-referenced test scores and decisions
Abstract
Many of the measurement models currently used in testing require that the items that make up the test span a unidimensional space. The assumption of unidimensionality is difficult to satisfy in practice since item pools are arguably multidimensional. Among the causes of test multidimensionality are the presence of minor dimensions (such as test motivation, speed of performance and reading ability) beyond the dominant ability the test is supposed to measure. The consequences of violating the assumption of unidimensionality may be serious. Different item selection procedures when used for constructing tests will have unknown and differential effects on the reliability and validity of tests. The purposes of this research were (1) to review research on test dimensionality, (2) to investigate the impact of test dimensionality on the ability estimation and the decision accuracy of criterion-referenced tests, and (3) to examine the effects of interaction of item selection methods with test dimensionality and content categories on ability estimation and decision accuracy of criterion-referenced tests. The empirical research consisted of two parts: in Part A, three item pools with different dimensionality structures were generated for two different tests. Four item selection methods were used to construct tests from each item pool, and the ability estimates and the decision accuracies of the 12 tests were compared in each test. In Part B, real data were used as an item bank, and four item selection methods were used to construct short tests from the item bank. The measurement precision and the decision accuracies of the resulted tests were compared. It was found that the strength of minor dimensions affect the precision of the ability estimation and decision accuracy of mastery tests, and that optimal item selection methods perform better than other item selection methods, especially when test data are not unidimensional. The differences in measurement precision and decision accuracy among data with different degrees of multidimensionality and among the different item selection methods were statistically and practically significant. An important implication of the study results for the practitioners are that the presence of minor dimensions in a test may lead to the misclassification of examinees, and hence limit the usefulness of the test.
Subject Area
Educational tests & measurements|Quantitative psychology|Educational evaluation
Recommended Citation
Dirir, Mohamed Awil, "The effects of dimensionality and item selection methods on the validity of criterion-referenced test scores and decisions" (1993). Doctoral Dissertations Available from Proquest. AAI9329592.
https://scholarworks.umass.edu/dissertations/AAI9329592