Off-campus UMass Amherst users: To download campus access dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.
Non-UMass Amherst users: Please talk to your librarian about requesting this dissertation through interlibrary loan.
Dissertations that have an embargo placed on them will not be available to anyone until the embargo expires.
Author ORCID Identifier
https://orcid.org/0009-0001-4328-0888
AccessType
Open Access Dissertation
Document Type
dissertation
Degree Name
Doctor of Philosophy (PhD)
Degree Program
Education
Year Degree Awarded
2024
Month Degree Awarded
February
First Advisor
Scott Monroe
Second Advisor
Stephen G. Sireci
Third Advisor
Frederic Robin
Subject Categories
Educational Assessment, Evaluation, and Research
Abstract
On low-stake tests, unmotivated test takers may show disengaged behaviors, such as rapid responses with short response times and such non-effortful responses can be a threat to the test score validity. Several methods have been proposed to determine rapid-guessing behavior, which can be classified as response time threshold methods and item response theory based methods. In addition, I suggest a machine learning method based on neural networks known as the autoencoder for anomaly detection. This research aims to compare the detection of rapid-guessing using various methods and their effects on inferences drawn from test data. In the simulation study, (a) the classification accuracy of the methods and (b) the effects of motivation filtering, i.e., removing rapid-guesses, on person and item parameter estimates are investigated and compared. (c) The methods are also compared using public data from the Programme for International Student Assessment (PISA). (d) Finally, to better understand rapid-guessing detection methods, multiple variables in PISA process data are investigated using another machine learning method, gradient boosting machine. The results showed variations in the detection of rapid-guessing across different methods. Additionally, the impact of motivation filtering differed among methods, exhibiting interactions with the level of filtering and rapid-guessing patterns. The results of gradient boosting machine suggested response time, item-centered response time, and person-centered response time were the key information to predict rapid-guessing behaviors identified by included RG detection methods.
DOI
https://doi.org/10.7275/36471360
Recommended Citation
Lee, Minhyeong, "Identifying Rapid-Guessing Behaviors: Comparison of Response Time Threshold, Item Response Theory, and Machine Learning Methods" (2024). Doctoral Dissertations. 3062.
https://doi.org/10.7275/36471360
https://scholarworks.umass.edu/dissertations_2/3062
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.