Loading...
Thumbnail Image
Publication

Identifying Rapid-Guessing Behaviors: Comparison of Response Time Threshold, Item Response Theory, and Machine Learning Methods

Abstract
On low-stake tests, unmotivated test takers may show disengaged behaviors, such as rapid responses with short response times and such non-effortful responses can be a threat to the test score validity. Several methods have been proposed to determine rapid-guessing behavior, which can be classified as response time threshold methods and item response theory based methods. In addition, I suggest a machine learning method based on neural networks known as the autoencoder for anomaly detection. This research aims to compare the detection of rapid-guessing using various methods and their effects on inferences drawn from test data. In the simulation study, (a) the classification accuracy of the methods and (b) the effects of motivation filtering, i.e., removing rapid-guesses, on person and item parameter estimates are investigated and compared. (c) The methods are also compared using public data from the Programme for International Student Assessment (PISA). (d) Finally, to better understand rapid-guessing detection methods, multiple variables in PISA process data are investigated using another machine learning method, gradient boosting machine. The results showed variations in the detection of rapid-guessing across different methods. Additionally, the impact of motivation filtering differed among methods, exhibiting interactions with the level of filtering and rapid-guessing patterns. The results of gradient boosting machine suggested response time, item-centered response time, and person-centered response time were the key information to predict rapid-guessing behaviors identified by included RG detection methods.
Type
openaccess
article
dissertation
Date
Publisher
Rights
License
http://creativecommons.org/licenses/by/4.0/
Research Projects
Organizational Units
Journal Issue
Embargo
2024-08-01T00:00:00-07:00
Publisher Version
Embedded videos
Collections