Thumbnail Image

Aberrant Response Detection: Incorporating Cumulative Sum Control Chart and Change-Point Analysis

The detection of aberrant responses in psychological and educational testing is crucial for maintaining the validity and reliability of assessment scores. This dissertation evaluates the performance of a new detection method, known as the bootstrap method, specifically targeting random guessing behaviors. A simulation design is used to compare the bootstrap method with traditional cumulative sum control chart (CUSUM) and change-point analysis (CPA) methods, considering various test lengths and severity rates. The findings indicate that the bootstrap method did not exhibit superior performance in detecting aberrant responses compared to the traditional methods. It demonstrated lower detection power and higher type I error rates, particularly when examinees had similar probabilities of providing correct responses regardless of random guessing or thoughtful answering. While the traditional CUSUM and CPA methods remain effective, the bootstrap method requires further refinement to enhance its detection power. To address these limitations, future research can explore the integration of additional statistics into the bootstrap method to improve the accuracy of ability estimation, especially when a higher number of aberrant responses are present. This research contributes to the field of educational measurement by highlighting the need for ongoing refinement and optimization of the bootstrap method to effectively identify various forms of aberrant behaviors in psychological and educational testing settings. The findings underscore the importance of robust detection methods for maintaining the integrity and validity of assessments, emphasizing the significance of ongoing research and development in this area.