Off-campus UMass Amherst users: To download campus access dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.

Non-UMass Amherst users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Dissertations that have an embargo placed on them will not be available to anyone until the embargo expires.

Author ORCID Identifier

https://orcid.org/0009-0001-6012-8132

AccessType

Open Access Dissertation

Document Type

dissertation

Degree Name

Doctor of Philosophy (PhD)

Degree Program

Education

Year Degree Awarded

2023

Month Degree Awarded

September

First Advisor

Lisa Keller

Second Advisor

Scott Monroe

Third Advisor

Bruce Skaggs

Subject Categories

Educational Assessment, Evaluation, and Research

Abstract

The detection of aberrant responses in psychological and educational testing is crucial for maintaining the validity and reliability of assessment scores. This dissertation evaluates the performance of a new detection method, known as the bootstrap method, specifically targeting random guessing behaviors. A simulation design is used to compare the bootstrap method with traditional cumulative sum control chart (CUSUM) and change-point analysis (CPA) methods, considering various test lengths and severity rates. The findings indicate that the bootstrap method did not exhibit superior performance in detecting aberrant responses compared to the traditional methods. It demonstrated lower detection power and higher type I error rates, particularly when examinees had similar probabilities of providing correct responses regardless of random guessing or thoughtful answering. While the traditional CUSUM and CPA methods remain effective, the bootstrap method requires further refinement to enhance its detection power. To address these limitations, future research can explore the integration of additional statistics into the bootstrap method to improve the accuracy of ability estimation, especially when a higher number of aberrant responses are present. This research contributes to the field of educational measurement by highlighting the need for ongoing refinement and optimization of the bootstrap method to effectively identify various forms of aberrant behaviors in psychological and educational testing settings. The findings underscore the importance of robust detection methods for maintaining the integrity and validity of assessments, emphasizing the significance of ongoing research and development in this area.

DOI

https://doi.org/10.7275/35961231

Download

Share

COinS