Off-campus UMass Amherst users: To download campus access dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.

Non-UMass Amherst users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Dissertations that have an embargo placed on them will not be available to anyone until the embargo expires.

Author ORCID Identifier

N/A

AccessType

Open Access Dissertation

Document Type

dissertation

Degree Name

Doctor of Philosophy (PhD)

Degree Program

Education

Year Degree Awarded

2016

Month Degree Awarded

September

First Advisor

Ronald K. Hambleton

Subject Categories

Educational Assessment, Evaluation, and Research | Quantitative Psychology

Abstract

In on-demand high-stakes testing programs such as GRE and TOEFL, some items are repeatedly used across test administrations to reduce the cost of developing new items constantly. Item exposure provides an opportunity for examinees to have knowledge of particular test items in advance of their administration. It poses a threat to test security and ultimately will result in invalid test scores. Therefore, many testing programs conduct quality control to monitor test compromise at individual and/or group level. A predictive checking method is proposed in this study to detect examinee preknowledge on exposed items. We consider a scenario where a test can be divided into two subsets of items: one consisting of secure items with very low exposure rates and the other consisting of possibly compromised items (i.e. unsecure items) which have been exposed for a while. An examinee’s proficiency distribution is first obtained from secure items and then the predictive distribution for the examinee’s test scores on the unsecure items is constructed. The extent of test compromise is determined by comparing an individual’s observed score on the unsecure items with the predictive distribution. To evaluate the effectiveness of this approach, three studies are conducted: the first study investigates the statistical properties (i.e. type-I error and power) of this method under four factors through Monte Carlo simulation; the second study applies this method to two simulated test compromise situations that are likely to happen in practice, and compares this method to three other detection approaches; the third study applies this method to a real dataset to demonstrate its practice use. Findings from the simulation studies suggest that the predictive checking method is effective in detecting examinees’ preknowledge in the unsecure subset given a moderate to large test compromise rate, while maintaining its type-I error close to or lower than the nominal level. It also demonstrates similar or better performance than the other approaches under investigation. These results have implications for conducting quality control at individual examinee level in an on-demand testing program.

DOI

https://doi.org/10.7275/8965479.0

Share

COinS