Off-campus UMass Amherst users: To download dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.

Non-UMass Amherst users, please click the view more button below to purchase a copy of this dissertation from Proquest.

(Some titles may also be available free of charge in our Open Access Dissertation Collection, so please check there first.)

Impact of several computer -based testing variables on the psychometric properties of credentialing examinations

Dehui Xing, University of Massachusetts Amherst

Abstract

The purpose of the study was to investigate the impact of several testing variables—level of item quality, item bank size, placement of passing score, and computer-based test design, on two important indicators of the quality of credentialing exams, decision consistency and decision accuracy. The computer-based test designs of interest in this study were linear parallel form tests, multi-stage tests, and computerized adaptive tests. The choice of test design model plays an important role in both minimizing decision-making error and increasing test efficiency. However, of equal, if not greater, importance are the resources available to test developers: the number of items in the bank, and the quality of the items. Simulation studies were conducted to investigate the effects of these testing variables on the accuracy and consistency of binary decision-making of credentialing exams. These variables are all very essential and manipulable in practice and therefore their roles are especially worthy of investigation. To the extent possible, realistic situations were simulated to increase the generalizability of the findings. The main findings from the study were as follows: (1) improvements in item quality had the desirable effect of increasing decision accuracy and decision consistency by a practically significant amount; (2) doubling bank size helped significantly in lowering item exposure but had little direct impact on decision accuracy and decision consistency; (3) the location of the passing score significantly impacted on test length necessary to achieve desirable levels of test statistics, and (4) all three computer-based test designs produced comparable results in the conditions simulated. Two main conclusions can be drawn from the findings. First, steps to increase item quality such as improving item writer training, cloning the best items, and improving and extending field testing of new items, offer the potential for improving the statistical characteristics of pass-fail credential exam decisions. Second, the general attractiveness of new computer-based test designs may be less important when pass-fail decisions rather than precise ability estimates are the focus of the examination. Clearly, more research to pursue these two conclusions, and confirm them, if possible, is in order.

Subject Area

Psychological tests

Recommended Citation

Xing, Dehui, "Impact of several computer -based testing variables on the psychometric properties of credentialing examinations" (2001). Doctoral Dissertations Available from Proquest. AAI3012196.
https://scholarworks.umass.edu/dissertations/AAI3012196

Share

COinS