Off-campus UMass Amherst users: To download campus access dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.

Non-UMass Amherst users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Dissertations that have an embargo placed on them will not be available to anyone until the embargo expires.

Author ORCID Identifier

https://orcid.org/0000-0002-5714-0122

AccessType

Open Access Dissertation

Document Type

dissertation

Degree Name

Doctor of Philosophy (PhD)

Degree Program

Education

Year Degree Awarded

2019

Month Degree Awarded

May

First Advisor

Craig S. Wells

Second Advisor

April L. Zenisky

Third Advisor

Stephen G. Sireci

Fourth Advisor

Timothy C. Davey

Subject Categories

Educational Assessment, Evaluation, and Research

Abstract

In multistage-adaptive testing (MST), there are many interrelated design variables that impact the nature and quality of ability estimation. Previous research has identified general principles for the effective design of MSTs in terms of measurement performance. However, those principles are unlikely to apply uniformly to every testing context.

The purpose of this dissertation is to propose a process of finding an MST design that has optimal measurement properties, given a specific set of test circumstances. To achieve this goal, an efficient strategy was introduced at each of three phases to discover the optimal design of the MST; constructing MSTs, systematically searching a design space of the MST, and evaluating the MST performance. For the first phase, a top-down approach was applied in this study. For the second phase, a way to systematically search the parameterized design space of an MST was used. For the third phase, a new analytical evaluation method for MST was proposed.

In the dissertation, Study 1 proposed a new analytical evaluation method for MST. Using this new approach, measurement precision of ability estimation and classification accuracy could be derived analytically. The simulation results indicated that the new analytical method produced more exact measurement properties of an MST than the Monte Carlo simulation method. Therefore, the new analytical method would be the most efficient and competitive tool to asses measurement performance of an MST among other evaluation methods.

Study 2 proposed a process to find a design of an MST that shows optimal measurement properties applying the three efficient strategies, given a specific set of testing context. The process consists of four important features: (1) setting a testing circumstance and MST design space, (2) systematically searching the MST design space using the top-down approach, (3) analytically evaluating measurement performance of an MST, and (4) computing objective functions. The suggested process was applied to a real item pool from a large-scale assessment. The results of the application study provided evidence that the process could be generalized to more complex and realistic test circumstances to create optimal designs of MST.

DOI

https://doi.org/10.7275/14218580

Download
COinS