Off-campus UMass Amherst users: To download dissertations, please use the following link to log into our proxy server with your UMass Amherst user name and password.
Non-UMass Amherst users, please click the view more button below to purchase a copy of this dissertation from Proquest.
(Some titles may also be available free of charge in our Open Access Dissertation Collection, so please check there first.)
The generalizability of systematic direct observations across items: Exploring the psychometric properties of behavioral observation
Abstract
Direct behavioral observation of children in a classroom setting has become a required assessment procedure by school psychologists in order to determine possible out of class placement, services, and/or interventions. However, the reliability of direct behavioral observations has come under criticism. The purpose of this study was to determine the generalizability of systematic direct observation across items. In this study, a partial interval and momentary time sampling observational system is used in which 102 second grade children are observed during math for on/off task behaviors for 15 second intervals for 15 minutes. Data from this study were collected from two Western Massachusetts Elementary Schools and two elementary schools located on Cape Cod. Generalizability theory was employed to determine how many 15 second intervals are needed for reliability. Repeated measures analysis of variance (ANOVA) was conducted to obtain the variance components for the three sources of variability (i.e., persons, items, items x persons, and residual). Data were analyzed in two ways. The first looked at a simple definition of on and off task behavior using a momentary time sampling procedure. It was determined that the majority of variance was attributed to error (88%). Person variance or the differences among individuals did not contribute much to the variance (12%). Items did not contribute to the variance. Gabsolute and Grelative were both .88 indicating high dependability. A second analysis explored a more multidimensional definition of on and off task behavior using both a momentary time sampling and a partial interval procedure. Similarly, the majority of variance contributed to error (82%). Person variance only accounted for a small portion of the variance (18%) and items did not contribute to the variance. Gabsolute and Grelative both increased to .93. Due to the large amount of error in both analyses D studies were not able to be conducted to determine the number of items necessary to obtain a dependable sample of behavior. Further interpretation of results from a behavioral assessment perspective, implications for practice, and future directions are discussed.
Subject Area
Educational psychology|Quantitative psychology
Recommended Citation
Clark, Tara M, "The generalizability of systematic direct observations across items: Exploring the psychometric properties of behavioral observation" (2008). Doctoral Dissertations Available from Proquest. AAI3325140.
https://scholarworks.umass.edu/dissertations/AAI3325140