Exploring Rater Effects for Direct Behavior Ratings and Multi-item Scales

This session is from the NASP 2022 Annual Convention

  • PMD: 1,4
  • Skill Level: Intermediate

Learner Objectives

This session will help participants…

  1. use direct behavior rating - multi item scales appropriately for progress monitoring.
  2. understand the strengths and limitation of indirect measures of student behavior.
  3. understand the nuances of how raters affect scores on direct behavior ratings.


Although indirect measures are common, they are often affected by idiosyncratic rater perspectives. Yet comprehensive analysis of these rater effects lags in social and behavioral progress monitoring. Thus, the pupose of this study was to use generalizability theory and many-facet Rasch measurement to evaluate rater effects on two Direct Behavior Rating Multi-Item Scales (DBR-MIS) in a sample of 126 second- through fifth-grade students. Results indicated problematic rater effects but also pointed to avenues for future improvement.



Christopher J. Anthony, University of Florida
Kara M. Styck, Northern Illinois University
Robert J. Volpe, Northeastern University


Letter to Supervisor

Looking for a concise way to pitch the convention to your employer? This letter and the corresponding talking points are here to help!