Overview

Success for All (SFA) is one of the best-known elementary school reforms. But is SFA effective at boosting student achievement? To find out, researchers randomly assigned 37 schools (i.e., “clusters”) within five school districts (i.e., “blocks”) to receive SFA or not and tracked students’ academic progress.

Such multisite cluster randomized trials can help us learn about a program’s or policy’s effectiveness. Researchers typically characterize such effectiveness with estimates of the average treatment effect for a specified population. Even this seemingly simple concept of an average treatment effect has nuances. For example, do we care about the effect of SFA for the average student, for the average school (i.e., cluster), or for the average district (i.e., block)? These targets of inference are known as estimands, and depending on which estimand we care about, the true average treatment effect can be different. Moreover, there are a surprising large number of ways to estimate the average treatment effect and to assess how precisely it is estimated. These different ways of estimating the average treatment effect and its precision are called estimators.

This project has multiple goals. First, in a technical document, we will provide a comprehensive overview of six estimands and 30 estimators that can be used in multisite cluster randomized trials. Next, in a paper providing practical guidance to applied researchers, we will use many of these estimators on secondary data from 11 large-scale evaluations so that researchers can appreciate the extent that the choice of estimand and estimator matters, with respect to the estimated average treatment effect and its associated standard error. We will also conduct a complementary set of empirically based simulations, using them to unpack some thorny issues about the estimators that are difficult to understand without complete knowledge of the data generating mechanisms. Finally, we will release an R package for researchers interested in being able to implement the various estimators we examine.

Like prior research by Miratrix, Weiss, and Henderson (2021) for multisite individually randomized trials, the ultimate goal of this work is to provide conceptual and practical guidance to researchers implementing, analyzing data from, and interpreting findings from multisite cluster randomized trials.