Phase 3: Interpreting the Results
Interpreting the results from an evaluation of a postsecondary intervention can be challenging:
- If an intervention causes students to earn 1.4 more credits, is that a small, medium, or large effect?
- Is a 5.7 percentage point effect on retention something to get excited about?
Empirical benchmarks offer one approach to answering these questions by placing an effect estimate in context—by comparing it with the effects of other interventions that were evaluated and that target the same outcome.
While many excellent empirical benchmarking tools exist, most are designed for K-12 education. THE-RCT’s empirical benchmarks app (and its associated article) is built specifically for postsecondary interventions.[1] It focuses on key outcomes such as credits earned, enrollment rates, and graduation rates, and it shows where a user-inserted effect falls relative to the distribution of effects from a set of past randomized controlled trials (RCTs). The app can help users interpret their findings with greater clarity.
[1] Michael J. Weiss, Marie-Andrée Somers, and Colin Hill, “Empirical Benchmarks for Planning and Interpreting Causal Effects of Community College Interventions,” Journal of Postsecondary Student Success 3,1 (2023): 14–59, website: https://journals.flvc.org/jpss/article/view/132759/138754.
[2] Carolyn J. Hill, Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey, “Empirical Benchmarks for Interpreting Effect Sizes in Research,” MDRC Working Papers on Research Methodology (MDRC, 2007), website: https://www.mdrc.org/sites/default/files/full_84.pdf.
[3] Matthew Kraft, “Interpreting Effect Sizes of Education Interventions,” unpublished paper (Annenberg Institute at Brown University, 2019), website: https://edworkingpapers.com/sites/default/files/Interpreting%20Effect%20Sizes%20-%20August%2019%20FINAL_0.pdf.
Key Resources
Tool
THE-RCT’s Empirical Benchmarks
Helps contextualize effect sizes by comparing them with results from prior RCTs at community colleges
Journal article
Benchmarks for Interpreting Effects of Community College Interventions
Explains the background, analyses, and findings that go into THE-RCT’s empirical benchmarks app
Journal article
Empirical Benchmarks for Interpreting Effect Sizes
Introduces benchmarks for interpreting achievement effects in education RCTs, improving on Cohen’s traditional rule of thumb[2]
Working paper
Interpreting Effect Sizes of Education Interventions
Proposes effect size benchmarks for RCTs in education that account for study design, cost, and scalability[3]