Phase 2: Implementation and Analyses
Implementation research plays a vital role in postsecondary impact evaluations. It deepens researchers’, policymakers’, and practitioners’ understanding of what works, what does not, and why, while also supporting the replication of effective interventions. Within causal evaluations, implementation research focuses on several elements:
- Intervention plans are detailed specifications for all aspects of an intervention. These plans outline the intervention’s components and often define dimensions such as quantity (frequency, time, duration, reach), mode, and quality.
- Intervention implementation is how the planned intervention is put into practice. Understanding the kinds of support that are provided for implementation—such as technical assistance—can help people interpret effectiveness findings and is crucial to those seeking to replicate the intervention in new settings.
- Implementation fidelity is the degree to which services are delivered as planned. Fidelity is especially important when a program fails to achieve its desired effects, as it helps distinguish between implementation failure (planned services are not delivered) and theory failure (services are delivered correctly, but they are ineffective).
- Service contrast is the difference between the services that people who were offered the intervention receive compared with the services they would have received without the intervention. The service contrast is the driver of intervention effects on outcomes. Examining it closely helps people identify the “active ingredients” behind positive impacts and understand cases where changes in student experiences do not lead to improved outcomes.
- Context is the characteristics of the organizational or programmatic setting, the intended recipients, and the external environment. Relevant contextual factors depend on a study’s logic model, research questions, and design, and they can act as facilitators of—or challenges to—successful implementation.
There is extensive literature on implementation research and implementation science. The Institute of Education Sciences (IES) guide to implementation research and the conceptual framework from which it was developed are particularly valuable for researchers who are conducting randomized controlled trials because they explicitly link intervention implementation to intervention effects.[1]
[1] Carolyn J. Hill, Lauren Scher, Joshua Haimson, and Kelly Granito, Conducting Implementation Research in Impact Studies of Education Interventions: A Guide for Researchers, NCEE 2023-005 (U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2023), website: https://ies.ed.gov/sites/default/files/migrated/nces_pubs/ncee/pubs/2023005/pdf/2023005.pdf; Michael J. Weiss, Howard S. Bloom, and Thomas Brock, “A Conceptual Framework for Studying the Sources of Variation in Program Effects,” Journal of Policy Analysis and Management 33, 3 (2014): 778–808, website: https://cepa.stanford.edu/sites/default/files/workshops/pam21760_Rev.pdf.
Key Resources
Report
Guide to Implementation Research in Impact Studies
IES toolkit with steps, templates, and examples for planning and reporting implementation research
Journal Article
Framework for Understanding Variation in Program Effects
Framework that draws connections between intervention implementation and effects to help identify factors that explain variation in effects