Integrating Implementation and Impact Research
While it’s critically important to find out whether a program or policy is making a difference, that knowledge is not enough. Understanding how and why programs work (or don’t work) is ultimately what policymakers and practitioners want to know. MDRC has been a leader in using high-quality implementation research — both quantitative and qualitative — to understand impact findings. Here are just three examples:
- In 2003, Howard Bloom, Carolyn J. Hill, and James A. Riccio published an influential article in the Journal of Policy Analysis and Management that analyzed a unique — and extensive — dataset from three large-scale multisite random assignment evaluations of welfare-to-work evaluations to examine the relationship between program implementation and effects on short-term earnings. Using data from nearly 70,000 sample members in 59 locations in seven states, Bloom, Hill, and Riccio were able to get “inside the black box” to explain why the programs performed as they did. They also helped make the case for investing in random assignment experiments with strong implementation studies and comparable measure and methods to build knowledge about what works best for whom.
- Today, MDRC is combining the latest methods to better understand how programs are implemented and how programs might affect participants. For instance, in the Supporting Healthy Marriage Project, which is testing relationship skills education in eight sites, MDRC developed a comprehensive MIS system for programs to report on recruitment and participation, videotaped couples interacting with each other to measure real-life communications and dispute resolution, and gave hand-held devices to program participants to allow them to keep simple daily diaries of their interactions with their spouses and children. Together, this information gives us a rich source of information to explain how and why the program might make a difference.
- Finally, MDRC is at the forefront of understanding what factors are critical in scaling-up proven and promising programs, a focus of much federal and philanthropic investment.
