Many organizations are committed to learning: engaging in continual improvement and demonstrating that the services they provide make a meaningful difference for the people they serve. However, it may not be easy for them to focus on improving their use of data and research when their primary focus is understandably on serving clients and managing their day-to-day operations. Throughout its history, MDRC has helped organizations to identify when they are ready for rigorous evaluation—and to use “evaluation roadmaps” to move toward learning and program improvements, whether or not they engage in a rigorous evaluation. This spotlight illustrates how MDRC partnered with one such organization to further its research readiness and, through that process, help it improve its program.

The Opportunity

Recent data from the National Assessment of Education Progress show reading and math performance among twelfth-graders at historic lows, and gaps between low and high performers growing.^[1] In this context more than ever, educators, funders, and policymakers want to know whether the educational programs they have been supporting are effective. MDRC works with organizations to travel along the road toward rigorous evaluations that can answer these questions. Some may be at a stage where it makes sense to refine their logic models (representations of their models that show the expected cause-and-effect relationships between their activities and expected outcomes) and to align their data systems with those activities and outcomes so they can monitor their service delivery. Others further along in their learning process may be ready for a rigorous evaluation. Ultimately, demonstrating that a program results in meaningful and positive change can lead to increased funding and allow an effective organization to expand its reach to more individuals and communities.

Cotaught classrooms—in which a general education teacher and special education teacher collaborate to meet the diverse learning needs of students in their classroom—are a promising model for effectively supporting all learners. However, coteaching well can be challenging. MDRC’s partner Blue Engine helps schools realize coteaching’s potential by providing structured coaching to coteaching teams and administrators, focusing on shared planning, instructional coherence (the alignment of curriculum, teaching methods, and assessments), and the continual use of data to refine practice. Blue Engine also uses data to inform its own practice: it gathers data on classroom collaboration and student outcomes systematically and uses the data to adapt its coaching approach, ensuring that schools implement coteaching practices as intended and effectively. MDRC has been collaborating with Blue Engine to build a learning agenda and prepare for an eventual impact evaluation of its model, and, along the way, MDRC and Blue Engine have been working on program improvement.

MDRC’s Response: Building a Learning Agenda

MDRC and Blue Engine began their partnership by engaging in preliminary steps as described in the roadmap. Doing so meant identifying the core features of the program itself and the outcomes the program expected to affect, as well as how each would be measured, drawing on MDRC’s deep expertise in implementation research, which involves codifying how a program is expected to work and what it is trying to achieve. The process also allowed Blue Engine to calibrate expectations with its staff about which aspects of the model were fixed and which were flexible. With this information, MDRC and the Blue Engine team generated a logic model that laid out how program components linked to educator practices and mindsets, which in turn linked to short- and long-term outcomes for students.

Tracking program activities and their expected outcomes helps an organization determine how well a program is working, creating a system to monitor and refine its service delivery whether or not it ultimately initiates a formal evaluation. To that end, MDRC and Blue Engine developed a performance-measurement framework built on the logic model: a system for measuring success with standard metrics to track model implementation and potential outcomes. MDRC proposed metrics based on existing research, when available, and then the Blue Engine team reviewed those proposed metrics internally with its staff to ensure they felt achievable. Blue Engine and MDRC also reviewed the organization’s existing data-collection and -reporting efforts, to gauge how well they would be able to monitor those metrics together. While Blue Engine had existing data to measure student and teacher outcomes, MDRC and Blue Engine discovered that together they needed to standardize metrics related to delivering coaching as well as school-level staff members’ engagement in coaching activities. New data systems that track Blue Engine’s activities with school staff members have helped the organization monitor its delivery of services on a large scale, and will be essential for assessing implementation.

Implications for the Field

The way MDRC helped Blue Engine move through steps in the Readiness for Evaluation Roadmap is an example of how MDRC can assist educational or social program providers to learn better, informing their own practices and preparing them to demonstrate their effectiveness whether through rigorous evaluation or ongoing program monitoring. MDRC knows how to guide organizations through the process of defining their models, identifying metrics for measuring implementation and outcomes, and developing formalized systems for using their data to track program success. This type of effort helps organizations learn more about how their services make a difference for their participants and how they can improve their programs, while serving as critical preparation for a strong test of a program’s effectiveness.