Adapted from a speech given at "Lancement du Grenelle de l'insertion: Les rencontres de l'experimentation sociale," a conference sponsored by the French government in Grenoble, France, to discuss the role of experimental studies in reducing poverty and helping the unemployed reenter the labor market.

Thank you. It is an honor to be a part of this historic meeting of leaders committed to developing, designing, and experimenting with new approaches to help the unemployed reenter the labor market. As an American, it is also a humbling experience. France has been a trailblazer in social welfare policy: from its remarkable system of child care to its commitment to comprehensive health care for all. While I have been invited to share the U.S. experience with experimentation and the role that MDRC has played, I also welcome the opportunity to learn from France's experience.

Our countries share many of the same social policy goals. We want to help parents support their families and children, and we want to assist both citizens and newcomers to succeed in the labor market. But we begin from very different places. France has a much more comprehensive social welfare system than the U.S. does. And our labor markets differ in important ways. The U.S. has been able to create a remarkable number of jobs, but nearly half of our jobs pay low wages. France has done better at creating "good jobs" and offers far more worker protection, but it has produced fewer jobs.

Because the context for policymaking in the two countries is very different, I'm not here to hold up the U.S. as a model to be emulated. Instead, I want to describe how we have used social policy experimentation to tackle employment and welfare problems. It is a remarkable story. But only you can decide the relevance of that story for France.

To help you appreciate the role that experimentation has played and is playing in the U.S., I have organized my remarks into four parts: First, I briefly describe MDRC and its mission. Second, I explain what "experimentation" means in the U.S. context. Third, I'll describe how experiments were used as a lever to help change American policy. Finally, I'll profile a demonstration that we have just begun in New York City that was inspired by a successful experiment in Mexico and adapted to the U.S. context.

Let me begin by previewing my conclusions:

• Social experimentation can be a powerful tool for bringing about change, but it works best when conditions are ripe for change and when there is a widely shared commitment to learn both what works — and what does not.



• Experimentation is seldom a panacea: program effects are often modest, some things do not work, and change takes time. But the alternative — failing to build a record of what works and what does not — leaves one to make policy on the basis of anecdote and ideology, and thus to repeat past mistakes.



• Experimentation leaves its greatest legacy when it builds both reliable evidence about what works and the program capacity to deliver effective services.

 What Is MDRC?

MDRC is a nongovernmental organization that was created by a consortium of federal government agencies and a private foundation. MDRC's mission is to learn what does and does not work in social policy. It was founded in 1974, a time when American policymakers wanted better evidence that social programs actually worked and that the benefits of these programs exceeded their costs.

The idea was to design, develop, and test new program ideas, using rigorous research methods to learn whether programs worked before making them national policy.

Our goal was twofold: We wanted to build infrastructure — by which I mean the capacity to implement programs in multiple locations and at scale. And we wanted to build reliable evidence about what worked.

What Does "Experimentation" Mean in the U.S. Context?

For MDRC and other organizations (including Mathematica, Abt, RAND, and Public/Private Ventures), experimentation does not mean just trying new things but also learning whether programs attain their goals.

To build evidence, MDRC and its colleagues had to answer several questions: What difference did the program make? How and why did it work? If a program did not work, knowing why was critical to designing a better one. If a program did work, knowing how was crucial to replicating the intervention on a larger scale.

But reliably determining whether a program is effective is challenging. People's lives are dynamic; they take and lose jobs, enter and leave public assistance programs, marry and divorce. To know whether a particular program caused a change in employment, we have to know what would have happened if the program did not exist. Did someone leave public assistance and take a job because of the program or because they would have done so anyway? To determine the net difference a program makes, one needs a counterfactual, a comparison (or control) group of similar people that shows us what would have happened in the absence of the program.

The most reliable way to create a counterfactual or control group is to use a random assignment research design — widely accepted as the "gold standard" — essentially the same research method used in medical research to determine the effectiveness of a new medicine. Random assignment uses a lottery-like process to create two groups that do not differ systematically — except that one is eligible for the new program and one is not. By identifying a pool of eligible people, and then randomly assigning them to a program group that is eligible for the new services or to a control group that is not, any subsequent difference in outcomes between the two groups — say, employment rates — can be confidently attributed to the effects of the program. Random assignment designs are fair — everyone has an equal chance to participate in the program. They are also ethical when: (1) you don't know if the intervention works, (2) there are not enough resources to serve everyone, and (3) the service in question is not an entitlement. Informed consent to join the study is usually required. The results from random assignment studies have the virtue of being simple to understand, and, when implemented well, such studies are seldom challenged.

While random assignment can sometimes be controversial, American researchers have shown that it is feasible to implement. For example, MDRC has conducted at least 40 large-scale randomized controlled trials in more than 300 communities in the past 30 years, involving more than 400,000 people. And such social policy experiments are not just a U.S. phenomenon. In the past 10-15 years, large random assignment studies have been successfully undertaken in Canada, Mexico, and the United Kingdom.

Why this preoccupation with evidence? In the starkest terms, if you want to make the world a better place, public policy has to actually do things that make a positive difference. When that doesn't happen, the result is cynicism about government among participants and among taxpayers.

Let me illustrate some of the opportunities and challenges of social policy experimentation by describing how the U.S. used experimental evidence to reform its social welfare system.

The Welfare-to-Work Example

The American welfare system was created in the 1930s as a response to the Great Depression. A new program called Aid to Families with Dependent Children was established to provide cash assistance to widows so they would not have to work and could stay home with their children. The program remained small for nearly 30 years, and then suddenly, in the 1970s, rates of out-of-wedlock childbearing began rising, the welfare rolls began to grow rapidly, and welfare costs began to increase. At the same time, many more women were entering the labor market. By the mid-1980s, more than half of all mothers with children were working. This situation raised questions about basic fairness: Why should some work for a living, while others got help from the government?

Political controversy raged. The right maintained that the welfare system was anti-work and anti-marriage and was hurting families more than it was helping; the left countered that every family was entitled to a basic level of income and support. Conservatives wanted to restrict eligibility for welfare; liberals favored higher benefits. The poor themselves preferred work and also despised welfare; it intruded in their lives and society looked down on them. The system was ripe for change that would better align social welfare policy with the bedrock American values of work, independence, responsibility, and family.

Somewhat surprisingly, a group of conservative and liberal leaders (including President Ronald Reagan and then-Governor Bill Clinton) supported legislation that gave states the right to reform welfare rules — in return for welfare benefits, recipients would have to prepare to find work, but the system would also offer new employment services and other supports to help them do so. The federal government provided the funding and the flexibility, and states and localities provided the program structure to deliver services.

But two key elements were missing from this political consensus for reform: knowledge about what interventions to try and a plan to capture evidence from all this innovation. MDRC secured a grant from the Ford Foundation and support from federal research agencies to work with the states in developing and evaluating these new reforms.

Eight states participated in what came to be known as the Demonstration of State Work/Welfare Initiatives. In return for participating, states received:

• In some cases, funding to help support their programs and offset data collection costs.
• Technical assistance from experienced MDRC staff.
• The opportunity to meet with and learn from their counterparts from other states.
• Formative feedback about their programs as the evaluation progressed.
• An independent, credible, and rigorous random assignment evaluation that would provide reliable evidence about their program's benefits and costs.
• National visibility.

It was the beginning of an extraordinary long-term partnership between government and nonprofit service agencies seeking to reform welfare and researchers attempting to assess the effectiveness of their new programs.

Eventually, 11 research and demonstration projects were begun, involving the random assignment of about 65,000 people to program groups that would receive the new services or to control groups that would not. Before these Work/Welfare Demonstration projects were initiated, state welfare agencies had focused on administering a cash benefit program — checking eligibility, protecting against fraud, making sure that the benefits were paid on time and without errors. Now the philosophy and the focus began to shift — the goal was to help people make the transition from welfare to work. The programs tried a wide range of approaches singly and in combination, including intensive job search, temporary public jobs, and short-term education and training. They also offered a range of support services, including counseling, child care, and other supports. Participation was mandatory for able-bodied welfare recipients whose children were six years of age or older.

What did we learn?

• It was feasible to operate these programs at scale, but participation levels varied across the sites.
• Participants thought the work requirements were fair, and most said they preferred work to welfare.
• The programs worked: employment and earnings increased in 9 of 11 demonstration sites, up by 10 to 35 percent relative to the control group.
• The programs were cost effective: welfare receipt fell and the resulting welfare savings exceeded the costs of running the programs — a $3 return per $1 invested in some programs.
• There was no evidence that children were harmed when their mothers went to work.
• Income generally did not increase; in these programs, welfare recipients traded a welfare check for a paycheck. Once they went to work, government assistance ended, with no net change in participants' overall income.

Although the programs did not help everyone, these findings (combined with those from studies begun by other organizations) had a profound impact on the political debate about welfare reform. Now both the right and the left had to argue within the bounds of the evidence. And the evidence challenged long-held beliefs of both groups. Conservatives learned that social programs could work and that the benefits could exceed the costs. Liberals learned that work mandates and requirements could produce positive effects, that children were not harmed by welfare-to-work programs, and that participants thought the programs were fair and preferred work to welfare. No longer could policy be based only on anecdote and ideology.

After two decades of failed attempts to reform the welfare system, this new consensus — based on experience and evidence — led to passage of federal legislation, the 1988 Family Support Act. As a Congressional staffer explained at the time: In all the years I worked on welfare reform, we never had a body of data that showed what worked...For the first time, we could characterize reform as an investment.

Importantly, while the results of experimentation told the nation what worked, it also showed what did not work, and what we still did not know. Remarkably, the 1988 welfare reform law included funding for a next generation of experiments that would use random assignment research methods. These and related experiments answered such questions as:

• What would help people with employment barriers who were left behind?
• Would more investment in education and training help people get better jobs?
• What would happen if the government supported people when they worked by supplementing their low wages?

Over the next decade, a new round of experiments was launched to answer these questions. Because it is so central to your objectives, I want to briefly summarize what we have learned about the third question, providing support for people when they work.

Because the U.S. public would never make welfare for those who did not work generous enough to lift them out of poverty, policymakers began to recognize that getting families out of poverty required both earnings and government support. Three experiments with earnings supplements (two in the U.S. and one in Canada) set out to answer the question: What would happen if we built supports around work rather than non-work? All three provided work incentives in the form of monthly cash payments to supplement the earnings of low-wage workers. Payment was conditioned on full-time work, and the payment amount depended on the amount of each month's earnings. Nearly 15,000 people participated in the three experiments; all used random assignment research designs.

Despite differences in program rules and differences in local labor markets and economies, results across the three projects were nearly identical:

• Incentives work: All three programs increased employment, earnings, and income, and they reduced poverty relative to a control group not offered earnings supplements.
• Marketing matters: Incentives work best when they are clearly communicated and people understand what behavior is being rewarded.
• Children benefited: Earnings supplements led to improvements in young children's school performance, partly because the family had more income and partly because the children were placed in higher-quality day care programs.

These findings stood in stark contrast to the welfare-to-work findings I described earlier. Earnings supplements clearly made participants better off economically, but they cost the government more. In contrast, the welfare-to-work programs described earlier saved the government money but did not make their participants better off financially, although participants generally preferred earning their income from work rather than from welfare handouts.

Earnings supplementation programs were subsequently put in place in most states, and the federal government continued to enhance the Earned Income Tax Credit, an earnings supplement program for low-wage workers that now costs more than $40 billion per year, making it the largest income assistance program in the U.S. Payment of this credit is conditioned on work. In short, earnings supplements and forms of incentive payments were effective in a variety of settings in the U.S., in Canada, and more recently in Mexico and the United Kingdom, and possibly, as the next speaker will describe, in Germany.

This brings me to the last part of my story, the test of conditional cash transfers that is just now getting underway in New York City.

New York City's Conditional Cash Transfer Program: Opportunity NYC

In New York City, more than one-third of children live in poverty, most growing up in single-parent households. Two years ago, New York City Mayor Michael Bloomberg established a commission of key stakeholders to make recommendations about alleviating poverty. Alarmed by the long-term decline in the real value of earnings, impressed by the results of the experiments I just described, and intrigued by the results of Mexico's Progresa (now called Opportunidades) program, the Mayor's Poverty Commission recommended that the city consider experimenting with a version of the Mexican program in New York. Mayor Bloomberg agreed, arguing that bold new action was necessary but that it should be tried on an experimental basis to learn whether it works before launching it for the entire city.

As you may know, Progresa was designed to address three problems: (1) poor nutrition that led to stunted growth among newborns, (2) children leaving school at very young ages to work in the fields, and (3) a grain subsidy system that was not efficiently targeting government resources to the poorest families. Progresa replaced the grain subsidy system with a relatively generous cash transfer that was conditioned on children staying in school, on their parents participating in nutrition classes that taught them how to improve their diets, and on the families getting regular health check-ups. Researchers randomly chose eligible communities to get the new program or to serve as the control group. After two years, they found that children in the program group were somewhat taller and better nourished than those in the control group, that children in the program were remaining in school for an additional year or two, and that families in the program were getting more preventive health care. On the basis of these findings, the Mexican government moved to expand the program nationwide.

Clearly, poverty in New York is not the same as poverty in rural Mexico. Nevertheless, the conditional cash transfer concept was adapted by Mayor Bloomberg to address poverty in New York City. A new pilot project with a rigorous evaluation component, Opportunity New York City will pay a cash supplement every two months to participants when they meet requirements in three areas: children's school performance; parents' full-time work and participation in education and training; and family health care and insurance, including getting annual check-ups. By meeting all the conditions in the program, a family could receive as much as $5,000 to $6,000 a year in conditional cash transfers.

The program's goal is to reduce family poverty in the short run, to help families obtain self-sufficiency through increased work and training over the intermediate term, and to reduce intergenerational transfer of poverty over the long run by increasing the school performance and graduation rates of school-age children.

Nearly 5,000 families in six high-poverty neighborhoods have agreed to participate in the study. Half of these families have been selected randomly to participate in the conditional cash transfer program; the others are in the control group, which is not eligible for the transfers. The program will operate for two to three years and the research will follow both groups of families for five years. If the program increases work, schooling, and health, the Mayor has committed to trying to expand it.

Conclusion

To summarize, when MDRC first began in the early 1970s, common sense, good intentions, and good ideas were the basis for making social policy in the United States. Reliable evidence from random assignment studies of large-scale operating programs was thought impossible. But poverty, unemployment, and welfare dependency persisted. The public, the government, and the poor began to see that good ideas and common sense were not enough. Resources were scarce, and the government was reluctant to expand pilot programs without evidence of their effectiveness. Thirty years later, the U.S. has learned that random assignment is possible, that you can learn what works, and that experimentation can be a powerful tool for social change.

Indeed, in the welfare-to-work story, where researchers were working in partnership with states and nonprofit agencies to build both the capacity to deliver services and evidence about which services were most effective, experimentation was decisive. Importantly, the political atmosphere was also ripe for change — the debate was fierce and the two sides were very far apart, but both wanted reform. Experimentation helped show the way. With earnings supplements, experimentation and evidence played a valuable but secondary role. The shift from a system that supported able-bodied people when they did not work to one that provided its most generous supports when they did was already underway. Evidence from experimentation helped to solidify and expand support for this shift.

Experimentation is not a sure thing. Impacts are often modest rather than dramatic, you will learn things that surprise and disappoint you, many things won't work, and progress requires a long-term commitment to learn from and build on your experience. But one must begin with the conviction that evidence matters — that knowing what doesn't work and what does are both key to improving the lot of the poor and the return on government investment.