Evidence First Podcast

Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people with low incomes. I'm your host, Leigh Parise. Over the last 50 years, MDRC has studied a number of workforce development approaches and sectoral training programs have emerged as a promising driver of economic mobility for people with low incomes. These training programs prepare people for quality jobs in specific industries where there's strong employer demand and the opportunity for career advancement. Studies find that sectoral programs can meaningfully improve participants' earnings, yet not all programs have the support they need to apply the evidence in their context. And there are still lots of open questions, like why only some programs are effective and for whom these models work best?

Today, we'll talk about some exciting initiatives like the Economic Mobility Lab at MDRC and the Sector Training Evidence-Building Project with the Ascendium Education Group – projects that aim to answer some of these questions and address the challenges these programs face. Joining me are Dr. Jessa Valentine, Deputy Director of Learning and Impact at Ascendium Education Group, a collaborator and funder of MDRC's work, as well as Frieda Molina, Director of the Economic Mobility Housing and Communities policy area at MDRC. And Kelsey Schaberg, Research Associate in the Economic Mobility Housing and Communities policy area at MDRC. Jessa, Frieda, Kelsey, welcome to Evidence First. I'm really excited for our conversation today.

Jessa Valentine: Thanks. Great to be here.

Kelsey Schaberg: Thank you.

Frieda Molina: Thank you. Excited to be here.

Leigh Parise: All right. Great. So Frieda, let's start with you. So tell us a little bit about the Economic Mobility Lab. What's unique about the project? What are its goals? I think that'll be great for people to hear.

Frieda Molina: So the rationale behind the Economic Mobility Lab came from not only our research but the research of others. As you've stated Leigh, we have found through experimental evaluations that there are programs, particularly sector programs, that are a bright light in our workforce development domain. And they're a bright light because we found that they have had impacts in terms of increasing employment in the early years and then in the outer years for some of these programs, increasing earnings over time, suggesting that there's advancement going on. And while there are these programs that we can point to as being effective, there's uneven evidence across the field and it's not the case for all programs. So for us, that triggered an idea. MDRC has been in the business of providing technical assistance really since our founding. So what that means is that we have a staff who understand program operations and know how to use evidence to improve the practice of those programs.

So the Economic Mobility Lab was really born out of this idea of taking what we know from our research and our experience, providing technical assistance to help programs improve their practice so that over time they can continue to produce the type of equitable outcomes that we'd like to see for all learners and all customers. So the Lab was conceived of a place where we can learn about and help providers learn about their operations and improve them. So we're not making any, passing any judgment about whether they're effective or not. What we're trying to do is work alongside them to address an area of their operations that they haven't really had time to explore.

So what we do is we work with them to improve their practice fundamentally. We don't come with a preconceived notion about what they should be focused on. Instead, we use a framework to guide a process of exploration with them and so they identify areas that they want to improve and then we work together to identify what's really behind the problem or the pain point, if you will. And then to craft solutions around how we might address that pain point. And then test it to see, if we make some adjustments, whether that, in fact, does help to move the needle. So I would say in short, I think what the goal of the Economic Mobility Lab is to teach practitioners how to use evidence to diagnose and improve their practice to increase participant outcomes.

Leigh Parise: That's really great. Thank you. I love that description and I love some of the language that you used about there being some bright lights out there and that we're trying to figure out how to help shine for some of the other programs as well. So that's great. And both your description, and I'll say for my personal experience of being in MDRC's office when you've got lots of practitioners from the Economic Mobility Lab programs together in the office, I know that this work is really collaborative in nature. Can you say a little bit about what we hope the programs are going to learn from one another by going through this experience with us?

Frieda Molina: As I mentioned earlier, each provider selects an area of practice that they want to improve on. That may be changing how they do their intake. It could be changing how they build relationships between coaches and their participants, or even how they work with individuals after they've completed training. These are issues and questions that are common across providers even though they may address it differently. So by bringing them together, they're sharing in what they're learning and in the techniques they're using. And sometimes that sparks an idea from one provider to another about they might try something differently. So we recently had a chance to bring the providers together, we have six at the moment, to New York to our office, and it was really satisfying to see them huddling during lunch hour or what have you to talk about something they heard as they described what they were trying to accomplish with their work in the Economic Mobility Lab.

So they collaborate with each other, they collaborate within their own teams because we bring them together in teams. So what that means is we asked them to at least have two to four people that participate with us on a pretty regular cadence. It could be once every week or twice a month. And those teams can consist of a manager who oversees the program operations, frontline staff as well as career coaches and someone who really understands their data. So they're working collaboratively as a team, and they're working together with us as well as get a chance to meet with other providers in the network of providers in EML once a year to share ideas. And we're hoping that that also will begin to establish relationships that will long endure past the time that we finish working with them.

Leigh Parise: That's great. Thank you for sharing that. And I want to actually pick up on that last thing that you just said of making sure that certain things endure. So you talked about some of the relationships between the organizations, but can you say a little bit more about how we work with them to help make sure some of the things that they are doing or learning actually stick over time? So it sounds like one of the ways that we try to do that is to work not just with one person in an organization, but to try to bring together a whole team who might be involved in working with their participants. But I'm curious if there are other things that we try to do with them to help make sure that whatever they are learning or anything that they might be changing actually has some staying power over time.

Frieda Molina: What we're trying to teach them is a process of diagnosing and identifying and really learning, going deep into understanding, what's behind a particular area of their practice, not just from the data, though the data is really important, but the data doesn't always tell us why a particular pain point exists. So we're helping to coach them to use other types of techniques to explore a particular pain point. And it's the framework that we use. We have four phases where we're up front, we're coming up with what we call a focal area, an area that they really want to explore. And we use different diagnostic tools to learn more about that focal area. And once we think we understand the problem, then we're moving into a design phase where we're coming up with crafting solutions that can help to address that pain point. That may be bringing in human-centered design. That may be bringing in some data analytics, or other types of disciplines and other staff at MDRC to help us come up with the design based on the issue that we've identified.

And then we're testing that solution. We come up with particular ways to measure, once we test that solution, whether we see any change in outcomes, and then we make some decisions on whether we need to tweak what we've implemented and whether there's something else that they want to address. So it is definitely a process that we're teaching them that hopefully they will use in the future as they continue to improve their practice. So there's always something you can tweak and change to improve practice. I mean, that's really fundamentally what we think. Even the best-run organizations have things that they can always improve. So it's really a framework that we use and work alongside the providers to help implement that they see how to use this framework for the future.

Leigh Parise: I appreciate that, wanting to build that capacity and help to build some of that muscle that hopefully can stick long after we've partnered with the organization. So that's great.

Frieda Molina: One thing that I would also emphasize is that we also take a human-centered approach, which I think is really helpful for the staff of these programs to put themselves in the shoes of someone who goes through their program, and really think about it from their perspective and how they meet up with all the processes and all the steps and all the requirements and what message that sends to individuals. Sometimes that's where individuals fall out and for good reason that there are some processes that we have to do in a certain way, but they can be major hurdles or barriers for folks to endure and persist. So using a human-centered design approach, we really try to center our work in understanding how a participant, a customer, a learner, experience the program.

Leigh Parise: That sounds really thoughtful and I'm sure the programs appreciate the opportunity to walk in the participant's shoes. Okay. So Jessa and Kelsey, let's turn to the STEP initiative. So again, that's the Sector Training Evidence-Building Project, or STEP for short. And it's a collaboration between Ascendium Education Group and MDRC. And STEP will select, fund, and support projects that will advance high-quality research to produce evidence that is both relevant to individual providers who will be part of these studies and also to the field at large. I know, Frieda, you said when you bring the programs together that they find a lot of common questions and challenges in the things that come up for them. So Jessa, it would be great to hear from you, why is investing in this area of research in particular a priority for Ascendium?

Jessa Valentine: Sure. I'm happy to speak to that. Well first, let me say just a little bit about Ascendium and our philanthropy arm for listeners who may be less familiar. Ascendium is one of the country's largest postsecondary education and workforce training philanthropies. And all of the work that we do at Ascendium in partnership with our grantees is really driven by the strong belief that education and training after high school gives people the power to build better futures for themselves and for their families. So we make investments that aim to strengthen postsecondary education and workforce training systems so that more learners, particularly learners from low-income backgrounds, can access and succeed in their education and training as a path to good jobs and upward mobility. Ascendium is also an organization that's committed to learning and impact and that's committed to building evidence that's needed and that can help us and that can help the field understand what's working in terms of programs and practices that can improve outcomes for low-income learners.

So coming back to the topic of sectoral training, this is an area where Ascendium has made a number of investments over the last three to five years. We've made grants to some of the nation's leading nonprofit-led sectoral training providers to support them in their efforts to provide pathways to quality jobs for participants. And these investments have been motivated in part by the strong evidence base that exists already regarding the effectiveness of some of these programs. Frieda spoke to this a little bit already, but through existing research we know that sectoral training programs can offer a path to upward mobility when looking at participants gains in particular.

And then at the same time, we also know the research that does exist has been fairly concentrated on a relatively small number of providers. And we know based on feedback that we've gathered from both researchers and from providers like Frieda was talking about, that there are many important questions that remain unanswered that are going to be really critical to address as providers seek to improve, as they seek to scale their approach to reach more learners.

Leigh Parise: Great. Thank you for that. In some ways, that's the higher level version of you might learn something, but there's always more to learn and to continue to ask the questions and think about where can you continue to advance what you're doing that applies for an individual program. But you're thinking about it even more broadly for the field. So I appreciate you walking us through that. All right. So Kelsey, what are the main questions that these research projects are going to try to get at?

Kelsey Schaberg: So we spent a lot of time thinking about the research questions that STEP was going to focus on in collaboration with Jessa and her team at Ascendium. I think there's so much that still needs to be learned in this area. So we wanted STEP to focus on questions that are broadly relevant to sector programs, and that are going to help fill evidence gaps that have been identified by researchers or going to fill needs that we've heard about from staff and sector programs. So after much discussion, as well as getting input from experts in the field, we narrowed down what was our initially very long list of research questions into two high-level research questions.

The first one is about what are the key elements that are needed to implement or expand access to effective sector programs? And by key elements we were thinking of everything from the specific program components or services that are offered by programs to the different target populations that programs seek to work with or the different approaches that programs are using to scale or strengthen their program models. And the thinking behind this question is that as we've talked about before, we've now had evaluations of several sector programs that have found that they can increase participants' earnings and those earnings gains can be sustained in the long term. But those evaluations that have been done were all of full program models. So they've evaluated the full package of services that are offered by programs.

And what we haven't done yet is study what parts of the model or what key elements of the model are really driving the effectiveness of sector programs. So we don't know things like whether it's the technical skills training that programs offer that is helping people get good quality jobs, whether it's the more general career readiness training that's offered or whether it's both of those things. So this question is really trying to get inside that black box and learn more about what sector programs need to focus on in order to be effective.

And then our second high-level research question is around what is the effectiveness of sector programs that haven't been evaluated before? And that can expand our understanding of the sector model. And the thinking for this question is, again, while we have had several studies of sector programs that have been shown to be effective, there have also been studies of other sector programs that haven't shown the same earnings effects. And we also know that the sector model is being widely implemented across the country. So we're interested in seeing if there are additional programs that are out there that seem promising and that as part of STEP can be evaluated to show that they're effective to build even more evidence for the model.

And in particular, we are trying to find programs that are working with a new target population that hasn't been studied before, or that are operating in a new industry or occupation or they're changing something in the way they operate their programs in order to reach more job seekers. So we think that the projects that fall under this question are going to build evidence around both the overall sector model as well as about the context in which it can be effective. And we think collectively across these two research questions, the evidence that is learned is really going to fill evidence gaps, and is really what the field needs now and has been asking for and is going to help us continue to strengthen the implementation of sector programs going forward.

Leigh Parise: That's great. Thanks so much, Kelsey. I want to pick up for a second on that last part of what you were talking about regarding the research questions. So it does feel like in the sector space we often hear about some of the same big players who've done incredible work over really long periods of time. But it sounds like you think there are a lot of programs out there where there may be opportunities really to build some additional evidence or expand their evidence or think about who other players in the field might be. And it might just be useful for you to say a little bit about the extent to which you think there's a lot of opportunity in this space because it sounds like you're pretty optimistic about that.

Kelsey Schaberg: Yeah. That's a great question. So yes, we hear the same names over and over again because those are the ones that have been studied, have really drawn and gotten the attention for the sector model. And now that sector strategies have been shown to work, a lot of other programs have said, "Okay. This is what we need to do now." So they have started trying to figure out how to implement this model, how to really embed themselves in the sectors in which they work. So there's a lot of new programs that have just gotten set up in the last few years and have been implementing what we know currently about sector strategies. There's still more to learn, but I think expanding this model more is really... We need to test it again to see if it's going to work in other contexts.

And a lot of people always ask like, "Does it just matter what sector a program's working in and that's really what's driving the effectiveness of it?” And I don't think we think that's the case, but we have studied a lot of programs that operate in the same sectors, and so we want to know, for example, if it works in other sectors too, what may need to be changed for the program to be able to, or the model to work in those sectors. So I think that's what we're really hoping to learn in STEP is how it can work in different contexts and also work with different types of job seekers.

Leigh Parise: All right. That's great, Kelsey. Thank you. I'm really excited for what's to come. So tell us a little bit about MDRC's role on this project. How are we using what we've learned about sectoral programs to design the initiative and to select the projects that are going to build the evidence in the field?

Kelsey Schaberg: Yeah. So MDRC is playing a somewhat different role in STEP than we typically play on a lot of our projects in that we are not actually leading any of the formal research projects here. And instead, we are managing and facilitating the implementation of STEP. So this includes everything from when we first started working with Ascendium, where we were developing the parameters and the design of STEP to when we reviewed and scored all the applications and proposals that were received. And we're also now working to select the projects that are ultimately going to be funded as part of STEP, and then we'll be entering into agreements with those teams. And we'll also be leading a community of practice among grantees who are selected for STEP.

So while this isn't MDRC's typical role in projects, I think it's a really exciting role for MDRC and I think we're also uniquely positioned to be able to play this role. We have years of experience and knowledge from working with sector programs, both as part of rigorous evaluations and in the ongoing work that Frieda was touching on earlier, we're playing more of a support and assistance role with sector programs. We're helping them build their internal capacity, we're helping them build evidence that they can use to strengthen their day-to-day program operations.

So we're bringing all of that experience and knowledge to all aspects of the STEP project. It helped us narrow down and focus the research questions as we talked about earlier, so that we're really building this cohesive and relevant body of evidence. It's helping us think through how the proposals that we've received are both going to, as individual projects and collectively as a group of projects, fill evidence gaps and advance the field. And I also think it's going to help us as we move forward and start having learnings from the projects, it's going to help us think about what the cross-project learnings are and what the cross-project takeaways are so that we can make sure that the evidence that is coming out of these projects gets into the hands of sector program staff and others in the field, so that the evidence is really actionable in helping to improve sector program operations.

Leigh Parise: All right. Thank you, Kelsey. Okay. So Jessa and Kelsey, can you tell us a little bit more about how we're thinking then about bringing the partners together to make sure that they're learning from one another?

Jessa Valentine: Yeah. I can start us off on this one. So in the early stages of thinking through what this initiative could look like, one thing that felt really important to us is that we wanted these research and evaluation projects to involve true partnerships between research teams and training providers. In the research field as you know, these types of collaborative partnerships are sometimes referred to as research practice partnerships or RPPs. And when funding research or evaluation projects at Ascendium, we tend to prioritize these kinds of collaborative partnerships since of course the ultimate goal of any evaluation is that the findings get used and that actions can be taken based on the findings to further improve programs, to further improve practices for the benefit of the folks that they're serving.

So time and time again, we see that evidence is more likely to be translated into action when these partnerships exist. So for STEP, each of the research projects that are selected are going to be comprised of these partnerships between researcher teams and sector providers. And then the initiative structure itself, which is going to be supporting a cohort of projects, creates a really exciting opportunity for these partnerships to learn from one another. And I’ll let Frieda talk a little bit more about what that's going to look like.

Frieda Molina: Sure. So building on what Jessa said, the design of STEP has always been to develop actionable research. So not just research that sits on a shelf that some folks may read that report, but research that practitioners and the broader policy field can really absorb and understand about how these programs work and operate and how to improve in the future other programs. So we want the practitioners, and the researchers in this cohort of grantees to be speaking to one another as well, and to incorporate what they're learning in their everyday work. So we'll be convening the grantees a couple of times, three to four times across the period of the STEP initiative in a community of practice. We don't have specific topics right now that we will be addressing in that community of practice. That will be generated by the range of grantees that are in the initiative and the common experiences or even common questions that they're seeking to answer.

So some ideas might be, for example, how do you incorporate some of the participants into the research and program design? Or what are the best ways to disseminate this information to get to a practitioner audience? Or what are we learning about some of the program components the research teams are investigating? So after we select the grantees and after they're selected, we will then be identifying commonalities across the grantees and figuring out whether there are a couple communities of practice based on common themes that they're exploring. So, to be determined, but we really do want to have this community of practice come together, and really think about what we're learning collectively as a field.

Leigh Parise: I really love that. I have to say as a researcher, I feel like I would've loved to have had the opportunity to learn from other practitioner, researcher teams while my study was ongoing. After the fact, you sometimes will go talk to people about what you learned and what was challenging and what you found. And getting to do that along the way and hearing how other people are handling what I imagine will be some similar challenges, partly because they're going to be operating their studies in the same context of the world that's going on. So they're going to be experiencing a lot of similar policy challenges or challenges that are just happening more broadly. So that is really exciting, and I look forward to hearing how all of that goes.

All right. Jessa, Kelsey, Frieda, thank you so much for joining me. This has been a really great conversation and exciting that this is all work that is ongoing or just starting, so we'll get to hear a lot more about it. Thank you. To learn more, visit mdrc.org. Did you enjoy this episode? Subscribe to the Evidence First podcast for more.