Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people with low incomes. I'm your host, Leigh Parise.

State government agencies increasingly engage in data analytics projects designed to answer important research questions and improve policy and practice. But what does it take to carry out these projects, and how can agencies partner with external researchers to supplement their capacity?

In this episode, we discuss the partnership between MDRC's Center for Data Insights and the Washington Student Achievement Council, or WSAC, a state agency with the mission of increasing educational attainment in Washington. MDRC and WSAC are collaborating on several research projects designed to learn what works to help students enroll in and succeed in postsecondary education. To learn more about these projects and what makes the partnership successful, I talk with Mike Meotti, who's the executive director of WSAC; Isaac Kwakye, Senior Director of Research and Student Success at WSAC; and Rick Hendra, who's the director of MDRC Center for Data Insights. Welcome, Rick, Mike, and Isaac, it's so good to have you here in person.

Michael Meotti: Thank you. Glad to be here.

Leigh Parise: So Mike, Isaac, tell us a little bit about WSAC and the kinds of issues around higher education that you all are really focused on for the state of Washington.

Michael Meotti: Well, our Student Achievement council, or WSAC, as we call it in the state—all states have a state agency that focuses on higher education issues. And there're a lot of things we might just do as far as programs—like we run the state's financial aid program, we run the state's 529 college savings program, and we do technical assistance to guidance counselors and school counselors throughout the state. But the goal, our big picture vision, is to significantly increase opportunities for access and success for Washington residents to education and training pathways that get them where they want to in their lives. And that's not something that's done by a mere, you know, telling other organizations or people what to do. It's trying to pull together the partnerships with K-12 education, higher education, employers, other government agencies, community leaders, whatever, to try to figure out how to make these opportunities come to life for all Washington residents.

Leigh Parise: That's great. Thank you. I will say, when I first heard about the work that MDRC was doing with all of you, I thought it was really incredible to hear about all the different kinds of linkages that you all were making across the systems in the state. So even in your description, Mike, I'm not going to say you're ...[you] seemed like a jack of all trades and a master of all, obviously. So we are excited to get to have this conversation; there's so many connections between your work and MDRC’s. All right. So Isaac, I think this might be a question for you. It would be great to hear a little bit about your vision regarding capacity to do data analytics within a state agency. What does that look like for you?

Isaac Kwakye: Yeah, so we are at the very early stages of building our capacity to integrate data analytics into our work. And our engagement with CDI—Center of Data Insights—at MDRC is helping us to grow that capacity. For instance, we have one project now that we are working with them on—the OTTERS project—and [it] is optimizing tech technology through engagement research with students, which is working to improve our statewide chatbots that help students with the financial aid college application process.

But really it's about college assets and how we can get more students to start thinking about their life after high school and thinking about some training or education beyond their high school graduation. And so as part of this project, we'll work together to create data analytics tools that will allow us to generate rapid insights about our chatbots’ messaging and adapting and improving our strategy in how we can get more students to access postsecondary education in our state.

We think this is really an important work because policymakers and other education leaders want insights in real time. What is happening with students? How can we increase enrollment to postsecondary? What is happening with our faster completion? How can we increase our faster completion of students so they can access the state aid, which is one of the most generous need-based aids in the country? And this type of real time analysis complements the traditional research that we normally do as a state agency. So this partnership is going to allow us to build capacity and to grow and develop techniques that we can use as a state agency to do this work.

Leigh Parise: I especially appreciated in your response that you talked about what people on the ground are facing and what are the questions that they have. And how, as an agency, are you all trying to come up with some useful answers and responses to those kinds of questions that people are really facing on the ground to help more students get access.

I feel like it might be interesting just for you to say a little bit more about how typical you think it is or how much capacity you think there usually is at the state agency level for teams to be able to do this kind of work, and why you've really focused in Washington to make sure that your team has capacity to be able to do this well.

Isaac Kwakye: Yeah, so I'm new to state agency work. That's not where I started my career. And so coming in and realizing: How can we use [the] evidence base to drive the policy and the work in our state? It's something that's helped me to think about this work. But now I'm learning from other states— agencies in other states that are equivalent to our higher education agency in Washington—that this is not a normal way that work is approached in state agencies. And building this analytics research evidence base driving policy, and making sure the research is informing conversations by policymakers and decision-makers. I think there's some level of work that's been done over the years by different state agencies, but we have taken that to a new level of making sure that that's what is driving the work. We want to find out what works so that we can implement those policies.

And if we don't know something that has worked, how can we design either a pilot or a case study or use existing research and evidence to shape how we implement our programs and then learn from that and create a learning agenda around that? So, I mean, I wouldn't say I'm an expert in how states utilize their research to do this type of work, but I think the more I hear from other state agencies that are trying to learn from us—what we are doing and how we are doing it—it tells me that this is a new area of work that we are helping to drive with MDRC.

Leigh Parise: Wow, that's really exciting. And I think you're right that a lot of other places will be able to learn from you. I will say also—we didn't get to do this as part of your intro, but for people listening—Isaac was once also an MDRCer, so maybe there's a little bit of those evidence kernels from the early days.

Mike, do you want to add anything about how you think other states might be thinking about this or how you all are hoping that what you will learn will be useful for other state agencies as well?

Michael Meotti: It's not just states. It's also—I would throw in the mix—not colleges, universities, and other organizations. I used to be in the nonprofit world. And I think what happens is, we now live in a world where there's so much data all over the place that a lot of times what goes on in organizations—be it a campus or be it a state agency or be it a big nonprofit or United Way or philanthropy or community foundation—is we're looking at basically just a few data points. They might be really interesting data points, and some can be very powerful. I went through the period of time in this country when we started focusing on the high school graduation rate. And that could be a big mobilizing call in a community, to do something about it.

But what we need to focus on, and what Isaac and his team brings to us in Washington, is the ability to go beyond just the data points and to actually start to create knowledge out of what we know about what's going on in the ground in Washington: knowledge that's actionable, knowledge tasked to the decision-making process that could include decision-makers in our state legislature and the governor; community leaders, whether they're in business or labor; or other groups who are trying to drive change.

It could be local leaders who want to see a change in their community—it's specific to their region or their place in the state—it could be at a campus. And this information actually helps them work through those decisions, decide where they might drive a change initiative, where they might make investment. Because that is where we go one step beyond just looking at the numbers that usually just paint the problem. And what Isaac's team helps us do is to understand the range of possible solutions that make sense to invest in.

Leigh Parise: Great, thank you. So say a little bit about how you think this project or partnership is different than what's been done before.

Isaac Kwakye: Yeah, I think our work with MDRC is part of what WSAC refers to as a beyond government agenda, where we engage with partners beyond the state agency to drive change in Washington. This type of collaborative work can build our capacity as a state agency. As a state agency we engage in innovative research [and] sometimes we don't have the skillset or the expertise. So this type of collaboration brings that to us. But the most exciting part of this work is really the partnership, from start to finish. So WSAC staff is working together with MDRC staff from project conception to final reporting and developing the final product.

We are co-developing ideas also, and research projects that are innovative and relevant to Washington in helping students and families in our states. I think that's the difference. Most times you'll have an agency putting on an RFP or contracting with an expert organization to do work for them and produce the results. But even if they work with them it’s kind of like, part of the project they collaborate, and other parts, they are depending on the expert organization to do the work. But here we work through the whole project from start to finish in a very close partnership collaborative way.

Leigh Parise: Great. All right. So I know you mentioned these when you first started talking about some of the work that you've got going on, but do you want to tell us about some of the projects that you've got underway?

Isaac Kwakye: Yes. We have a lot of very interesting projects underway at WSAC right now. I already talked a bit about the OTTERS  project with MDRC, but I'll just mention a few others. One notable project is our inaugural Regional Challenge Grant; we are providing funding for regional partnerships in our state that are allowing them to do work locally that aligns with the agency's agenda to improve educational attainment for the state. And we will be awarding those grants—Michael was saying earlier today in another meeting—tomorrow there will be awarding of those grants. But this is another area [where] we've been talking to MDRC and we are learning how we can better evaluate this partnership and try to better understand how those partnerships affect the change that we are looking for.

And another exciting project is one that we are piloting soon, which is around enrollment. We are working with MDRC, with Central Washington University, in trying to better understand students who are admitted but they don't show up or either enroll and drop off very quickly—to understand what is going on. And this is work that we are collaborating with Central Washington University, with MDRC, to use the data to better understand the characteristics and profiles of who these students are, and then allowing the behavioral science (in terms of design thinking) to lead us to a student-centered solution that we can then share with Central Washington to use to help them in managing their enrollment.

We're also working with MDRC on another community-based organization project, which is a United Way of King County Bridge to Finish project, which helps students to succeed or complete community college by providing them additional supports, non-academic supports. And this is a project we are trying to evaluate the effectiveness of Bridge to Finish and trying to see whether the program is meeting the setup objectives by United Way of King County.

And so that's another way that we are collaborating and working from start to finish on a project. And I think what is also unique about this type of collaboration is we involve whatever organization or agency that is needing that solution to be part of that collaborative process. So there's a lot of learning that goes on for each of the organizations participating in this collaborative way.

Leigh Parise: That's great. When you think about WSAC's work, just from what you're sharing, it is clear that you are really thinking how do we plug into what's happening across the state? How do you get people who are really busy at college systems or at K-12 districts to buy in to what you're wanting to do with them and to say, “No, really, we actually are here to help you solve some of the challenges that you're facing.” I think, when I talk with people sometimes from other states, they say it can be really hard because people have a million different things that they need to be focused on. But it sounds like you're really partnering in real ways with people on the ground in this state.

And I think probably this piece about collaboration and making sure they're in on the front end is one of your secrets to success. But I'm curious about how, as an organization, you think about those partnerships within the state to help make sure that what you're doing is both really well received and actually leading to actionable knowledge—which is, Mike, one of the things that you mentioned.

Isaac Kwakye: Yeah, and I think I will go back to Mike on this because he's doing a lot of the early work, having conversations, setting the agency's goals and agenda, and [has] the ability to then pass on those opportunities for us to develop those partnerships and collaborative work. So Mike, do you want to...

Michael Meotti: It's interesting in higher education where you go on any campus, there are people who have expertise and knowledge in so many different areas that in higher education and higher education policy generally, sometimes we have a hard time learning from others. And so informing a lot of my work is things that I've picked up both career wise—I worked in the business world, I worked in a nonprofit, early childhood, K-12 space, whatever—and I've also been a voracious reader of different approaches to better management that are not about government management, not about public sector stuff. And I'll give you a trait saying from the sales management world, which is “Don't sell your solution, sell your ability to solve your client's problem.” So when we do approach people, we're not coming to them trying to say, “We have this great thing for you to do, won't you please, please, please do it or join us in doing it,” right?

That may be where it ends up. It may end up happening that way, but instead, I think we're trying to build our partnerships by finding the common ground. That's there, I mean, most college and campuses, they're looking for the same thing: more Washington residents continuing and more Washington students succeeding. And many places in the communities are looking for this in different ways, for opportunities for their young people, a more skilled workforce for employers, ending racial and ethnic disparities that exist in these pathways, that kind of stuff.

And so there's that natural gravitation around core goals. But then it's not about coming in as government saying, “This is what you need to do, or this is what you need to let us convince you you should do.” But instead, “Let's come together and figure out how do we solve the problem that is important to you,” if we're actually able at that point in time. And I think that leads people to go, like, “Oh, I want to have that.” First, you got to get the first conversation. Then you earn the second conversation by engaging people that way. And then what you find out is later on, coincidentally, you earn word of mouth, where these people say to others who you've never met before, “If you want to do that, you better go talk to those people at WSAC, because they can help you do that.”

Leigh Parise: That's super important. Thank you. That’s great.

Rick, let's hear a little bit from you. And in what Mike and Isaac shared, it's clear [that] as they think about interacting with folks in the state, it is important to both help them see why you are going to be a good partner and help them solve their problem. And I think it'd be great to hear a little bit from you as the person who leads MDRC’s Center for Data Insights. How do you think about building these partnerships, and why the focus on helping to build capacity? Because some people have asked us before, "Well, but if you build capacity, don't they not need you anymore?” Tell us a little bit about how you think about that.

Richard Hendra: I think it is an extension of our mission. Our mission is to help individuals and families toward upper mobility. And the opportunity and the scale of that problem are so large that it is critical to work together with other organizations if we're really going to make progress. We are a mission-driven nonprofit organization, so we center on trying to move that mission forward. And there's a lot we can do as an organization, but historically we would work with the state or a government agency or a school or a system, and we’d receive some data, write a report, and hopefully have some positive effects on public policy. Working together with lots of organizations and building the field is really more likely to move the needle, given the scale of the problems that we confront.

Leigh Parise: So Rick, tell us a little bit, then, about the goals of the Center for Data Insights and what you think makes this partnership with WSAC unique.

Richard Hendra: The Center for Data Insights formed about four years ago, and our main goal is to help organizations take advantage of data they already have. Lots of organizations are swimming in data, but the data are often used for compliance purposes, reporting purposes, and there's seldom use to drive program improvement. So as MDRC's internal data science center, we're all about trying to work with other organizations to unlock the insights from those data to drive program improvement.

We know that data science is only a piece of that. So collaboration is critical to what we do. Interdisciplinary approaches are critical. Forming inclusive teams with subject matter experts, with engineers, with behavioral scientists, with participants all at the table, is how we're going to really get the best insight to unlock the potential hidden in data. Data are often sitting on networks and just untapped, really. So that's what we're all about, is trying to leverage already existing data to drive program and policy improvement.

Leigh Parise: All right. Can you describe how you're working with data together, side by side, because of FedRAMP and Sprout and open source data?

Richard Hendra: Traditionally, government agencies or nonprofits will send us a data file and a couple months later—after some discussions along the way, but a couple months later—we'll send back a report. And that was the nature of the collaboration. So if we're really going to collaborate deeply—like Isaac was talking about—from the beginning of our project to the end, we have to be side by side somehow. And because of data security constraints, it's very difficult to do that. Now with our Sprout system, which has what's called a FedRAMP authority to operate, we are able to work in a very secure way, side by side, as if we're just a seamless team. So we are looking at the data together, we are creating variables together, we're running analyses together. Anyone in data analytics knows that learning happens really through osmosis.

It's sitting side by side, an analyst and her supervisor walking through, running a result, looking at it, making a decision, running another result. So how could you do that if you're just doing it in slow motion, from “Send me a file, then I'll run something and a week later send it back to you.” Now with our Sprout environment, we have this sort of sandbox where we could work together side by side and transfer knowledge much more quickly. And the learning is really bidirectional because WSAC is the expert on the data, the programs, and Washington state, and they have a lot of capacity in data analytics. So we're learning a lot about this too as we work together on problems. We're getting a lot of lenses that we've never would've gotten in the past, just by working together. A lot of the insights are often very informal on the spot, when you're looking at a file or result together. And that's what it's enabling: true collaboration.

Leigh Parise: That sounds like a really different way of doing research and working on data with a partner, so that's great. Just to make sure that I really understand, is it actually that people from MDRC and people from WSAC are coming together in a meeting and looking at the same data and both being able to get into that environment to access the same data, so that you can see what each other are doing, you can ask the questions in real time? I think often we'll get data from a partner, go through it, and then be like “Wait, so what does that variable mean? And where did that come from? And how do we interpret these things?” But this is actually something that you're doing in real time because you can both access it at the same time.

Isaac Kwakye: Yes, yes. And that's the beauty of this work is that the analysts at WSAC are running calls, and it's in the same place as the analysts at MDRC running calls, can review the calls. And we are having conversations about what we are seeing in the data, and talking through the knowledge that we have around the Washington data to the MDRC folks as we are reviewing it. So everything is being done in one place. All the codes are in one place, everybody sees it. And that's the beauty of being part of the FedRAMP collaborative space.

Leigh Parise: And are you seeing, Isaac, already people on your team able to think differently about analysis or learning things from the MDRC team?

Isaac Kwakye: I think it's going both ways, like Rick said, and so people on my team are learning different ways. The very simple and typical example is how MDRC codes around the program with a lot of information around the data that we normally didn't do in our coding. And now the analysts are learning and doing that. But then the MDRC folks are also understanding why we define certain variables in a certain way in Washington state, and they are picking up that knowledge too. So it's kind of like a knowledge that is flowing back and forth between the analysts in both teams.

Leigh Parise: Yeah, that's great. I could see how that, in the end, is really going to lead to a much stronger project, and get you all together moving toward more actionable insights and information. So that's great. Thanks. All right. So Rick, I would like to hear from you a little bit more. How do you think MDRC and the field can benefit from these kinds of partnerships?

Richard Hendra: We have such unique insight into the data. We can see like a state when we work with the state this way. We can see the problems through their lens because we're working with them so closely. We could understand how the data were generated, what the data really mean. What are the constraints that state employees are working under as we work with them? Because we're just working so collaboratively that we have unparalleled insight compared to what we used to have. That results in a much more insightful product. It also results in a much more sustainable product, because all the code that we're developing together, Isaac's analysts know how to use that code and take it forward.

We're going to leave that code behind, and that's going to be part of WSAC's code base for the future. So for the next project, they're not looking at a blank screen, they're looking at a fully well-documented set of syntax that they could use for this project or that project, to build on in the future. So it helps with sustainability, it gives us a product that's going to be more relevant to our audiences, that's going to have much more insight than we'd ever get from where we sit, because we're researchers. We try to tap as much as we can into the policy environment, but working as closely with the state, we just get a much better sense of what the opportunities and challenges are in the programs that we're studying.

Leigh Parise: Mike, Isaac, is there anything that you want to add about how you think the field can benefit from these kinds of partnerships or things that you would want other states to know about engaging in this kind of work?

Michael Meotti: I think that, having been involved in state government in different subject matters beyond higher education for many years, it's to understand that the ability to drive change comes from many levers that a state agency or a state can pull on. An authority—which is the normal way people view government, as “the people in government can tell me what to do” —that is only one of them. And that is not a lever that has much impact in the challenges in the areas we're talking about. And so we are consciously basing our work on the notion of a model of influence, rather than authority, that we want to be able to influence outcomes so that things are better for the people of Washington. And influence requires many things, and is based on many things, including partnerships with all sorts of organizations that are trying to achieve the same goal.

But one of the things that makes a partnerships work, makes your influence grow, is to be able to be someone who brings something of value to the work. And sometimes you're bringing financial resources and sometimes you're bringing thought leadership, but many times you can bring knowledge and insight. And if you can be seen as a regular resource for knowledgeable insight and actionable knowledge, to flip a term, that will win you the trust and the engagement of the partners you need to be able to work with. And our relationship with MDRC makes us that much better at being able to be seen as a center of knowledge and insight and how to get to our goals in the state of Washington. So it's a huge benefit for us.

Isaac Kwakye: And I think one way that other states can benefit from a partnership like this… You know, you look at WSAC and we have a research unit, but we are not a research organization. There's evolving methodology, there's innovation around research that as a state agency and a research unit within a state agency, you are not at the forefront of that knowledge that is transforming over time and growing over time.

So a partnership like this allows us to then expand our research methodology in terms of different things that we can do, from a pure research organization that is on the cutting edge of this work. I think that's one big benefit that other state agencies will have, because their research unit is not going to be right in the forefront of what is new, what methods are evolving, what innovation in research is emerging. And so this is going to help, really, any state agency that wants to use research data and evidence to drive their work.

Leigh Parise: All right. Well, Mike, Isaac, Rick, thank you so much for joining me today. It has been great to have this conversation. It's been great to get to do it in person. Really, thank you for taking the time.

Well, this is clearly a really unique research partnership that will result in answers to really important questions around higher education attainment and helping more folks get there. So it's really been great. Again, thanks so much to Rick, Isaac, and Mike for joining me. To learn more about this partnership, visit wsac.wa.gov and mdrc.org. Did you enjoy this episode? Subscribe to the Evidence First podcast for more.