Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people with low incomes. I’m your host, Leigh Parise. In a competitive labor market, students graduating from high school need skills that can translate to meaningful careers. PREP-KC is a leading K-12 education intermediary in the Kansas City region, bringing educators and industry professionals together to give students real-world learning opportunities and prepare them for in-demand careers.

PREP-KC partners with six urban school districts and three charter schools and aims to equip students with market value assets, industry-recognized credentials, and college credits before they graduate high school. Responding to increasing demand for data literacy across all industries, PREP-KC launched its Data in Action! course in 2023 to give eighth-grade students early exposure to data analytics. It aims to prepare students for high-demand careers and develop their interest in math and science fields. MDRC will conduct an evaluation to test whether this course can help boost academic achievement and whether more students will opt for a high school pathway in science, technology, engineering, or math as a result. To learn more about this evaluation, I’m joined by Doug Elmer, vice president of PREP-KC, and Zeest Haider, the research associate at MDRC leading the evaluation. Doug and Zeest, thanks so much for joining.

Doug Elmer: Thanks for having me. I’m really happy to be here.

Zeest Haider: Yeah, same. Thanks, Leigh. Excited to have this conversation.

Leigh Parise: I will say—selfishly, Doug—we’ve now known each other for many years and I was excited when I heard that you were going to be joining Evidence First. Both because PREP-KC does amazing work—and we’ve, as an organization, talked about partnering for so many years, and we’re getting to partner on this study—but it does also give me a chance to talk about my love for the Buffalo Bills, as you will not be surprised to hear me say. Although I have to admit that I did owe Doug an order of buffalo wings when my Bills lost to his Chiefs. (I guess this could apply to many years, but this was a couple of years ago.)

It’s really great to have you both here. I’m super excited about our conversation.

Doug Elmer: I was wondering if the playoff bets would come up. I’m glad we got it out of the way early.

Leigh Parise: [laughs] One day. One day. We’re going to have you back, one day, after the Bills finally make their way through the Chiefs.

Alright, Doug, let’s start with you, but not with football. Tell us a little bit more about PREP-KC and the framework behind the prep that students receive.

Doug Elmer: PREP-KC really focuses on providing students who are attending the Kansas City region’s urban schools with the opportunities and the experiences and the information they need to succeed after high school, regardless of what they want to do. That takes a couple of different forms. We spend a lot of time connecting students with industry professionals so they can learn firsthand about all the different amazing careers and industries that drive the Kansas City economy. They can hear from industry professionals who love their jobs and are passionate about what they do so students can start to see themselves in those roles. That early interaction between students and industry professionals helps students create a mental model of the jobs and the careers that are out there. With our high school students, it’s then helping them think about How do we help get you a head start on what you want to do after high school?

That really focuses on going beyond the basic requirements for a diploma and giving students the opportunity to earn what we call (and what the Kansas City region has adopted) market value assets. These are rigorous experiences and opportunities that go beyond just a standard high school diploma and give you a competitive advantage as a young person. Whether you’re getting ready to apply for college or whether you’re getting ready to enter the job market, you will really stand out among other candidates. These are things like earning an industry-recognized credential while you’re still in high school, taking college coursework, completing an internship or a similar work-based experience, or starting an entrepreneurial experience. That’s what our work looks like in high school. And again, [it] really runs through Kansas City’s workforce. All of those opportunities give students a chance to build social capital and expand their networks as they interface with advisors and mentors and industry professionals who are guiding that real-world learning and helping them earn those market value assets during high school.

Leigh Parise: That’s great, thank you. Can you say a little bit more about how the whole region has adopted this approach to market value assets? I think that’s one of the really unique and exciting things about the work that you all are doing in Kansas City. Can you say a little bit about how that came about and what it means for the whole region to have adopted the approach to market value assets?

Doug Elmer: Absolutely. About seven or eight years ago, the Kauffman Foundation started looking at the current state of K-12 education and how it was preparing students for success after graduation. At the same time, PREP-KC was wrapping up a much smaller third-party evaluation looking at cohorts of students who had participated in early iterations of our career pathways at the time. We noticed a group of students who really stood out in terms of college enrollment, college persistence, and degree attainment. We dug in with an evaluator to look at what the differentiator was for these students.

What we found—it was this group of students who all had the opportunity to do something beyond a standard high school education: college coursework, career certification, lots of interaction with industry while they’re still in high school. Our founder and president and CEO, Susan Wally, coined this term “market value assets” to describe those opportunities. She had an opportunity to connect with the team at the Kauffman Foundation that was thinking about how school could look different across the Kansas City region. They were really excited and energized about this concept of market value assets, and it became the backbone of the real-world learning initiative here in Kansas City.

Leigh Parise: That’s great. Thank you so much. I have to say—if anybody who’s been a long-time Evidence First listener is hearing this and having deja vu—two years ago, we did an interview with Dan Tesfay from the Kauffman Foundation. It’s really cool to get to connect several years later, when you’ve advanced what you’re doing in Kansas City and we get to hear directly from you. PREP-KC has really been such a leader in this field. So thank you.

Some of the things that you said were about how, as an organization, you think about how to help prepare students for what’s ahead. I want to go to what we’re talking about more specifically today: Tell us about this Data in Action! course. What’s the impetus behind launching this with Kansas City eighth-graders?

Doug Elmer: The work we do with employers—and also operating in this real-world learning ecosystem in Kansas City—is a big piece of what led us toward data science and data literacy. As you can imagine, we spend a lot of time sitting down with employers and listening to what they’re looking for in young professionals. What are those skills that are really differentiators at your place of employment? We started hearing about the need for data literacy and the need for data science skills in places we expected to hear them—so IT and tech, obviously. But no matter where we turned—whether it was healthcare, finance, marketing, advanced manufacturing, or skilled trades—we were hearing more and more that data literacy and the ability to be able to understand and work with data was one of these differentiators. At the same time, we went back and looked at the K-12 curriculum, thinking, Wow, data science is really important for our kids. Where do they have access to it?

We found it was, at best, an indirect focus of math and science curricula—in our two states, at least. That seemed like an opportunity for us to think about how you introduce students to data science and data literacy through the lens of how folks are using those skills in the real world. No matter what you want to do after high school, having a solid understanding of how to work with data will help you get there faster and be more successful when you arrive. That was the kernel that started us on this path.

Eighth grade—middle school, but especially eighth grade—is this critical transition point where not only are you getting ready to jump into more rigorous academics, but it’s also, in a lot of our schools, where kids are being asked, “Who are you and where do you want to go?” We realized this is such a great place to introduce this idea of data science in the real world—and not only get kids thinking about data but about all these other amazing pathways into Kansas City’s economy where those skills are adding value and helping people solve these really compelling, really interesting challenges. That was, for us, where this course (Data in Action!) was born.

Leigh Parise: I love that. The way that you just talked through that, it’s so clear that you’re thinking about what the needs are locally. You’re learning about what was happening nationally and what some of the broader recommendations are and thinking about the timing of eighth-graders’ personal and academic pathways. This is really exciting and sounds like a very cool innovation to be able to introduce to your schools.

Zeest, let’s talk about this study. What do we hope to learn from the evaluation, and what do you think others in the field can learn from this?

Zeest Haider: What’s really exciting about this study is that it’s a randomized controlled trial, the gold standard of program evaluation. Through this study, we’ll be able to learn about how taking the Data in Action! class has impacted student outcomes. This study randomized these eighth-graders to take the Data in Action! course and be part of the treatment group or to take any other elective offered at their school—and that [latter] group has formed the study’s control sample. We’re hoping to learn a few important things from this evaluation—which I’ll highlight, and then I’ll talk about why this is important to the field.

The main study’s impact question is whether taking the Data in Action! course has a positive impact on students’ eighth-grade achievement in math and science. The second two questions shift the focus to the ninth grade—which is interesting because this course and this evaluation present a unique opportunity to follow these students into their ninth grade and see if taking the DIA [Data in Action!] course in eighth grade also has a positive effect on their Algebra I and their Biology I course outcomes at the end of ninth grade.

Secondly, are students who have taken the Data and Action! course more likely to select STEM pathways in their ninth grade? These could be courses related to computer science, related to finance, related to health. This is a really unique opportunity because the course links STEM and career and technical education for students at the eighth-grade level and starts planting that seed so that they are well aware of these pathways when they get to choose that early in ninth grade.

Alongside the impact study, we’re also conducting an implementation study of the Data in Action! course. This is to understand what it takes to integrate this course into the fabric of the school; what the stakeholders are saying about what their views are of the course; and how teachers, school administrators, and students are all responding to what they’re being exposed to through the Data in Action! course. We’ve completed all of our qualitative data collection; we’ve done our site visits. We’re now starting our data analysis.

There are a couple takeaways that I’d like to highlight here that have stood out so far and that we’re looking forward to digging into. I’ll start by saying that the other unique feature of this course is that PREP-KC has not designed this course for only STEM teachers to teach. You can come from a different content background, and the coaching and the professional development that PREP-KC provides will guide you through the course for the entire school year. The weekly coaching provided by PREP-KC’s teaching and learning coaches has really been a valuable asset for all the teachers regardless of their tenure, regardless of what their teaching background is. For example, we spoke to a teacher who had a math education background, and they found the coaching to be really helpful in enhancing their teaching practices and making changes in real time to their classroom instruction and classroom management. Similarly, we spoke to a teacher who did not have a STEM background, and she found it really helped her teach the data concepts and work with the data sets themselves.

The other important players in all of this are the students, and we’ve been lucky enough to speak to a number of students in all the schools that we’ve had a chance to visit. Repeatedly, we’ve heard from students that “what I’m learning in this class is also helping me in my math class, and I’ve been able to transfer those skills from my DIA class to my math class as well,” which is great to hear. The other thing that has stood out to us is students talking about the experiences they’ve had going out to the industry partners—whether it’s their offices, whether it’s college campuses—and then really engaging with them, seeing their data being used in the real world. I think seeing those links being made for students at the eighth-grade level has been a really unique takeaway from our learning so far.

As far as what this will add to the field—as Doug has mentioned to you, the Kansas City region and the market value asset work is really unique. I think this course—[offered] at the eighth-grade level and exposing students early on to this work and to making those connections and to setting up expectations of what is to come in high school—is a way of preparing the students for success. This evaluation places us in a unique place to be able to see whether that’s actually effective for students.

Leigh Parise: That’s really helpful to have you walk through all of that. I appreciate it. That sounds really exciting. I’ll say a couple things I was thinking as you were talking about the detailed study: Why do an RCT [randomized controlled trial]? Why not just give it to everybody? I imagine this is not a minor investment for PREP-KC as a program, or for the schools. So understanding what it means for the school feels really important.

And then—putting on my implementation researcher hat and, I guess, also my teacher hat—being able to figure out how to do this well, making sure it’s feasible to pull off and that both teachers and students think it’s valuable: That’s huge. That’s certainly something that matters a ton if schools are going to continue to do this or if other schools are going to pick it up and decide to implement this course. So thanks for walking through that.

It’s also really great to hear you say that students are seeing how what they’re doing in the classes is playing out in the real world. Doug, that goes back to one of the places where you started, which was You can’t be it if you can’t see it. It sounds like that’s actually something, Zeest, that you’ve been hearing from students themselves. Doug, is there anything else you want to add before we move on to our next question?

Doug Elmer: One of the things that I was reminded of, as Zeest was talking about the randomized controlled trial—it was one of the first places where MDRC and PREP-KC were able to work together with the schools from a couple of perspectives. One, we were able to convey that we wanted it to be an equitable shot at the class. Zeest and her team were really helpful [at conveying] that to schools: With a limited number of seats, one of the benefits of a randomized controlled trial is that every kid has an equal shot at participating in the course.

The other thing that it helped reinforce—and that the MDRC team was helpful in helping us reinforce with principals and district admins—was that Data in Action! is what my colleague Cecil Christwell (who leads our data science team) loves to refer to as “low floor, high ceiling”: Almost any student benefits from participating in the class. Some will just get standard data [skills] that will serve them well. And then, [for] other kids, this is the spark that sets them on a path to go deep into data science. I think even though the RCT is ending after this year, that principle really stuck with schools. Now, as they’re thinking about expanding the course, they’re not trying to exclude certain groups of students in their middle school. I think the messaging and the framing that the MDRC team helped us do during the initial stages of student recruitment and student assignment really helped lay that foundation. I’m reminded of how much we appreciate that.

Leigh Parise: I love that you highlight that, especially because there are so many opportunities that could be really cool opportunities but sometimes people think, Oh, right, only the top 10 percent of students get to have access to that. So something that is leveling the playing field and opening the opportunity to everyone is great. I’m happy to hear that participating in this study helped to get there. Awesome.

Alright, Doug and Zeest, you’ve talked about a lot of different people being involved, so it seems like it’s a really collaborative project with different people in industry and in education involved. It’d be great to hear you talk a little more about how we’re all working together. Doug, do you want to first start by talking about the partners involved or how you’re working with MDRC?

Doug Elmer: Sure. We’ve got several full-time science coaches. Their role is to get data out to the buildings and work in the classrooms with teachers throughout the year. That, I think, has been instrumental in helping teachers feel supported and also giving us real-time feedback on curriculum adaptations. Every semester this course is offered, it’s a slightly different iteration because of the teacher and student feedback. I think those coaches are key conduits of that feedback.

We also have industry professionals. We have, for the students, integrated campus visits where they’re going out and visiting several of our colleges or workplaces that emphasize data science in their work. As part of that experience, they’re meeting anywhere from six to a dozen or more industry professionals from different companies who are all using data science in their work. At one of those campus visits, students might meet a data scientist working for a bank, sit next to somebody who’s doing population health at a hospital, or sit next to somebody who works in tech.

And so, again, [we are] really emphasizing for our kids that this isn’t an industry; it’s infused in all industries. Our administrators and our district leaders have provided input and also been great listeners; from their teachers and students, they’re learning more about the role that data science plays and what we’re starting to call a real-world learning toolkit.

Last but not least, the MDRC team: They have been a focal point to troubleshoot; to reflect; and, ultimately, to take action to improve the student and teacher experience. I don’t want to speak for Zeest and I’ll turn it over to her, but I know [that] for the PREP-KC team, those meetings are not just about staying on track with the project milestones. They have become one of our most valued sources of dialogue. The MDRC team probably understands the project better than any other single group of stakeholders, because they’re the only other folks that see the entire spectrum of what is going into the project.

Zeest Haider: Thanks, Doug. I will just reiterate how great PREP-KC has been to work with and what amazing partners they are. We always look forward to our conversations, which happen on a biweekly basis. I think I just, I want to add about this collaboration-- When this work started and when the evaluation started—it was essentially at the same time. We got the grant, and a few months later, the program is being implemented; the course is being implemented.

Everything was happening pretty quickly, and a lot of learning was happening in real time during the first year of the course’s implementation. For example, what’s a better way to get principals and administrators to buy into this course, to see the value of it? Principals obviously have a lot going on, and this is one other thing. So we work together to come up with ideas on how to communicate with the schools [so that communication is] more streamlined, how to identify a point of contact that maybe isn’t the principal and is someone else who can help support this [course] at the school level, how to leverage the Data in Action! teachers for them to be the voice of support at the school.

I think all of that worked well, and in the second year, we saw the fruits of that labor in having a smoother transition to the start of the course and for those teachers and students and principals to see the value that this course brings to the school. That’s just one example of how we’ve worked together. I think it’s been really great. I can’t believe how quickly it’s gone, and I’m really looking forward to continuing to collaborate as we move toward the data analysis part of the project.

Leigh Parise: Wonderful. Thanks to you both. It’s great to hear about how you were able to get so many partners and moving pieces functioning together and moving in one direction. I also will say, Doug, it’s really great to hear you talk about how getting to partner on this study and getting some formative feedback along the way was genuinely useful to your team. I think sometimes when people hear, “We’re going to look at the outcomes of kids at the end of eighth grade and then in ninth grade,” they think, Oh, I’m not going to learn anything until a year or two years into the study. But clearly, that is not the case. That is always good to hear. Thanks so much for joining, Doug and Zeest.

Zeest Haider: Yes, thanks, Leigh. And thanks, Doug.

Doug Elmer: Thank you so much for having me. This was a great conversation.

Leigh Parise: To learn more, visit MDRC.org. Did you enjoy this episode? Subscribe to the Evidence First podcast for more.