Evidence First Podcast

Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people with low incomes. I’m your host, Leigh Parise.

In 2024, MDRC will celebrate its 50th anniversary. For those of you who don’t know, MDRC was founded in 1974 with just a handful of people and a tiny office in New York City. Since then, we’ve grown into an organization with more than 300 talented staff and four offices across the country.

To commemorate our 50th, we’re having conversations with some of our long-standing partners. These are people we’ve been lucky enough to work with and grow with over the years.

Per Scholas is perhaps one of the best-known nonprofit training and employment service providers that focuses on a specific economic sector—in this case, information technology. They’ve been around for 25 years. Their first office was right here in New York City, but now they’re in 20 different cities across the country.

We were fortunate enough to get to know Per Scholas and do a study with them way back in 2011, where we found that Per Scholas’ sector-focused program model substantially improved earnings and led to sustained career advancement for adults.

Sector-focused employment programs aim to improve the economic mobility of people with low incomes by training them for quality jobs in sectors that have strong local demand and opportunities for advancement. At the same time, these programs partner with employers to provide them with a skilled workforce.

The 2011 study was the beginning of a long-term research partnership with MDRC dedicated to helping Per Scholas improve its program and expand its reach. Today I’m joined by my colleague, Donna Wharton-Fields, a senior fellow at MDRC. And we’re very fortunate to be joined by two colleagues from Per Scholas: Shondra Tobler, senior director of national admissions, and Tamara Johnson, chief of staff.

We discuss our long-standing relationship, answering research questions, and expanding their program. Donna, Shondra, Tamara, welcome to Evidence First.

Donna Wharton-Fields: Thanks, Leigh.

Shondra Tobler: Thank you.

Tamara Johnson: Delighted to be here.

Leigh Parise: Shondra, I think maybe we’ll start with you. I know Per Scholas was part of two major randomized controlled trials, one of them with MDRC through the WorkAdvance Study. The initial findings were promising—as well as the seven-year follow-up—showing that Per Scholas significantly increased participants’ earnings and led to sustained advancement over time, which is very exciting.

But you didn’t stop there. Can you tell us about how Per Scholas approaches learning and improvement over time?

Shondra Tobler: Yes, absolutely. Thank you so much. Glad to be here.

Quite honestly, we were honored to partner with MDRC on this critical study, and humbled to be able to demonstrate results that actually show impact. Learning and development are integrated into our culture here at Per Scholas. Here, we have something called People Principles (which include nine principles), with the common thread between them focusing on just that: learning and improvement.

Principles such as challenge the status quo and apply creativity and innovation—as well as another principle being exceed expected competence—these principles provide a foundation (and an expectation, really), to ensure that we are always pushing the envelope, looking for opportunities to know better and to do better, and learning along the way.

Additionally, every staff member here at Per Scholas has money allocated to their professional development, and is encouraged to use it. I guess—specifically within the admissions team—learning and improvement show up in a few ways. One of those ways is that we’re always striving for better. We pride ourselves on continuously raising the bar.

In fact, when we partnered with MDRC a few years back, they recommended customer journey mapping and the experience was very insightful because it allowed us to develop greater empathy for our prospective learners and helped us to focus on them as we tried to figure out what was happening in our application funnel.

As a result of that, we actually developed a learner profile called Maya. (I think I’ll have an opportunity to talk a little more about Maya later.) Another example that I’d like to add, there: Learning and improvement is truly embedded in our DNA, as I said.

Within the admissions team, we are constantly having open and honest conversations about existing processes and how we can further improve them. For us, it doesn’t just stop there. We then come up with a strategic plan that will take the idea from words on paper to implementation and rollout.

Examples of this include more options for learners being able to attend our admission overviews, enhanced assessments, and even more productive interviews with our prospective learners, just to name a few.

Overall, learning and improvement over time have produced greater efficiencies within our admissions process. So we are tremendously grateful for the partnership and opportunity to work with MDRC.

Tamara, is there anything else that you’d like to add to that?

Tamara Johnson: Yeah, I’d love to jump in. I think one of the things I love about the example, Shondra, that you shared is that it really demonstrates how at Per Scholas, learning and improvement are really integrated into our dual focus on innovation and evaluation. Those things are really linked here.

Innovation at Per Scholas happens everywhere. And while we have a consistency in program application across the country, our newest and best ideas for how to expand our work really come from the field—from those working directly with our learners and with our employers.

We follow a method of innovation, and Shondra just gave a really great example of this—several examples—where we focus on iterative testing: start small, test, learn, and then iterate to increasingly greater levels of scale. While this method can take longer, it enables us to ensure we are moving forward with what works, stopping what doesn’t work, and creating an environment to fail, learn, and then move forward.

Most of the time these ideas and the work really start in our campuses or with our people doing the work every day.

Leigh Parise: Oh, that’s so great. Thank you so much. I really appreciate that thorough response. The examples that you gave make it seem like it’s really part of everything that Per Scholas is doing.

I’m curious; it would be great to hear a little bit about what it took to get there. It sounds like, Tamara, you said, “We’re really focused on What is the reason for doing this, and how would we measure it, and how do we iterate on it?”

Can you say a little bit about what it took to get there, as a nonprofit? I know that’s really hard. And there’s probably a lot of nonprofits listening, thinking, Well, what do you mean that’s where you are now? How do I even think about beginning the journey to get there?

Tamara Johnson: It’s such a great question. I have had the honor of working at Per Scholas for 2.5 years, and Per Scholas has been around for 28 [years]. Really, innovation at Per Scholas is ingrained in the culture in all of the ways that we just talked about.

We have grown and expanded over the last eight years—really we have doubled in size every three to four years for the last eight years. So, as we think about that growth, it has required the organization and the staff here to think differently about the work. How do we expand at this rate? How do we serve more?

I think a really intentional focus—not only by the leadership team, but again, throughout the country—[is] to inspire and create opportunities for local innovation, and then share that across the country to all of the different campuses. I’m honored to lead the innovation team here at Per Scholas.

We keep the innovation team, at a national level, intentionally very small, because we believe that the innovation happens locally and we want to create the space for that to occur. Our team is really focused on working with our teams across the country to really understand how we are doing the work. Where are some opportunities that we’re seeing impact that we can then bring and scale to other parts of the organization?

Leigh Parise: I know, now, that MDRC and Per Scholas have partnered on multiple research and technical assistance activities over the years. And I know from engaging with Donna and others on the team at MDRC that we have really valued this partnership over time.

But I also imagine that it could feel a little bit complicated to have an outside person coming in and looking carefully at what you’re doing. Can you tell us a little bit about what that experience has been like for Per Scholas staff, to be part of so many studies?

Tamara Johnson: It’s so great. One of the best parts of preparing for our conversation today was actually thinking back and reflecting on our partnership with MDRC. You know, our long-standing relationship dates back almost a decade, if not more than a decade. We originally worked together on our second randomly controlled trial, the WorkAdvance study, as you had mentioned before. In 2020, we worked on redesigning our admissions process.

Shondra also talked a little bit about Maya, which is our profile of our learners on which really all of our major decision-making throughout the organization is centered. It really helped to crystallize how we ensure the decisions that we are making across the organization are truly focused on our learners.

We also have worked together to understand the impact on our organization and our learners during COVID, when we shifted from fully in-person to completely remote learning, as many training organizations had to during that time.

In 2023, we worked on understanding our learner recruiting pipeline and assessment approach. And now, in 2024, we are working together on understanding the impact of our newest coaching model for our alumni.

What I love about working with MDRC is the broad span of work that we have done together, from rigorous evaluation to following a constituent-centered design process to deep statistical and data analysis (to drive insights and actions). Not only did we work on various methods, but in every conversation and every project, MDRC has kept us grounded in the research in the field.

You’ve really partnered with us to not just replicate results done in other studies, but to push the field forward, to ensure that we are contributing to the advancement and understanding of what works in the areas of workforce development and economic mobility in this country.

I‘ll say one other thing: We have a partnership—and I think a mutual commitment—to try new things, figure out what works, and then scale it to as many people as we can.

Leigh Parise: Thank you so much. That’s so wonderful, to hear all of that. I swear we did not ask Tamara to say any of those things. But if I got to work with Donna all the time, I probably would have lots of positive things to say too.

Donna, you’ve worked with Per Scholas over the years. What would you say sets them apart? Is there something that you think feels especially useful for other organizations to learn from them? What are your thoughts?

Donna Wharton-Fields: When I stop to think about it, what comes to mind most for me is Per Scholas’ commitment to continuous improvement in learning. I think Per Scholas is to be applauded for continuing to ask the questions, “How can we improve? How can we have greater impact in the communities that we serve?”

I think what this all really means is that Per Scholas is not resting on its laurels. They’re not satisfied with the achievements and accomplishment from years gone by; they’re continually trying to improve. This growth mindset that I think they have seems to permeate the entire culture of the organization. For that reason, I think Per Scholas is truly a learning organization. We’ve heard, from both Tamara and Shondra, examples of that.

I also think, through their work, Per Scholas is addressing equity gaps and providing people with skills and support needed to earn middle-class incomes, which is fundamentally what they’re all about: trying to improve mobility for people who are most marginalized.

And they’re really helping the tech sector to diversify its ranks, which we all know is critically important today—always has been. Per Scholas is always pushing forward to learn more about how to improve performance and it’s all very, very impressive.

Leigh Parise: Thank you, Donna.

All right. Most recently, I know, Per Scholas and MDRC have partnered on a project called Lever for Change, and its focus was helping the program scale up. So clearly, Tamara and Shondra, what you’ve talked a lot about: How do we scale?

The focus of this particular project was how to help the program scale up by increasing the number of learners that it trains as the organization expands to new locations and to new remote courses.

Can you tell us a little bit about some of the insights that MDRC was able to share with you, and then how you’re implementing them as part of that work?

Shondra Tobler: Yes, absolutely. I will say the insights that MDRC has provided have been tremendously beneficial, to say the least. For one, we learned that the enrollment rate of those who attend our admission overviews almost doubles the enrollment rate of all applicants. So basically it behooves us to try to get them in front of us, learning more information about our programs, in order for us to have a higher conversion rate in the end. Statistically speaking, that was 13 percent compared to 7 percent.

Additionally, they provided information on applicants: their preferred learning environments. At the time we predominantly only offered courses during traditional office hours, that being Monday through Friday, 9 a.m. to 4 p.m. And quite honestly, just that information—and being able to gather that information from candidates to see what their preferences were—has been game changing. We’re actually taking that information and beginning to make decisions about the types of courses that we’ll be offering in the near future.

I can go on and on. But one of the things that I’d like to highlight now is the journey. The journey with MDRC was a two-part and multiyear journey, as we talked about already.

The first part of this journey was about redesigning the admissions process. Through a deeply impactful series of sessions with MDRC, we realized that we had more than 30 touch points, in terms of our applicants, before they actually enrolled in one of our programs.

So we were advertising that our admissions process was, like, 5 steps, and that’s what we thought. That’s what it looked like. But as a result of going through the process with MDRC, we realized that those 5 were really over 30 touch points in order for an applicant to say, “Hey, what is this? Oh, I’m interested. Yes, let me learn more. And now I am officially enrolled.”

I mentioned earlier that MDRC recommended customer journey mapping. I think I’ve said this already—I just can’t say enough how game changing it actually was. I’ll say it that way. And if you don’t mind, I’d like to elaborate a little more about Maya. I mentioned her earlier. Is that okay?

Leigh Parise: Yeah, that sounds great.

Shondra Tobler: Awesome. Okay, so you’ve heard Tamara mention her; you’ve heard me mention her. She’s actually our learner profile, and, basically, she’s a persona. We created her out of this partnership and out of this experience—this customer journey mapping experience.

She’s a black woman—a single mother of two young boys—and she supports her family by driving for a ride-sharing company. I’m sure you’re like Okay, why are you giving us this information about this persona?

The reason is because it was able to help us really increase our empathy toward our prospective learners and have the opportunity to step into their shoes and understand where there were some gaps or areas of opportunity within our own admissions process that may have been excluding prospective applicants.

What I mean by that is, we intentionally selected a black woman’s journey to map out because the data at the time showed us that black women fell out of the front end of our pipeline more than any other group of applicants. And we really wanted to understand why.

Closely following Maya’s journey, we implemented many changes to our admissions process. Some of those changes—and I’m sure I’m going to forget some of them—but some of those changes actually included implementing an interest form in lieu of a formal application.

At that time, prior to working with MDRC in this capacity, our application was an extensive application. It would take, on average, 30 to 45 minutes to complete. It wasn’t mobile friendly; it asked for everything.

Another thing that we learned and changed was introducing automation into our admissions process. That helped streamline the process, and it also reduced the labor for staff as well as our applicants.

We created an onboarding experience with learners that mirrored the onboarding process of a new job, so that learners were familiar with that process once they graduated and actually landed their first entry-level job in the tech industry. After all, the ultimate goal, postgraduation, is job placement.

That prompted us to upgrade our acceptance emails—refreshing them with graphics that gave you pops of confetti. Basically, [it was] like a virtual explosion in their inbox, congratulating them and welcoming them to our family—to the Per Scholas family. It also provided all the details pertaining to their cohort, a checklist for them, outlined documents that they needed to complete and submit, and all of that good stuff.

It also linked them to our learner support team, which is an internal group in our organization. Because of that, they would have access to their learner support manager, financial coach, and other key members who would be supporting them along their journey with us.

Let’s see, what else was in there? A holiday calendar—just all sorts of resources and materials. But I’ll say this: None of this work would’ve been possible without MDRC’s assistance and our continued partnership.

Tamara Johnson: I think that was great. As we’ve already talked about, we have worked with MDRC on so many different aspects of this process. What Shondra has walked through is really the first phase of that work, and so I’m going to share a little bit more.

Donna, I’m so excited because I feel like we have an opportunity to really highlight the work that we have been so closely partnered on over the last couple years, and your fingerprints are all over it. But really, the second part of this work with MDRC was driven by the realization in 2021 that—through this first phase that Shondra just walked through, as a result of that work—we had a significant increase in the number of our applicants.

Which was very exciting, but at the same time, we were still having trouble filling our classes. We fixed one problem, and we unearthed another challenge. In typical style, MDRC really dug in with us, and really tried to understand that, while we were having trouble filling our classes, our model had grown increasingly complex.

We had expanded into new markets; we had added new modalities. We’ve now had in-person, remote, hybrid [classes]. We have 13 different training tracks. We were infinitely more complex, and we were having trouble filling all of these different types of classes throughout the [organization]. We really needed to better understand why this was happening.

So again, following that constituent-led process, we worked and partnered with MDRC to better understand what was going on here. What we found was that there were a fair number of individuals who were filling out the interest form [but] weren’t actually sure whether or not they were going to move forward in the process, and very often did not. And that’s fine; that’s part of that process.

But what was happening is that Per Scholas was then spending the same amount of time on those individuals [as] those who were highly certain that they would actually continue. So through a process, MDRC helped us figure out whom the folks are that we really need to spend our time working through, because they have a higher likelihood of going through the entire process.

We also discovered that the demand for remote training is still very high. I don’t know if you can call it post-COVID, but at least in this environment that we currently find ourselves now, that demand for remote training is high. But there are individuals who are intentionally looking for that in-person training, and [they] value the hands-on learning that Per Scholas is known for.

Finally, our applicants are dealing with a complex set of personal circumstances. We set out to try to better understand which of those circumstances were getting in the way of them actually enrolling in our programs.

Through that, we were able to figure out [that] they’re trying to balance their responsibilities. They’re trying to meet their financial obligations. We were better able to understand what those challenges were and have since implemented a number of programs and initiatives to help directly address those challenges.

We did another set of work with MDRC around our assessments, and this is an area that Per Scholas has additional work to do. MDRC really helped us to better understand the variety of assessments that we currently have, and the initial outcomes associated with those assessments.

So based on these and other findings, we are implementing a variety of things: [a] more consistent application and assessment approach across the organization. At the interest form [stage], we’re really seeking to better understand those who are more likely to move forward and focus our limited resources there.

We’re updating our website experience. We’re testing a zero-interest loan in partnership with Google and Social Finance—and MDRC is also involved in that—to enable our learners to manage their personal financial obligations.

Again, as Shondra said, we could go on and on [about] the different impacts that we have had that have ultimately led to very specific actions that we have taken to improve the experience for our learners and for our staff.

Leigh Parise: Wow, that’s super impressive. All right, I have to ask: How do you prioritize what you’re going to focus on? It sounds like you’ve identified so many different areas where you think, Oh, we want to address that! We want to address that! We want to address that!

How, as an organization, do you figure out Okay, no, this is really the place where it makes sense for us to spend our time and energy, and figure out how to make some changes to be able to serve people better? I’d be really curious to hear you just say a bit about that.

Tamara Johnson: I think a lot of this, Leigh, has to do with really understanding the problems that we’re trying to solve. What are the specific actions that we think are going to have the highest impact in solving those problems? And prioritizing those actions. And allocating resources against them.

As we look at all of the different things that we’re implementing, there’s a focus on [what is] highest value, highest impact to the problem that we’re trying to solve, and then focusing our resources there. The other things we continue to work in over time.

Shondra Tobler: Thanks, Tamara. Just to add to that, I would definitely agree. But I think additional layers are excellence and efficiency. The reason is because, you mentioned impact, but quite honestly, I feel as though it doesn’t just stop there. One of the things that we’re constantly thinking about is the Maya.

Not necessarily the learner profile, who—as I described earlier—because of the data happens to be a black woman. But Maya could be any race, could be any gender. Maya represents our prospective applicants.

I say that because we’re constantly looking at things from her, his, from their perspective. How would they feel? How would Maya feel? Donna, I’m sure you remember this. How would Maya feel?

Every decision, every opportunity for us to push the boundary—stretch, enhance, et cetera, everything that Tamara talked about—it’s not just internal collaboration and “How do we prioritize and identify because it’s so many things?” But it’s “What would improve the process, yes, for staff, for employees, for us—but what would improve the experience for our learners?”

Leigh Parise: That’s great. Well said.

Donna, I want to ask you one other question. This work is really different than where we started in our work with Per Scholas. We talked about this randomized controlled trial that we did, I think, starting in 2011.

So some people might hear what Tamara and Shondra have described and think, Wait, that’s not the stuff that MDRC does. It’d be good for you to talk a little bit about how this work is really different than where we’ve started with them, and what it’s looked like to work with our team—like how quickly things are being turned around. Can you share a bit about that?

Donna Wharton-Fields: Sure, sure. When MDRC conducted an evaluation of Per Scholas’ trainings, their question was, “Do the trainings work?” Are they effective at supporting learners in securing employment in the sector, and at getting people to earn good wages? (Wages better than what they would’ve earned if they hadn’t gone through the training.)

We use an RCT, or a randomized controlled trial, to answer that question because that’s the best method. It’s the gold standard for establishing program effectiveness. But RCTs can take years to design, launch, and implement before there are any results to share.

It’s not surprising that, with such positive results from the RCT, Per Scholas wanted to expand their impact—that is, enroll more people into their training. But during the evaluation, our analyses revealed that despite the good effort, the good effects, many people were falling out of the application process at various steps along the way.

So to expand the impact, if you will, Per Scholas realized that they would need to focus on a narrower question—the leakage from the application process. For this, an RCT wasn’t required. This type of question can be addressed through a shorter engagement—through technical assistance, or TA, which we say often. That generally focuses on providing more targeted support [for] a specific issue.

More recently, we’ve been providing TA support to help Per Scholas gain insight into how they might improve the application and enrollment process. To do this, we use a human-centered design technique called customer journey mapping. With Per Scholas, we conducted an analysis of the enrollment process from the perspective of Per Scholas’ learners, and we identified a number of things that Per Scholas could do to improve the process. This ultimately has improved the application experience for learners.

The solutions that Per Scholas came up with made the process faster and more efficient for applicants. This process also resulted in Per Scholas adopting more efficient procedures behind the scenes; new staff training priorities were identified, and there was an overall more human-centered perspective about welcoming new learners.

Again, this type of TA engagement is much shorter, more targeted, and can provide insights into why certain operational issues are happening—and through a human-centered approach, suggest solutions that can be designed or redesigned to reflect the needs of applicants.

When we did the customer journey mapping—this human-centered design approach—we had results, really valuable information for Per Scholas, within months. One quarter, we focused on mapping the learner journey, and the next quarter, we were brainstorming solutions that they would then put into effect rather immediately. Some of the recommendations that came from the process required additional planning on the part of Per Scholas’s team, but they prioritized things that they wanted to change, and they went about working to put them into practice.

So compared to an RCT, technical assistance is much quicker. Its focused; it’s very, very targeted. And again, the human-centered approach in this case helped Per Scholas to really focus on the experience, the quality of the experience for learners, and also for staff as well.

Leigh Parise: Great. All right, so I think we are nearing the end and being able to wrap up. Where we started was kind of like “Talk about how Per Scholas has approached learning and improvement over time.” I don’t know if there [are] any quick summary points of some of the places where we started that you feel might be helpful to share.

Shondra Tobler: One thing that comes to mind is how different our admissions process is, in such a short time. The reimagination of the admissions process, as we call it. When we partnered with MDRC the last time, we set out to enhance and improve the admissions process. I talked a great deal about that customer journey mapping experience, and Maya, and all of that.

Believe it or not, though we had tremendous improvements and enhancements to our process, we only touched, like, the very beginning of the funnel, and the very end. So it’s amazing when I reflect back on some of the initial conversations and the amount of work that’s been able to do—yet the amount of work that is still undone. And what we’re discussing and working on and all of that.

I would say that the partnership and the working sessions and the interaction has been tremendously beneficial. I’ve enjoyed every moment of it, and I’m really looking forward to continued opportunities to collaborate, just to see how much further we’re able to grow and how much more we’re able to improve the experience for prospective learners.

Leigh Parise: I really appreciate your continued focus on “This is ultimately what we want to achieve, and where we’re trying to get, and how we’re trying to support the learners that are part of Per Scholas.” So thank you for bringing us there.

That feels to me like a great place to wrap. I really have appreciated this super engaging discussion. Shondra, Tamara, Donna, thanks so much to all of you for being here.

To learn more, visit MDRC.org. Did you enjoy this episode? Subscribe to the Evidence First podcast for more.