The Detroit Promise Path Evaluation: Outcomes After Four Years


The Detroit Promise offers recent high school graduates from the city of Detroit a scholarship covering community college tuition and fees. To help more students stay enrolled in college and overcome barriers to college success beyond the cost of tuition, the Detroit Regional Chamber partnered with MDRC to design the Detroit Promise Path, which adds proven student support services to the existing scholarship. The services include a comprehensive coaching component with a dedicated campus coach and monthly financial incentives to offset other costs of attendance (for example, transportation costs). The program, operated centrally by the Chamber, relies on data tracking through a management information system to ensure it is operating effe­ctively, to monitor students’ progress, and to communicate with students. (A management information system is a computer-based system used to capture information about program participants and the activities they engage in with the program’s staff.)

MDRC evaluated the Detroit Promise Path using a randomized controlled trial research design. The study enrolled a total of 1,268 students across five Detroit-area community colleges in the 2016 and 2017 academic years. All students continued to receive the Detroit Promise scholarship; program group students were offered the new Detroit Promise Path coaching program, while control group students received the scholarship alone. Complete three-year research findings, as well as thorough descriptions of the intervention, student population, study design, and policy implications, can be found in MDRC’s report of record, Motor City Momentum. In brief, that report found that the program was well implemented at four out of five colleges; that program participation was high, with more than 90 percent of students responding to cold outreach; that large numbers of students dropped out of college, including 40 percent who dropped out within the first year; and that at the three-year mark the program had statistically significant, positive impacts on students staying enrolled in college and earning more credits, but no statistically significant impact on earning a credential.

This brief reports on students’ outcomes at the four-year mark. Extending the observation to a fourth year is valuable for two reasons. First, the Detroit Promise Path demonstrated improvements on intermediate academic measures at the three-year mark, and many students were still enrolled in school during the third year, including many part-time students, who take longer than full-time students to graduate. Second, the COVID-19 pandemic hit Detroit hard and greatly disrupted students’ educational trajectories. Thanks to support from The Kresge Foundation and the Ford Foundation, MDRC was able to collect another year of data to track student outcomes. For the four-year outcomes, MDRC collected data from the National Student Clearinghouse (NSC), which contains information only on enrollment and credential attainment. Data on credit accumulation are not available from NSC.

Table 1 presents students’ academic outcomes at the four-year mark for all colleges. It shows that the Detroit Promise Path continues to have a statistically significant, positive impact on some academic outcomes, such as number of semesters enrolled. However, for credential attainment—the primary outcome of interest at the four-year mark—the estimated impact is not statistically significant. These findings mirror the ones at the three-year mark, when there were positive impacts on several academic outcomes but no statistically significant impact on graduation rates.

Table 1. Academic Outcomes at Four Years

Outcome Program Control Difference   P-Value

Average number of semesters enrolled

  3.2 2.9 0.4 *** 0.008

Average number of semesters enrolled, categorical (%)

0 17.9 19.2 -1.2   0.593
1-2 31.5 39.1 -7.6 *** 0.007
3-4 18.3 15.6 2.8   0.207
5-6 15.1 12.5 2.6   0.204
7-8 17.1 13.6 3.5 * 0.094

Earned a credential from any college (%)

Semester 1 0.1 0.0 0.1   0.382
Semester 2 0.3 0.7 -0.4   0.426
Semester 3 0.8 0.7 0.2   0.745
Semester 4 2.6 2.7 -0.1   0.941
Semester 5 4.0 3.8 0.2   0.847
Semester 6 7.3 7.0 0.3   0.831
Semester 7 9.2 8.0 1.1   0.483
Semester 8 11.1 9.7 1.4   0.415

Highest credential earned (%)

Certificate 1.7 2.3 -0.6   0.466
Associate’s degree 8.2 6.8 1.4   0.356
Bachelor's degree or higher 1.0 0.4 0.7   0.190
Degree of unknown type 0.2 0.2 0.0   0.987
Sample size (total = 1,268) 829 439      

SOURCES: MDRC calculations using data from both the National Student Clearinghouse (NSC) and the Detroit Promise Path colleges for the first three years of the study and solely from the NSC for the fourth year.
NOTES: Estimates are adjusted by site, interaction between race and gender, and ACT and SAT score. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Weights are calculated to make the effective (weighted) random assignment ratio the same in all random assignment blocks. The effective random assignment ratio is equal to the full sample’s random assignment ratio

Beyond the main academic outcomes, the study also found the following:

  • While a relatively low proportion of the study sample had earned a credential at the four-year mark—11 percent of program group students and 10 percent of control group students—a fair number of students were still engaged in postsecondary education: 21 percent were enrolled in college in semester 8.  
  • A combined measure of graduation and transfer shows that at four years, more than 20 percent of program group students and 17 percent of control group students had either graduated or transferred to a four-year college.
  • An exploratory analysis looked only at the four colleges where the program was implemented as designed. At those colleges, the program had a statistically significant, positive impact on credential attainment in the fourth year. (See the three-year findings report, Motor City Momentum, for more information about what happened at the other college.) In semester 8, 10 percent of program group students earned credentials, compared with 7 percent of control group students. This 3 percentage point impact is statistically significant. While 3 percentage points is not large, it is in line with the graduation impacts found in many other college interventions. That said, this impact should be interpreted cautiously, given the smaller sample size involved and given that the subgroup of these four colleges was not prespecified in the original analysis plan for the evaluation.  

Conclusions. The Detroit Promise Path was previously shown to have statistically significant, positive impacts on enrollment, full-time enrollment, and credit accumulation. At the four-year mark, there remains a statistically significant impact on college enrollment: program group students enrolled in more semesters. (Credit accumulation could not be measured at four years.) There is no statistically significant impact on credential attainment for the full study sample. These patterns closely mirror the patterns at the three-year mark: The Detroit Promise Path helps students enroll in and stay in college, yet there is no significant impact on students’ likelihood of graduating. However, in an exploratory analysis of the four colleges where the program was implemented as designed, a statistically significant impact on credential attainment of just over 3 percentage points emerges in semester 7 and semester 8.

Why? It is possible that when the program is implemented as designed and coaches are supported by colleges as well as by the Chamber, the Detroit Promise Path has the ability to help more students earn their college credentials. It is also possible that students may yet need more time to graduate; students who enroll in college part time—as most Detroit Promise Path students do—take longer to graduate, and the ongoing effects of the pandemic in Detroit could also be slowing their academic progress. On the other hand, it is possible that the program by itself may not be enough to help Detroit’s students overcome the wide variety of barriers to graduation they experience. Bigger, more systematic changes to institutions or solutions to other issues students face (such as a lack of academic preparedness stretching back into K-12 education or the lack of reliable bus service in many areas of the city) might help more. Colleges and Promise Scholarships looking to improve their students’ academic outcomes might consider the Detroit Promise Path model as one strategy to do so alongside other changes to help students.

The final word on this Detroit Promise Path randomized controlled trial is that the program can help students stay in school, and more can and should be done to raise graduation rates.

Document Details

Publication Type
Issue Focus
Locations
Date
October 2022
Ratledge, Alyssa and Stanley Dai. 2022. “The Detroit Promise Path Evaluation: Outcomes After Four Years.” New York: MDRC.