For this report, the research team had access to five different sources of employment information for individuals from the Portland site of the National Evaluation of Welfare-to-Work Strategies (NEWWS). This presented a unique and important opportunity to look at whether and to what extent the choice of an employment data source matters in studies like NEWWS.

Purpose

This report is part of the Long-Term Employment Outcomes project and presents findings from three sets of analyses that explore whether the outcomes and effects of the NEWWS Portland site differ over a 20-year period when estimated using different data sources. The findings in this report identify insights on the strengths and limitations of the various data sources used. They also provide the workforce and research fields with information on which data sources to prioritize in future research on employment-related interventions targeted to individuals who are poor or near poor.

Key Findings and Highlights

• There are differences in the employment outcomes of NEWWS Portland depending on which data source is used to estimate employment. The employment rates are higher when estimated using survey data and when using data sources with national coverage than when estimated using administrative data that only cover employment in Oregon.
   
• There are also differences in the employment impacts of NEWWS Portland across data sources, though only for some years. The differences are mainly concentrated in the five to eight years following study entry. Around then, NEWWS Portland increased employment by a statistically significant amount when using administrative data only covering employment in Oregon, but not when using administrative data with national coverage.
   
• The evidence also shows little difference in the employment captured by (1) the Oregon state unemployment insurance data and Oregon Longitudinal Employer-Household Dynamics (LEHD) data and (2) the national LEHD data and the National Directory of New Hires data for a population that is poor or near poor.

Methods

The analyses presented is this report involved roughly 4,000 adults who were enrolled at the NEWWS Portland site. This sample includes around 3,500 individuals assigned to the program group and around 500 individuals assigned to the control group, who were subject to a full five-year embargo on receiving program services.

The first set of analyses compares the employment-related outcomes of individuals in the NEWWS Portland site across data sources over a 20-year period. This is done by first comparing the overall employment rates across data sources and then by comparing each individual’s employment statuses according to different data sources.

The second set of analyses examines whether the employment-related impacts of NEWWS Portland differ depending on the data source used. This is done by first seeing whether the employment impacts are statistically significant in one, both, or neither data source and then by examining whether the differences in impacts across data sources are statistically significant or not using logistic regressions.

The third analysis explores whether differences in employment outcomes vary across data sources for individuals with certain characteristics. Here, employment in one data source is used to predict employment in a second data source separately among groups of participants with distinct characteristics. Then, logistic regression is used to see whether there are differences in the likelihood of being able to predict employment in one data source based on a second data source across groups of individuals with dissimilar characteristics.