Agenda, Scope, and Goals

The first year of this project (Phase 1) includes in-depth market research on the effectiveness of OtterBot messages to encourage financial aid completion. The goal will be to learn more about (1) how students are engaging with OtterBot and (2) how OtterBot can be more effective in helping students receive financial aid and access education and training opportunities after high school. We are conducting focus groups and fielding surveys with students and parents to better understand their needs, questions, reactions to OtterBot messaging, and what could be most useful for them.

OTTERS is using human-centered design and behavioral science to identify participant engagement problems and generate solutions. In addition, we will use data analytics to gain insights on OtterBot usage and engagement. The team will develop an open-source code repository of statistical analyses, a data model, and behavioral science tools, all of which can be used in the future by WSAC and other states that use similar chatbots.

In second year (Phase 2), the team will set up technology workflows for A/B testing to quickly assess various improvement strategies generated through the market research in Phase 1.  After Phase 2, we will seek funding for a rigorous evaluation of a strengthened OtterBot chatbot. 

In addition to these learning objectives, another key goal of this project is to provide training and technical assistance to WSAC so that they will be able to continue the types of analyses done in this project on their own.

Design, Sites, and Data Sources

The OTTERS project is focused on learning about the OtterBot interactive chatbot, a tool that aims to serve all students from families with low incomes in Washington state.

Phase 1 of the project will focus primarily on student characteristics, OtterBot engagement data, and College Bound Scholarship data. The team will also use scholarship data (containing information from both FAFSA and Washington Application for State Financial Aid), National Student Clearinghouse Data, and/or data from the Washington State Education Research and Data Center to better understand the outcome levels in this population and to gain initial insights into the extent to which OtterBot engagement may be associated with outcomes, such as financial aid application completion and postsecondary matriculation. The team will make use of a variety of analytic methods, including funnel analysis, sentiment analysis and natural language processing, clustering, sequence analysis, and variable importance analysis. The exact methods used will be adjusted based on data quality and exploratory analyses.

Qualitative data collection will explore insights from the quantitative data. For example, focus groups and surveys aim to understand why people engage or do not engage with OtterBot. Are messages culturally competent? What environmental and behavioral conditions might explain rates of completion at key steps between different groups? What additional supports might be required to equitably serve Washington college-bound students?