Small rural colleges are a rarity in higher education randomized controlled trials (RCTs).^[1] This is despite nearly one-third of all colleges in the United States being located in a small town or rural area, according to the National Center for Education Statistics.^[2]

There is wide agreement among researchers and practitioners that there is a dearth of research evidence on what works in rural colleges and for rural students. MDRC has been trying to boost the inclusion of small rural colleges in multisite RCTs where a variety of institution types can take part as well as developing RCTs that are focused on rural colleges and rural students. This paper presents observations and thoughts from several projects to help inform other researchers on considerations that will support the inclusion of these colleges while preventing threats to validity in the research.

Why Including Rural Colleges in Causal Studies is Worth the Effort

Including rural colleges and students in causal studies is vital to ensure the usability of rigorous higher education research. The majority of American college students attend college within 25 miles of home.^[3] Recognizing how many students are attending rural institutions—and how few of those institutions are reflected in the rigorous, causal evidence of “what works” to improve graduation rates—researchers must find ways to include these students and colleges, both to improve studies’ generalizability and to improve the trust that policymakers and practitioners have in research results for their contexts. In the words of one college leader, “I don’t care what happens in New York City or Chicago. I want to know what works in a college like mine.”

The reasons for rural colleges’ historical omission are perhaps obvious. Rural colleges often have small enrollments, making it hard to reach a large enough sample size in an RCT to detect a statistically significant effect. Research budgets are limited, and recruiting three colleges that each serve 5,000 students makes reaching the goal sample size much easier than including a college with an enrollment size that is just a small fraction of that. There are other reasons: many rural colleges are some distance from city centers or airports, and many researchers are not eager to travel out to them. Many rural colleges are short-staffed and lack the capacity to take on research projects, or would find it impossible to implement a program or policy only for a subset of students. Finally, for historical reasons, many rural populations do not have trusting relationships with researchers or may even resist participating in research projects, as previous research activities have been extractive or exploitative. Such skepticism persists in part because the larger education research field often fails to provide direct value to the colleges where research takes place.

It is with these issues in mind that MDRC researchers have tried to improve the inclusion of rural colleges in multisite RCTs. This paper presents some considerations for others who may want to do the same.

Small Sample Sizes

The most obvious challenge with small colleges is their small sample size. The average rural college enrolls fewer than 1,000 students and the average town college enrolls fewer than 1,200 students. For research studies testing programs or policies that affect only a subset of a college’s student body, many researchers assume that it is a nonstarter to consider including a rural college.

In typical multisite RCTs, the main analytical sample is a pooled sample across colleges. This significantly alleviates the question of the total sample size and its effects on statistical power to detect an impact, making the inclusion of a smaller college almost always possible in multisite trials from a sample size perspective. Even when a pooled analytical sample is primarily obtained from urban and suburban colleges, the inclusion of rural colleges is likely to make the study’s results seem more trustworthy to many rural practitioners and policymakers. Including rural colleges may also contribute significantly to qualitative data and implementation lessons.

Anecdotally, MDRC’s postsecondary education studies have found that individual-level intake in rural colleges often goes more smoothly than at other colleges in the study, with a higher rate of the eligible population recruited successfully into joining the study. A common MDRC rule of thumb is that one-quarter to one-third of eligible students can be converted to join an RCT (depending on the intervention, the outreach plan, and other considerations). In rural colleges, these rates have been much higher—often as much as half of eligible students have chosen to join the study. In the Montana 10 RCT, for instance, every single rural college met their sample size goal sooner than the participating colleges located in cities because they succeeded in recruiting a higher proportion of the eligible population to participate in the intervention and evaluation.^[4] This may be because of the rural colleges’ high rates of personal and frequent interactions, as well as the trusting relationships between college staff members and students in the smaller environment. It may also be because college staff members have fewer research burdens placed upon them or, as often expressed to this researcher, feel excited to be participating in a study for the first time.

In MDRC’s experience, all colleges want to see their own individual study results in addition to the pooled sample results. With small colleges, there are extra caveats around small sample sizes and statistical noise, but in every case, colleges were very understanding of those caveats and very eager to see and co-interpret their own findings. Do not let this stand in the way of sharing their outcomes with them one-on-one.

Effects of Rurality Distinct from the Small Sample Size Issue Alone

There are other considerations for implementing RCTs with rural colleges that researchers should consider.

Relationship-building and study support. Researchers must be sensitive to the historical context of any study location. Many rural communities are among those historically harmed by research. For example, the history of extractive and exploitative research in Appalachia, Native American communities, and the Black Belt still resonates strongly in those regions today and for good reason. It is imperative that researchers avoid “helicoptering in” for these projects. Instead, researchers should spend time building relationships, listening, co-designing research questions or measures, and reporting findings as the project progresses, not just at the end of the project in a paywalled journal article. Including researchers originally from the region can help with initial rapport-building or advice on how to approach the colleges. It is also possible that colleges will need additional support from researchers to participate in the study. For instance, many rural colleges are understaffed (or, sometimes, un-staffed) in the institutional research department. Streamlining data requests helps reduce burden on staff members. Examples might be collecting data from state agencies instead of the colleges themselves, asking colleges for the same data delivery file structure that they already submit to their state agency or to federal data collections (such as the Integrated Postsecondary Education Data System, or IPEDS), or reducing the number of fields to only essential data elements.

Service contrast. Ensuring a meaningful service contrast is a concern in all RCTs. In rural colleges, there is the possibility of both stronger- and weaker-than-average treatment contrast. On the one hand, in rural colleges, students in the control group are less likely to be able to access similar services elsewhere: like all rural places, the overall availability of resources and the number of providers is lower than in urban and suburban areas. As a result, the provision of a new service may produce a stronger treatment contrast than in heavily resourced colleges, where students in the control group may, as part of the campus counterfactual, enroll in any number of other programs or support services.

On the other hand, the small size of rural colleges may produce insufficient treatment contrast or none at all. On smaller campuses where there are fewer staff members, the diffusion of an innovation can occur more quickly as staff members learn best practices from each other; with fewer students around, it is likelier that the students experiencing a “new normal” on campus will include control group students. Sometimes small colleges must make decisions based on their limited staffing that have implications for an RCT’s service contrast. In one Montana 10 college, for instance, in order for a freshman seminar course to enroll enough students to operate, the college had to open seats to students who were not in the program group, including, inevitably, students in the control group. Here, the college had to choose between diluting the intervention, by not including one intervention component, or diluting the service contrast, by allowing some control group members to experience part of the intervention. This is an important area of inquiry for researchers especially because it is not a permanent condition—service contrast can evolve over time or vary by semester or study cohort.

Crossovers. Any RCT testing an intervention given only to some students carries some level of risk of crossovers between experimental groups, where some students in the control group receive all or some of the program components. One of the most common causes of control-to-program crossovers is well-meaning college support staff members who continue to assist control group students. (This researcher has seen no evidence of higher rates of crossovers in rural colleges from other causes, such as miscoding of program or control group students in internal systems.) This is understandable from a staff member perspective. Unfortunately, it also dilutes impacts and makes it less likely that the study will generate evidence that the intervention is working.

MDRC’s common advice for support staff members encountering questions from students who are not part of the intervention is to refer these students to other staff members and services on campus. But what happens if there is no referral to make or if the referral is to the same person?

As one adviser at a rural college explained, “The person I have to refer out to is myself.” When advisers serve multiple programs or take on other roles on campus, which is a common occurrence, there may be no other place for students to land and staff members may continue to serve some control group students. Researchers should work with college staff members to understand the extent to which crossovers are happening. For instance, is a program staff person chatting with a handful of highly motivated students or giving comprehensive advising to most of the control group? Researchers should also strive to understand the comparison in dosage of intervention components. For instance, in an advising program, is there a difference between program and control group advising meetings, even if there are control group students seeking out program advisers? This can inform the RCT findings and provide useful lessons for other colleges that are scaling up interventions.

Conclusion

This paper presents thoughts on considerations to reduce the likelihood of confounders in RCTs that include rural colleges. This paper is meant as a starting place for researchers, and MDRC’s rural higher education research staff members are eager to hear from others doing this work on what they have observed, the successes and challenges, and ways to move the field forward. You can contact the author directly at alyssa.ratledge@mdrc.org or reach the larger staff of MDRC’s National Rural Higher Education Research Center at RuralHigherEd@mdrc.org.

This is a publication of MDRC’s National Rural Higher Education Research Center. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305C240065 to MDRC. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.