Phase 1: Planning
Researchers sometimes aim to understand the effectiveness of an intervention in a broad population of colleges. When that is the goal, it is essential to develop a thoughtful college recruitment plan that helps ensure the study sample of colleges reasonably represents that broader population of interest. Doing so strengthens the generalizability of the findings and their relevance for policy and practice.
The Generalizer is a powerful tool that is designed to help researchers identify a sample of colleges that reflects their target population. It guides researchers through the following steps:
- Defining the inference population. Clarify the types of colleges the findings should apply to and identify which colleges meet those criteria.
- Identifying important college characteristics. Select the characteristics—such as size, region, or control (public or private)—to be represented in the study sample.
With interactive visualizations, The Generalizer helps researchers create a recruitment plan that balances feasibility with representativeness. It produces ordered lists of “similar” colleges that can be used as a foundation for recruiting institutions.
In addition, this report—commissioned by the Institute of Education Sciences (IES)—provides an excellent overview of the steps involved in planning a study with generalizability in mind; implementing the plan; and assessing, analyzing, and reporting findings.[1]
[1] Elizabeth Tipton and Robert B. Olsen, Enhancing the Generalizability of Impact Studies in Education, NCEE 2022-003 (U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2022), website: https://ies.ed.gov/sites/default/files/migrated/nces_pubs/ncee/pubs/2022003/pdf/2022003.pdf.
[2] Elizabeth Tipton, “Stratified Sampling Using Cluster Analysis: A Sample Selection Strategy for Improved Generalizations from Experiments,” Evaluation Review 37, 2 (2013): 109–139, website: https://journals.sagepub.com/doi/abs/10.1177/0193841X13516324.
Key Resources
Tool
The Generalizer
Identifies colleges that are representative of a population
Report
Enhancing the Generalizability of Impact Studies
IES guide on designing studies to improve generalizability, with tips on defining populations, sampling, and recruitment
Journal Article
Sampling Using Cluster Analysis for Improved Generalizations
Proposes a sampling method to improve generalizations in education RCTs[2]
A subscription is required to view this article.
Website
Generalizability.org
Provides resources for researchers aiming to conduct evaluations that are generalizable