Evaluation use in Science Festivals

by | Apr 24, 2019 | On the Blog, Publications

Drawing on data from surveys and interviews, the EvalFest team found high rates of evaluation use among its science festival partners across the United States.

Recently published in Evaluation and Program Planning, this article uses data from festival team interviews (n=23) and surveys completed by individual partners (n=45) to document examples of evaluation use occurring within a community-created multisite evaluation (MSE) embedded within a community of practice (CoP).

As a community-created MSE, EvalFest’s design and structure differs from traditional MSEs in important ways.  MSEs typically encompass two or more sites and include systematic cross-site data collection across non-uniform contexts, but variability across sites and inconsistency in evaluation design are often cited as shortcomings of results (Sinacore & Turpin, 1991; Straw & Herrell, 2002). The EvalFest community was able to circumvent many of the challenges by applying a negotiated centralized evaluation model that entailed: (1) creating local evaluations; (2) forming the central evaluation team; and (3) negotiating and collaborating on the participatory MSE.

Through an arduous process of negotiation and collaboration, the EvalFest community co-created a set of shared measures with unique variability that were collected annually using shared data collection systems and data analysis and reporting tools provided by the central evaluation team. The result is a dataset consisting of over 30,000 data points from festival attendees across the U.S. and an online dashboard that provides access to each partner’s data as well as the larger sector-level dataset.

The community-created MSE approach promotes a broader range of evaluation use than traditional models. Drawing on the literature on evaluation use, four types of evaluation use were coded: instrumental, conceptual, symbolic, and process. All partners cited at least one type of use and 11 partners (48%) provided examples of all four types of use. One in five partners also reported using the sector-level data generated by the entire EvalFest community, which underscores the unique value of the design characteristics featured in EvalFest’s community–created MSE.

Although the EvalFest case study provides evidence of evaluation use within a single sector, the EvalFest team believes the design characteristics featured in the community-created MSE can be applied broadly to promote evaluation use across other sectors.

Peterman, K. and M.J. Gathings. (2019). “Using a Community-Created Multisite Evaluation to Promote Evaluation Use Across a Sector,” Evaluation and Program Planning, 74:54-60.

Meet the Catalyst Team

We're glad you're here! We are evaluators, researchers, and strategists who study and support STEM education programs. We specialize in the in-between spaces where we can use data to get context, tell important stories, and help our clients take strategic action in their programs.


Recent Blog Posts

More Blog Posts to Read:

Tengamos una plática (Let’s have a talk)

Tengamos una plática (Let’s have a talk)

The country's Hispanic population is increasing daily, leading to the incorporation of new languages, traditions, customs, and cultures. While it is important to recognize that the Hispanic population is not monolithic, it is also true that many Hispanic populations...

read more
Reflections on academic publishing

Reflections on academic publishing

At Catalyst Consulting Group, we love a good, data-driven story. We recently published an article in the Journal of Geoscience Education that is just that. While Catalyst and our larger group of co-authors for this paper have a history of publishing this kind of work,...

read more
Reflecting on Evaluation in Service of Racial Equity

Reflecting on Evaluation in Service of Racial Equity

At Catalyst Consulting, we enjoy honing our skills, growing, and deepening our understanding of evaluation and research methods. We come together once a month for a lively discussion on a variety of topics. Over the past year, we spent three of these meetings...

read more