How to measure big societal change?

We love learning about new methods! And we recently had the opportunity to do just that. Last year, Catalyst was hired by a private foundation who is funding professional organizations to make big societal changes. Our role was to join the client’s team to create an external evaluation review process that will happen about 5 years after the organizations are funded. No one expects the groups to have made their full impact in that time, and so how will the foundation know if they are on the right track? What may be the indicators that the groups are on their way to having their intended impacts? What if the groups are having unintended impacts – how can we design an evaluation that will pick those up? 

We had a few ideas, but decided to reach out to our evaluation networks to ask for recommendations. People shared several suggestions with us including Appreciative Inquiry, Journey Mapping, Outcome Harvesting, Ripple Effects Mapping, and Value Creation Stories. Ultimately, the two that were most relevant for us that we decided to do deep dives on were outcome harvesting and ripple effects mapping. We share a little more about each plus our recommended resources below.

What is Outcome Harvesting?

Outcome Harvesting flips more traditional evaluation approaches on their heads. Often we define our intended outcomes and design instruments that will measure progress toward those outcomes. Instead, outcome harvesting starts with the outcomes – it looks for what has changed and then works backwards to determine the likely causes. Outcomes and their causes are substantiated and verified by people who are independent but knowledgeable. As you might suspect, this substantiation makes Outcome Harvesting a time- and resource-intensive method! This approach is particularly well suited for complex projects that are likely to have unpredictable (even negative) outcomes. For more information about outcome harvesting, we recommend reading Wilson-Grau & Britt (2014) available here.

What is Ripple Effects Mapping?

Ripple Effects Mapping brings together stakeholders to share stories of the project’s impact and visually map out the changes or “ripples.” The process begins by applying an Appreciative Inquiry* approach, where participants interview each other in pairs about the best parts of the project – the elements that they would want to carry forward and build upon. Participants then come together as a group for a facilitated discussion and reflection to identify the key effects from the interviews. Then the group works together to create the map to depict the causal chains between all the effects. After the mapping session, facilitators may need to further reorganize the map and substantiate the data through additional interviews. For more information about outcome harvesting, we recommend reading A Field Guide to Ripple Effects Mapping, available here.

What we picked and why.

While there were many strengths of Outcome Harvesting, we decided that Ripple Effects Mapping was the better fit for this evaluation plan because it uses both a participatory approach and utilizes elements of appreciative inquiry. The participatory approach aligned very well with the foundation’s values – it was important to them that all stakeholders have a voice and share their stories. We also saw great potential for the appreciative inquiry questions to help the funded organizations identify successes for them to build upon.

*Appreciative inquiry is an organizational change approach that goes well beyond the questions used in Ripple Effects Mapping. The method begins with the same types of questions used in Ripple Effects Mapping (e.g., What is working well? What gives the project life?) in the inquire phase. In the subsequent three phases, participants build upon the answers generated to imagine what might be, innovate what should be, and implement who will do what (see Preskill & Grindle, 2015 available here).