Five steps to evaluating systems change

Step 1: Set expectations 

In general, evaluating systems change requires more patience and acceptance of uncertainty than normal. That’s because systems change is non-linear and takes time; lots of different factors contribute to that change; and you’re sometimes one step removed from the people you’re ultimately trying to benefit. 

There are two core elements to this step: 

• Identify your key stakeholders whose expectations about evaluation and evidence you need to manage. The most important stakeholder group might be your funders. But it might also include the people or communities you’re supporting, trustees, staff, delivery partners, policymakers, and even evaluation organisations. 
• Make your case about patience and uncertainty. You can use the content from this guide, as well as content from elsewhere. The more you can apply that content critically and carefully to the specifics of your unique systems change context, the more compelling your case will be. 

Step 2: Agree your overall aims 

The aims of any evaluation work can essentially be two things: improving the work you deliver, in order to increase its impact; and proving and evidencing your impact, for accountability and/or to attract funding and support. 

We think learning and improving is even more essential than normal for systems change, and should always form a part of systems change evaluation; whereas systems change generally makes ‘proving’ your impact more difficult. Which means the main part of this step is deciding whether it is a worthwhile use of resources to try and prove or evidence your impact. To help do this, there are several interconnected questions to ask yourself: 

• What’s the need and demand for the evidence? Who wants or needs the evidence, and what will change off the back of it? Do you agree that need is justified and worthwhile? How robust and definitive does the evidence need to be? When is it needed by, and why? 

• What is possible methodologically and practically? Would it be possible in your specific context to meet the required robustness and timeline? (Consider issues from the previous section like whether you can access impact data, how long it would take for the impact to happen, whether there might be natural comparison groups (e.g. across geographic areas), how many other factors are likely to be contributing to the impact you are trying to achieve).  

• How much resource will that take? Both in terms of time and monetary investment in internal capacity, and likewise if you commission external support.

• How much resource is available? Is it enough to cover what’s needed? What’s the opportunity cost of spending this? 

When the answers to all these questions are aligned, it makes sense to try and prove/evidence your impact. Whereas if there isn’t a clear or worthwhile need, and/or that need likely can’t be met, it might make sense to focus resources on delivery and learning instead. 

Step 3: Identify your evaluation questions 

There are three main ways in which the questions you want to evaluate and learn about might be different when it comes to systems change:  

• Understanding broader context is more important than ever, because systems change is inherently and explicitly about changing that context.

• Your impact might be less tangible or visible than normal – on things like mental models, power dynamics, relationships and connections, which you can’t easily count or measure.

• You need to interrogate your own role within the system, because you are always an active part in it, leading to both intended and unintended consequences.

For these reasons, we think systems change requires a different set of evaluation questions to normal. We have compiled some sample questions below. Feel free to draw on these questions, directly or as inspiration, when setting your own evaluation questions. 

Step 5: Plan your approach 

Regardless of your evaluation aims, questions and methods, there are characteristics of systems which mean your evaluation approach will always have to incorporate the following three principles. 

The system is always changing, so you must too.

The nature of complex systems means they will always be shifting and evolving. This means you have to continually adapt your delivery activities to respond to these shifts, and adapt your evaluation questions and potentially methods correspondingly. There are two key enablers to doing this well: 

• Culture. Taking an agile and adaptive approach to your evaluation work will need understanding and buy-in from staff, trustees, funders, partners, evaluation suppliers, and the people and communities you’re supporting. We recommend facilitating group sessions where everyone seeks agreement on what an adaptive learning culture looks like in practice, what the barriers and enablers are, and how people can be incentivised and held accountable.  

• Structures and processes. Without them, adaptive approaches can leave you feeling like you’re navigating complexity without a compass. Your evaluation questions can provide direction and security, as a core reference point that can be updated as needed. It can also help to plan the timing of methods and data collection in smaller windows than normal; have specific review points and spaces, between which things are kept as consistent as possible; and clear and transparent governance structures for signing off changes in direction.  

There will be many perspectives on what matters and how impact is occurring, and you should try to incorporate them all.

In systems change there are often a huge number of different stakeholders who are part of that system, and they will all have different perspectives on the system itself, what’s changing and why. So to evaluate and learn effectively, you will need to seek out the full diversity of perspectives from across different stakeholders – not just from the people you’re ultimately trying to benefit. 

We think there are three key things to consider here: 

• Which stakeholder groups, and which evaluation questions? Different stakeholders occupy different positions in a system and therefore have access to different forms of knowledge. Start by identifying which groups are best placed to answer which evaluation questions, to help ensure that evidence is both relevant and proportionate. 

• What kind of diversity within and/or across those groups? It’s important not to treat stakeholder groups as homogeneous. Ethnicity, gender, and other personal characteristics might influence people’s views and experiences, at least as much as their role or position in the system does. You can use stakeholder mapping to identify underrepresented voices, intentionally recruit them, and review sample composition throughout the evaluation rather than just at the end. 

• What are those groups’ priorities and potential biases? Whilst all perspectives are valuable, none are entirely neutral. Once you have data and insight from across all your groups, it’s not enough just to aggregate it – you need to interrogate how those views are shaped by the incentives, goals and constraints associated with people’s positions in the system. Practical approaches include explicitly discussing stakeholder interests during analysis, triangulating evidence across multiple sources, and – where possible – making sense of the findings and different perspectives together, will all stakeholders in the same room. 

Effective systems change requires collaboration, so shared learning is more important than ever.

Sharing learning and insight is vital in maximising the impact of your work within a system, because your work cannot be disentangled from the complex interactions and influences of other actors in the system.  

This shared learning can come in three main forms (none of which are mutually exclusive) – decide which of them are most appropriate and useful for you and the system(s) you’re operating in, and plan for them at the very outset of your evaluation work. 

• Sensemaking together: this means not only bringing different groups together as one (rather than engaging with them separately), but also seeing this is as an exchange of insight and learning rather than an extraction. Good practice also includes following up afterwards to share the collective learning in writing. 

• Sharing learning even more widely: there’s only so many people you can fit in a room! So amplify the learning to as many people as you can from each stakeholder group, through the connections, channels and spaces that will most effectively reach each group. 

• Evaluating and learning in partnership: it might make sense to go one step further and go through the whole evaluation and learning process in partnership with one or more other organisations – pooling capacity and expertise to collect data, organise sensemaking and share learning.