In most sectors, impact measurement is regarded as standard practice. We haven’t found good ways to measure all our outcomes, and sometimes tracking long-term data or getting the counterfactual is difficult, but on the whole there is a recognition that charities do need to understand whether what we do results in change for the better. However, despite how far we have come, impact measurement isn’t delivering all the benefits that we would want.
The current state of play enables charities to make statements about their outcomes. A good start, but I often struggle to interpret those results. For example, looking at a recent evaluation report that said an arts project ‘had a great impact on 30% of the participants’ confidence’, I felt more questions were raised than answers provided. Is that a high number? Or a low one? Does the project therefore represent good value for money?
And then I began wondering about the methodology. How did they found out about this rise in confidence? If they asked people whether their confidence had improved because of the event, how does this compare to another project that saw a 20% rise in confidence but used the more rigorous method of a before and after measure?
The charity has spent thousands of pounds on an evaluation but I am still confused about what conclusions I can draw.
When reading any evaluation, the two main questions I ask are: ‘Is this a good result?’ and ‘Was this a good use of money?’ At the moment, most fail to provides the answers.
Because, in order to know whether the money for this arts project was well spent, I would need to have some kind of benchmark. This would allow me to understand whether a 30% result is good because it compares to an average of 15%. Or I might instead learn that funding could be spent better elsewhere because other projects have an impact on 50% of participants.
To do this requires some common method of measuring. Without being able to place results in a wider context, charities will continue to struggle to know if and how they should improve and the uses of impact measurement will remain limited.
Thankfully, initiatives are moving in the right direction, with sectors keen to understand and demonstrate their impact working together to find standard ways of measuring their results—from employability, to financial capability and violence against women.
NPC has been working with charities in the financial capability sector to find a common understanding of outcomes and the evidence that lies behind them. Impact measurement that is grounded in this approach has the potential to be more than just about data. Creating common outcome frameworks can become the basis for thinking strategically about how the sector is tackling a problem. For example, what sort of interventions does a sector deliver and do these match the outcomes we need? What do we now know about what works and for whom?
We need to find answers to these questions on a sector level so that organisations can work together to be more than the sum of their parts. Good impact measurement can provide them, but it has got to be coordinated.