We use the terms ‘impact’ and ‘impact measurement’ a lot. But what do they really mean?
For me ‘impact’ has the following qualities and conditions:
- Positive. When we talk about impact in the charity sector we actually mean ‘intended impact’. Of course, negative and unintended impacts also occur and shouldn’t be ignored. But in working with charities we are usually stating what we want to happen.
- Meaningful and important, bringing about a change in a recognised social problem. I often say that impact is something a funder or commissioner will be willing to pay for. For example reduced crime, increased educational attainment, better health and quality of life.
- Sustained. Impact is the lasting effect of a service, usually after it has ended.
- Achieved by individuals and communities themselves. Crucially, our services don’t make people healthy or educated—people do that for themselves. Our services aim to give people the attributes, capacities and resources—which we call ‘outcomes’—to help them achieve this impact.
So given this definition, what is ‘impact measurement’?
- Long-term. If ‘impact’ means something sustained, then we have to have a way of getting data from people after our work is done—potentially long into the future. In practice this means longitudinal research, which is the hardest and most costly type of research to do.
- Comparative. Even if we can get longitudinal data we still have the problem of attribution: how do we know our programme has been the thing that made the difference? To test this formally we need a counterfactual or control group.
- Robust. To actually ‘measure’ impact we need to do both 1) and 2) with enough rigour and scale to be confident in the results.
Using this definition, it should be clear, that ‘impact measurement’ is actually very challenging to do. However, at the moment we tend to use the phrase ‘impact measurement’ much more loosely; as a kind of catch-all for all the different types of data that charities could collect about the effects of their work. While I’m happy that we have a term that prompts people to start thinking about the issues generally, I worry that this particular term causes confusion and can lead people down the wrong track.
So I would like to see us to use the more formal definition of ‘impact measurement’ above; and then to use the term more sparingly. I think this will help bring some useful clarity:
- It should lead to more conscious decisions about when ‘impact measurement’ is appropriate and when it is not (for example, when evidence already exists to show that an approach works)—and prevent some of the poor-quality or misguided ‘impact measurements’ that we see too often.
- It may educate some funders and commissioners to only expect ‘impact measurement’ in exceptional circumstances (and only if they are willing to fund it).
- It could bring greater recognition of the importance of other types of data, such as who is using a service and what they think of it (which we might call ‘performance management’ or ‘impact management’).
- It may help organisations that are currently struggling to ‘measure impact’ to feel more confident they have the right approach given their circumstances.
The phrase ‘impact measurement’ is so ubiquitous that this change will not happen overnight, and perhaps it doesn’t need to. So long as one of the first things people learn when they engage in the topic is that ‘impact measurement’ in its true sense is a tall order, and there are a great many other useful things charities can measure to improve their effectiveness before they tackle this.
Let us know what you think about all of this in the comments below.
Still confused? Take a look at our blog about the 5 different types of data.