As a leading exponent of impact measurement you might expect NPC to take issue with Caroline Fiennes’s assault on the usefulness of evaluation (‘We don’t all need to throw wellingtons’). But sadly she is exactly right.
Evaluation in the voluntary sector is under-funded, often of poor quality, and too frequently motivated by the wrong thing. If charities evaluate their projects to prove success to funders, rather than from a genuine desire to discover what’s working, it does little to help individual organisations improve what they do. Nor, crucially, does it help the wider sector understand what works and what does not.
Evaluation has a role of course. The third sector has a moral and practical obligation to understand their impact so that they can learn and improve. And yes, letting funders know how your charity is doing is one legitimate function of evaluation. But Caroline makes the point well: when the need to communicate success overwhelms the desire to learn, the result is not evidence but spin.
So how should the sector respond to a problem like evaluation? I would suggest in two ways.
Firstly, impact measurement should be a collective rather than an individual concern. If impact measurement is dominated by the efforts of lone organisations then all we are left with are people better equipped to compete with one another. Any collective understanding of how to improve the charity sector will be lost—it will remain as atomised as ever. NPC is devoting more and more energy to collective approaches for precisely this reason: shared measurement makes it cheaper and easier for everyone to evaluate what they are doing, and shares knowledge across the sector.
All of us should see this as an essential part of our jobs. A robust, shared evidence base would benefit charities across the board. It is a common good. As Caroline puts it, citing research is easier than producing it.
And secondly, it’s time for funders, of all shapes and sizes, to take evaluation much more seriously. Funders, with some honourable exceptions, often remain part of the problem. All too regularly they fritter away small sums on poor quality evaluations and demand the collection of data of which they then make no use. Funders too seldom share what they learn with others, and they don’t always take the measurement of their own impact seriously. These failings are well-documented by now, and there’s no excuse for them.
The greatest missed opportunity lies in funders taking such a narrow view of their responsibility: too few see their responsibilities as going beyond funding individual projects and institutions, or to think about the glue that holds organisations together and helps them perform. The performance of the charity sector as a whole will not be lifted until more knowledge is shared.
Charities and their beneficiaries will ultimately pay the price—but funders face responsibility, too, for creating a sector which struggles to generate good quality evidence.
This is a longer version of a letter published in Third Sector magazine.