In a recent blog Jonathan Lewis blasted evaluators as being the enemies of social change. The founder of Opportunity Collaboration claimed that social evaluators have taken over the world of charitable funding. As a result, he claims, small, innovative, and ‘in-the-trenches’ anti-poverty organisations are being neglected in favour of those that can prove their impact. Though he raises some interesting points, his main assertions fall short of the truth. A long debate about the merits and shortcomings of impact evaluation has already taken place under his blog; so I won’t get bogged down in this general discussion.
Instead, I want to pick up on his analogy that evaluators being valued over charities is like art critics being more valued than artists. This situation, he claims, is upside down.
Jonathan’s blog post reveals his commitment to the ‘charity as artist’ mode of thinking. His assertion is that there are a whole load of great organisations out there, making their contribution to the huge task of reducing poverty. What precise contribution are they making? If you take the “charity as artists” way of thinking then the answer is ‘measuring their contribution is very hard, the important thing is that they’re contributing.’ In his vision of things, charities, like artists, beaver away doing their own thing, and contribute in an ephemeral way to the great world of poverty reduction. Trying to count their contribution would be like herding cats, so why bother? Because each charity is unique, comparing them is impossible too. Conclusion? It’s hard (if not impossible) to chose which charities are making a bigger contribution than others to reducing poverty.
I think NPC would prefer to see charities as scientists, or, perhaps more accurately, as thousands of tiny scientific experiments. In this way of thinking, their role is to experiment and test out ways to solve social problems. To do this, they evaluate the impact of projects, share findings publicly and use the findings to learn and improve.
By sharing the findings they contribute to wider knowledge of what works. This makes them part of the great ‘scientific endeavour’ of working out the most effective way of addressing X or Y social problem. Yes, charities are only operating in the world of ‘social sciences’ – findings always need caveats and (unlike biology or physics) no single solution to a social problem is likely to work to the same degree in different contexts. Instead, social science is about building up a clearer and clearer picture of what works and in what setting. With each new evaluation we know a bit more about what approaches are effective and why. Over time, as evaluations accumulate, we build up a body of evidence suggesting ‘what works’. This will only ever be an imperfect picture, but it will be a lot clearer than when you started.
What does this mean for charities and social enterprises? It doesn’t mean they all need to do randomised-control trials (the gold standard in social sciences). Instead, they should all make a start – however small – towards evaluating their impact and sharing their findings. This could be as simple as surveying people before and after the charity supports them to work out any change that had occurred during their support. Then they could publish the results in a short report and stick it on their web-site.
As well as helping charities work out what impact they’re having, impact evaluations can be used to influence the approaches of other charities or government. If the charity has an exceptionally high impact, others might start to take notice, and may even adopt aspects of the charity’s approach. In this way, effective approaches can spread, generating more impact.
There will always be a role for creativity in the charity sector. Charities will always innovate, developing new solutions to social problems. So in a sense Jonathan is right, we shouldn’t let the egg-head evaluators take over – some space needs to be left for the blue-sky innovators. But innovating isn’t enough, you then need to assess the impact and promote approaches that work well.
I’ll leave the final word to Steve Case, co-founder of AOL: “For better or worse, that is true with any new innovation… There’s many good things that come out of it, but also some bad things. All you can do is try to maximize the good stuff and minimize the bad stuff”.