Comparing apples and oranges?
1 October 2012 3 minute read
A social enterprise that I really respect and I consider to be doing great work did not want me to quote their evidence of impact. It was not that the results were negative, but they had decided to take a rigorous approach that really looked at the effect they had had in addition to what might have happened if they had not been there. Sadly others looking for the same funding and contracts had not done the same and so were able to quote figures suggesting greater impact.
It got me thinking about how impact measurement is being used. Our research has shown that there is a huge amount of discretion involved in any impact measurement. You can decide which indicators to use, what timeframe to look at, and who to talk to. If you are looking to examine your own performance, to learn and to decide which direction to take in the future, using discretion does not matter because you can use the same approach each time. I don’t have a problem with people using their discretion or judgement as long as this is set out clearly and consistently.
I do get worried when people compare results of one evaluation with another that has taken a different approach. It is easier to do these sorts of comparisons with measurement approaches that try to show value for money by putting on a monetary value on impact. Sometimes this is presented as a ratio – for example for every pound invested, there is a social value of five pounds. This can be useful but in a review of these approaches we show the dangers of comparison. You can only compare if you have used exactly the same approach, with the same indicators, same methods and same assumptions. Otherwise it is just comparing apples and oranges.
So where does this leave the charity I started with? They have a good ratio but if they publish it, they are worried they will be compared unfavourably with others. Another charity I am working with found that their competitors were using a highly spurious approach to reach a higher ratio figure and then asking funders to switch funding based on this. I am also noticing a steady rise in ratios being reported as there is pressure to show bigger and bigger returns- what you could call ratio inflation – the record so far is £60 for every one pound put in. My advice is to educate the users of the information (such as funders) and look at how we can build trust in good measurement though auditing impact reports and being transparent in what is being measured.
I am a great fan of measuring impact but I worry about how it is being used and how it is being abused. It is such a powerful tool that it has to be used carefully or it might loose its legitimacy. Measure your impact, but be sure to be transparent and educate those using the results – especially if they are funders.
Professor Fergus Lyon is part of the ESRC Third Sector Research Centre at Middlesex University.