Evidence through rose-tinted glasses

I recently spoke to the All Party Parliamentary Group on Boxing’s Inquiry into the social contribution of amateur boxing clubs. The question I and other panel members were asked was: ‘How can sport demonstrate its social impact more effectively?’ A fair question, but given the state of evidence on the social impact of sport (there is little) and the level of investment in generating such evidence (very low), it is possibly jumping the gun. The first, and I think more important question, is how can clubs generate evidence about the social impact of sport?

Here is an illustration. A few years ago I founded, and now chair, a “veterans” football club in the village where I live. I am pretty confident we have minimal social impact beyond our own “well-being”. But as an experienced evaluator I could quantify that change in well-being using a survey, while suggesting potential benefits to the health service through the increased exercise of a range of middle-aged men at risk of high blood pressure, stroke etc.(while conveniently ignoring our increased use of the local minor injuries unit!).  I could also calculate the positive economic impact on the village businesses (ie, the pub and the Indian restaurant).

If I was so inclined, I could also survey our families and tell a story of detachment from our children (a number of us, on some days, choose to play football rather than watch our sons/daughters play) and increased marital tension (one of our squad has spent more than one night on the sofa!).

Of course this is trivial, but the point should be clear.  Evaluations with weak designs can be useless, or worse, misleading.  In place of strong evaluations and monitoring data, I often find charities having to tell their story in the manner of a missionary—trying to persuade others of their impact through the use of stories of individual conversions or miracles. This is not a criticism, and there is nothing wrong with the use of such stories. Indeed, these bring life and immediacy to discussions about the impact of a charity in a way that numbers cannot. But the problem is that, by themselves, stories or case studies are only likely to convince existing “believers”. Without the prospect of getting better and better evidence over time, I would find this very frustrating.

One challenge our Measurement and Evaluation team at NPC faces is requiring people who are passionate about their cause to be dispassionate about the evidence that supports the cause. I think this is a better combination than being passionate about their cause and viewing the evidence through rose-tinted glasses. I expect that if you know the evidence and all its weaknesses, you are a more powerful advocate for your cause than simply being a cheerleader. But this is my speculation. I would love to hear from you if you have any examples that either support or count against this view.