Courage, theory and creativity

Causality, correlation, attribution: three words that strike fear—or should!—into the heart of any charity researcher, evaluator, or performance manager. Or to put it another way, how do you answer the all-important but awkward question: ‘what difference do we make?’ If your charity did not exist, would your users fall through the cracks, would they find other services that are more or less just as good, or, even, would they possibly somehow find their lives getting better? On October 16^th at NPC and Third Sector’s second annual Charity Impact Measurement conference, I, along with three other evaluation ‘experts’, will be talking about how to answer this question.

The answer is, unfortunately, unknowable, if you require certainty. We do not have an alternative universe without your charity to compare to the current one. But the problem facing charities is they are caught between the demands of (some) funders who require some kind of ‘proof’ of their impact, and evaluators and academics who tell them that proof is impossible.

What is to be done?

Let me offer three other words: courage, theory, and creativity. You need these if you want to get a grip on causality.

You need courage because before you put resources behind understanding what can be attributed to your charity, you need to be willing to accept whatever answer your evaluation comes up with and deal with the consequences. You may think you may make a big difference, but you should be open to the possibility that you don’t make the impact you think you do. In particular you need to think about how staff, trustees, and funders will react. If you have a culture of evaluation and continuous improvement, then you should not have a problem if disappointing results arise. But if your culture finds doubt difficult to handle, you must prepare yourself in case you find bad news. You also need some courage to engage in discussions with you funders about what level of proof is possible and appropriate. This may not be an easy conversation.

You also need theory because to understand whether you bring about change, you should be clear as to how you bring change about. Theory describes the causal links between what you do and what you hope to achieve. If you cannot describe these causal links using words, you will be hard pressed to collect any data that shows causality.

Finally, you need to be creative, because, typically, you do not have sufficient resources to test your theory using the gold standard methodology, namely randomised control trials (RCTs). For an RCT you deliberately choose not to work with a randomly selected sample of clients because you want to see what happens to them compared to the people you do work with. The differences in the outcomes of the two groups usually counts as the impact you cause. But RCTs consume time and resources that most charities do not have. Instead, you have to figure out how to best work with what you do have. This is where creativity comes in. With a few changes here and there to data you currently collect, you can probably make stronger claims about attribution than you currently do. You will not be able to convince someone who is not open to being persuaded. But you will learn things about your impact you do not know already. To learn more about what these changes might be, come along to the conference!