Impact or no impact? It’s a little more complicated than that…
20 October 2015
Today thousands of people around the world are celebrating an event that comes around only once every five years. Some obscure lunar event perhaps? Think again. Today is World Statistics Day! As one of NPC’s data enthusiasts, I’ve had it marked on my calendar for some time. But why should anyone else take notice?
Charities often struggle to prove that they are making the impact that they exist to achieve. One way of proving this impact is to use data and statistics. By comparing a charity’s beneficiaries with a similar group of people who have not accessed that charity’s services (known as a ‘comparison group’), it is possible to see whether a charity’s work has achieved any kind of result. These results are judged by their statistical significance ie, whether a causal link can be established between the charity’s work and outcomes for its beneficiaries.
The troubling thing about something being deemed statistically significant or insignificant, however, is it implies that a charity’s programme either has an impact or it doesn’t. This is a rather binary way of looking at things. With either result, there are several things to consider:
-
Statistical insignificance can mean more research is needed
Statistical significance does not show conclusively that an intervention causes a particular outcome. It is better seen as a test of how precisely the relationship between an intervention and an outcome can be measured with the data collected. This means that it is difficult to know whether statistical insignificance is the result of a) having an impact, but not having sufficient data to confirm it, or b) not having an impact over and above the comparison group.
-
Economic significance is also important
The potential implications of a result, whether statistically significant or not, matters too. We could have a statistically significant result, say a 1% improvement in an outcome that is delivered by an expensive intervention, but the high costs of the programme mean that the result is not ‘economically significant’.
-
Is the right outcome being targeted?
Improvements in long-term and ambitious policy outcomes can take many years to measure. A youth employment programme’s impact on long-term unemployment, for example, may be statistically insignificant if measured several months after the programme has finished, but it could be statistically significant if measured years later. This makes measuring intermediate outcomes—like improved work readiness skills—particularly important.
-
Does our comparison group benefit from other interventions?
If we are comparing the outcomes of a charity’s service users with a comparison group, then it is important that the comparison groups differs only in that one group has received the service and the other has not. But what if the comparison group received other interventions? In this instance, statistical significance may become a measure of the relative impact of a programme compared to other programmes. Statistical insignificance, therefore, would merely suggest that your programme is no worse than other programme that are out there.
-
How good a ‘match’ is our comparison group?
Some approaches to identifying a comparison group can only find matched comparisons for individuals on characteristics for which we have data available. But what if there are other factors, such as motivation to receive an intervention, that are likely to influence our final outcome and for which no data is available? Often, limitations in the matching process offer an explanation as to why a difference has or has not been found.
The quote attributed to Disreali about ‘lies, damned lies and statistics’ may come to mind upon reading the above list. But while it may seem like a list of caveats, it in fact shows exactly why statistics are worth celebrating: when approached with the correct consideration, they help us to comprehend the complex and multifaceted relationship between cause and effect. The use of statistics is vital in helping many of us to understand the world around us. That’s why we should all celebrate World Statistics Day.
For more information on control groups and statistical significance, download NPC’s short guide.
