A foothold on the uphill struggle towards measuring impact

From a distance a hill might not seem too difficult to climb. Similarly, it is very easy to tell charities to ‘measure your impact’, but when you get up-close you appreciate how hard this.

There are two big challenges: impact itself takes time, so you need to stay in touch with people to measure it; and impact is caused by many things, so you need a way to determine your own contribution. Addressing these requires expertise and resources rarely available to the sector. So most of our effort goes on finding other routes that might take us half-way up; like measuring short-term outcomes, perceptions of impact, qualitative research and so on.

Sometimes the government could help by allowing charities to access to official data about the outcomes their service users achieve. But this raises an obvious privacy issue; we can’t allow all and sundry to look at people’s health, education and police records.

The Justice Data Lab (JDL)—suggested by NPC and run by the Ministry of Justice—is an elegant solution. It lets charities that have worked with offenders to analyse reoffending rates of their service users without compromising anyone’s privacy.

Because of this we at NPC are a bit obsessed with the JDL. It is the first service of its kind, and we see it as the first toe-hold in a mountain we’ve been clambering at the bottom of for a long time. The eventual aim is that charities in a wide range of sectors (beyond criminal justice), will be able to access to government data and with it an indication of their impact—effectively bypassing all the challenges mentioned above.

But even this small step forward has been difficult and the JDL isn’t perfect. With this in mind, today we publish an academic review of the JDL—commissioned by NPC and undertaken by the University of Middlesex. The review has already contributed to recent changes to the JDL that have made the reporting of results more accessible.

One of the points we are keen to emphasise in the review is that the benefits of the JDL are not just restricted to those organisations that use it. Rather it exists to help us all learn about what is effective in reducing reoffending. Hence the review looks across all the reports to see what they tell us so far. Among the main observations are:

• Most interventions tested through the JDL have been associated with a decrease in reoffending (although because of the rules of statistics only a subset of these are deemed ‘significant’).
• Success rates are slightly higher for voluntary and community services than for private and public organisations. But unfortunately we can’t shout about this because not enough voluntary organisations have used the Justice Data Lab yet. If more of us used it, the sector could make a stronger case for their contribution as a whole.
• There is some early aggregate evidence that educational interventions run by the Prisoners’ Education Trust and those programmes focused on employment are more likely to demonstrate positive, statistically significant effects for reducing reoffending.

A further observation from the report is to remind us of the scale of the challenge in reducing reoffending. Across all the services analysed so far, the average effect size is around two percentage point reduction in the one year reoffending rate.

Let’s not underplay this. Nationally that’s a lot less crime and many hundreds—if not thousands—of lives turned around. But it also confirms how difficult it us to steer people away from crime, and how it’s fairly unlikely that one intervention can have a dramatic effect by itself.

The Justice Data Lab shows that nobody has an easy way up that particular mountain. And that’s exactly why the Lab is needed—so organisations can learn from their own work and others’ approaches; can see how far up they are, and how else they can progress.

Take a look at the full review of the Justice Data Lab.

Find out more about the Justice Data Lab and how you can use it.