Government sharing data with charities improves impact – that’s why we need more Data Labs

How do you evaluate if your charity project is working?

One the biggest problems in evaluation is having good, longitudinal data to see what has happened over time. For most organisations, programmes, and interventions this is very difficult. Particularly for charities.

Following up with the same people a few years later to see what happened to them is hard. For vulnerable people who often move around a lot, change mobile numbers, don’t have computers and don’t like being followed up much – it can be almost impossible. So many evaluations are in truth pretty heroic.

And even if you can, this can only tell you what happened to the people you worked with. But this is less than half the story. It can be very misleading. If you see a good result, it may be that this would have happened anyway – or at least some of it might have.

You ideally need to know what would have happened to these people or that place without your intervention. But that seems like an impossible task!

The net result is money being spent – in charities, in public services – with little clue as to their real impact. In times of limited public money that is unforgiveable.

That is why one of my proudest achievements in my time at NPC – along with help from colleagues at NPC and others elsewhere – is to have got what we call Data Labs set up in number of areas. They have helped to make what seems like an impossible task a little more possible.

What are Data Labs and how do they work?

Data Labs help charities improve their evaluation by making them more like Randomised Control Trials (RCTs)

RCTs are the ‘gold standard’ of evaluation. They are commonly used in medicine. For an RCT, you start with your ‘target group’ of people with an illness. Then randomly divide them into two groups – one who get the treatment, and one who don’t. It’s then easy to compare and see the effects of the treatment.

Randomised Control Trials (RCTs) have a lot of advantages but for charities there are two major downsides. One is that they are very expensive. And two, they may raise hard moral issues. You have to deny or delay help to someone who needs it to create a control group. That’s not something many charities are in a position to do.

Data Labs are designed to be a cheap and easy way to get somewhere near an RCT. I call them a ‘quasi-RCT’. Data Labs work by using government data.

Data like employment status, school achievement records, prison and probation records, and health records are all routinely collected by government. Used properly and can tell us an awful lot.

And this is one area where our centralised form of government becomes an advantage. That’s because a lot of data is collected and collated by the same central body, like the NHS or the probation service.

Most of this ‘administrative data’ must be kept anonymous and generally cannot be shared outside of government – but government can look and analyse this data for charities.

This type of summary data solves the problem of the control group. Because now you have data on a large population who haven’t received your help. It’s a natural control group you can compare to, without any cost or ethical dilemmas.

So now you can compare the people you worked with to all the rest to see if anything different happened. (The technical term for this is ‘propensity score matching’.) It’s not perfect. For example, you can’t take account of things like the fact that those who chose (or you chose) to come on your scheme might be the motivated ones. But it gets us a lot further than ever before.

Of course, for this to work for a charity, you need to have access to this government data. When we started, there was no to get it. But we lobbied government about the importance of sharing this type of data. They listened. In time, the Ministry of Justice (MoJ) set up the Justice Data Lab. And later, the Department for Work and Pensions (DWP) set up the Employment Data Lab.

These Data Labs are make the relevant data publicly available. Anyone can access them. It’s been a real achievement.

The Justice Data Lab has helped many charities and other organisations get a feel of how their programmes are going.

The Employment Data Lab is only just getting going. But we think it’s come at a vital time, when devolution means more employment programmes are being commissioned and run by local areas. This means that we have much variety going on, and so a greater need to understand what works best.

An example of how Data Labs help charities

Take Resurgo – an employment charity whose Spear programme helps young people not in education, employment and training in London, Bristol, Brighton, Bournemouth, and Leeds.

In 2022 they wanted to do an evaluation of people who had completed the programme between 2015 and 2019.

If they tried to follow up these young people themselves, how many do you think they would have tracked down? And for how many young people would they have obtained complete information about their employment status? And as mentioned above, even if they did that would only be half the story – the people they had worked with, not without a control group.

Instead, Resurgo’s Spear programme was the first programme evaluated through the DWP’s Employment Data Lab.

The DWP searched their databases to create a group of people who were ‘like’ those that Resurgo worked with. By doing this, the DWP created a ‘synthetic control group’. They followed both this ‘synthetic control group’ and the group of young people who took part in Resurgo’s Spear programme over time, seeing what type of work if any they were doing.

This allowed for an easy comparison between the two groups that could show the impact of Spear.

The evaluation showed that young people who had been through Spear

• Were employed for 7-13 more weeks longer over two years on average than young people who didn’t complete Spear.
• Were 7-14 more percentage points likely to be working two years after finishing Spear.

There was also evidence of sustained impact over time. But this clear evidence that Spear is a programme worth backing wouldn’t have been available without the Data Lab data to compare their outcomes to. And that data wouldn’t have been available without our efforts.

What’s next for Data Labs – more data, more sources

It is vital that we keep working and improving Data Labs.

They have got better over time as more data sets have been added.

For example, data from the Justice Data Lab can now tell us not just if someone reoffended or not, but whether any re-offence was more or less serious and if they found work – possibly even how much they progerssed in their job.

This sort of extra information makes it easier to build a ‘quasi-control group’ that is more like the group of people you worked with. For example, if you worked with people with poor mental health you may be able to build a control group who also experienced mental health problems.

This is powerful. It could be so much more powerful, but we need to keep up the pressure on government. We are grateful to our funders for making any of this work possible. But the truth is funding for this sort of work is very hard to secure.

And we need not just better Data Labs, but more Data Labs. We should have Data Labs in education, in health, in social care.

But the really vital thing is that we need decision-makers to start using the information that Data Labs give us. We need to lean hard on charities and funders (and government) to use these tools and learn from and act upon the results.

Resurgo, for example, took the plunge and were rewarded with a positive result – but charities need to anticipate that they might receive negative results and be ready to learn from them.

More than that, we need to have an ambitious research agenda to move beyond isolated studies to a wider programme of work exploring areas where the evidence-base is weak.

For example, take the role of skills in employment outcomes. We don’t yet know what is effective in helping young people not in education, employment or training back into employment or training. That’s not some niche charity problem. That’s a huge problem for our economy.

Similarly, the country spends over £6bn each year on prisons alone and still faces a prisons places crisis. If better data can make even a small improvement in government procurement for programmes to reduce re-offending rates, it will be worth its weight in gold.

Conclusion

There are reasons for hope.

There are not a lot of easy wins in trying to address things like re-offending and getting people into jobs effectively. This is one of them. The Ministry of Justice already recognise it.

With the right support and promotion from our new ministers, and continuing prodding from NPC, I believe Data Labs can play an important role in better government policy in the future.

---

With thanks to our supporters for making this work possible, including the Gatsby Charitable Foundation for supporting the Employment Data Labs.