Two weeks back, the new UK Justice Secretary, Kenneth Clarke, made a bit of a stir by pointing out the fact that prison doesn’t always work. That in fact, ‘too often prison has proved a costly and ineffectual approach that fails to turn criminals into law-abiding citizens.’

This doesn’t seem that controversial. The shortcomings of the prison system are pretty well-documented: the massive overcrowding; the fact that just under half of all prisoners reoffend within a year of release; the disproportionately negative impact of short sentences; and the cost—more than sending a child to Eton. The key question is what you actually do about it, particularly in a time of massive public sector cuts.

Mr. Clarke’s vision for criminal justice reform references a lot of the mood music I’ve come to expect from the new coalition government—effectiveness, voluntary sector involvement, cost efficiency, the Big Society. All of which I find vaguely reassuring, particularly given the wealth of useful expertise and experience that exists in the charity sector—and which are explored in NPC reports on prison and on youth offending.

Where I tend to feel a bit more apprehensive is on the question of exactly how it is going to work in practice—particularly the question of ‘payment by results’, where charities are to be given clear financial incentives to keep offenders away from crime. Working out an individual organization’s impact on reoffending rates is massively complicated, time-consuming, and imprecise. Attribution is a nightmare, with ex-offenders often receiving help from multiple different organizations, both in prison and on release. Did someone stop offending because of a literacy course, housing advice, drugs treatment or simply because they get a bit too old?

Cash flow is another issue. How are organizations supposed to operate if they have to wait two years—to see whether their clients reoffend—before they get paid? Where do they access the capital and how do they manage the risk involved? And also who is going to capture these results? One of the reasons that charities struggle to demonstrate their impact in this sector is because they can’t get access to government data on reoffending and track their former clients as they move through the prison estate and on into the community. Will this information now become available?

It’s tempting to judge success by proxy measures—such as jobs or housing—that are easy to measure. But what about the areas like family relationships and social skills that aren’t so easy to pin down, but which are also massively implicated with reduced reoffending? And where’s the incentive to help the most prolific and persistent offenders, who are least likely to produce a pay-out?

As you can probably tell, I have a few questions about how this ‘rehabilitation revolution’ will translate into practical, clear policy. Not just because I’m naturally a bit critical, but also because of its importance. The potential pay-off could be so high—making our communities safer, preventing people from falling into a downward cycle of crime and deprivation, and cutting prison costs. Mr. Clarke has taken a brave step in recognizing the criminal justice system’s current limitations, but the tricky bit is converting this talk of effectiveness and voluntary sector involvement into reality and doing it well.

Luckily there are a whole host of charities that are grappling with the issues of effectiveness and impact in the criminal justice sector—I just hope that Mr. Clarke has the time and the inclination to listen to them.