Free evaluation support for criminal justice charities

Since the beginning of September, NPC and Clinks have provided free evaluation consultancy to criminal justice charities. It’s been a great opportunity to work with a range of organisations and learn about the challenges they face.

The conversations have been wide-ranging. We’ve discussed practical issues such as access to official data, designing questionnaires, reporting formats and getting to grips with the MoJ’s Justice Data Lab, but it’s also been valuable to think about underlying questions along the way.

First of all, what or who you are collecting evidence for? There’s nearly always a desire to impress or attract a funder. We understand why this is important, but don’t think it’s the best place to start: rather charities that are focussed on learning tend to do better evaluation. We also believe that funders should be more impressed by a charity that can show how it is reflecting and  improving than one that only uses evaluation to make impact claims.

The next question is often overlooked: what are you evaluating? Charities often talk in rather general terms about the need to “evidence our impact”, but most organisations do a range of different things – which makes this a very complex question. It’s easier and makes more sense to look at evidence needs on a project-by-project basis.

The next stage is to articulate what does the service actually do? This means thinking through the intended outcomes (both short and long-term), and how you are going to achieve them. We call this process the theory of change—also known as logic modelling or outcomes mapping—and it’s the commonly accepted starting point for all good evaluation. Developing a good theory of change takes a bit of practice and a number of organisations have used our help on this.

Next, it’s helpful to think about which aspects of the service you most need evidence about? For some projects there may already be good enough evidence on impact from external sources (eg, reducing crime by tackling substance misuse), in which case the priority is to assess the quality of that service (eg, the extent to which you engage and retain service users and the feedback they give you). Alternatively, if you are doing something new or untested then assessing the impact on users is vital. We suggest raising this question with your funders: what do they most want to know about? Is it how well you operate or whether your ideas are any good in the first place? When talking to funders, charities also need to understand the standards of evidence they expect (we’ve written some guidance on this here).

It’s only at this point that charities should start thinking about which methodologies to use (eg, questionnaires, focus groups, outcomes stars, and randomised control trials), because this choice needs to be determined by what we need to know, as well as what is practical.

If you are a charity working in criminal justice and think we might be able to help—either around specific methodologies or your overall evaluation strategy—then please get in touch with me at james.noble@thinkNPC.org. We can offer up to a day of support, which is available until March next year.