Choosing the right evidence level for your impact evaluation
28 April 2016
Many organisations come to my colleagues and I with the question: ‘How do I get to Level 3?’. Level 3 is the new pot of gold at the end of the rainbow. It’s the third of five levels used to define the robustness of evidence for the impact of social interventions. Variations of this level are used by organisations including Project Oracle, Dartington Social Research Unit, and Nesta.
Level 3 involves demonstrating causality using a control or comparison group. In other words, using some sort of comparison with a similar group of people who haven’t experienced the service or help that you provide. This can show that you’ve achieved a positive change for your beneficiaries. I think many charities and social enterprises can aim for this.
For example, we worked recently with a tiny Nottingham-based charity called Imara—which supports victims of child sexual abuse and their families—on the evaluation of its service. This involved looking at relevant police data, which showed cases where Imara was involved were more likely to get to court. This suggests that the support Imara provides helps children through the trauma of giving evidence, thus strengthening the case. Police staff also reported that by translating the legal process for the family, Imara saved them time, so they could concentrate on getting the case to court.
Choosing right level of evidence for your organisation
It is great to see ambition when it comes to trying to understand your charity’s impact, but some of the organisations asking us about Level 3 are getting ahead of themselves. A common issue is that charities attempt it too soon in their organisational journey; it’s unlikely that such an evaluation can be carried out when a charity is barely two months old.
This means the ‘magic’ evidence level is sometimes being applied out of context (something charities, funders and commissioners are all guilty of), because it doesn’t apply to every stage of organisational development.
And the important initial question—‘will achieving Level 3 be useful to me and my organisation?’—can also be missed. As a result, more achievable and useful evaluation options end up being overlooked.
Getting the right research question
One of the first tasks on your evaluation journey is working out what your research question should be. With a brand new organisation, for example, the research question might be ‘are my services reaching the right people?’. Whatever data is collected needs to match this question.
Collecting quantitative data on outcomes for service users of a brand new intervention is unlikely to be worth doing until you’ve dealt with any teething issues (although baseline data may be useful if you won’t have any ‘new’ service users to conduct research with further down the line). So in this case, qualitative feedback from staff and service users is likely to be most useful.
The research question changes as an organisation grows and develops—and as the needs of your beneficiaries, stakeholders and funders change over time.
Ensuring potential for learning
I always encourage organisations to focus on what they plan to do with the data that’s collected and how they will learn from it. Even if Level 3 data shows evidence of success, it needs to show something about why it was successful to help inform future work.
Perhaps we need learning standards to complement evidence levels, assessing the extent to which an organisation is making use of the data it collects. In any case, charities must ensure they build learning opportunities into the process of collecting and analysing evidence of their impact.
So, before getting carried away with Level 3 plans, charities should: be sure to make wise use of precious resources; maximise the potential to learn; and use an evidence standard appropriate to the organisation and its development.
Take it one step at a time—and make sure you can walk before you try to run.
If you’re interested in exploring comparative evaluation methods, sign up to our upcoming seminar.
A version of this blog was originally posted on NSPCC’s Impact and evidence insights hub.