Developing a theory of change is transforming our approach to impact measurement

At St John, we have developed an organisation-wide theory of change to clarify the impact we want to have in society, and how we plan to achieve it.

This process has not only helped clarify our 150-year-old charity’s purpose and role in society, but also helped guide us on what we need and (just as importantly) don’t need to measure. It’s helped make what initially felt like an impossible task to measure our impact, very possible.

St John is a huge organisation with a powerful mission to save more lives through first aid. We do this by training the next generation of life savers through our youth programmes, training volunteers who share first aid skills in communities and providing first aid at events and in response to local and national health emergencies. This presents a multitude of potential ways to measure impact.

Tackling big assumptions

At St John, we know that emergencies happen every day and lives are lost because people didn’t receive the first aid they needed, despite there being someone there on the scene.

Most health emergencies happen at home. Therefore, St John’s purpose is to put the power of first aid into everybody’s hands, so we’re all closer to help in an emergency.

One way we do this is through the provision of free community education sessions in which our volunteers empower others by sharing life-saving first aid skills with communities. The basic idea being:

Except, we knew that this was making some pretty big assumptions. Would someone always help if they’d been shown what to do? We knew that the reality wasn’t that simple. Also, how could we possibly measure impact?

Although we often hear incredible stories from people we’ve trained, who’ve gone on to save lives using the first aid skills we’ve taught them, unless someone gets back in touch, we wouldn’t know.

Following up with everyone we’ve trained at some time point after their training isn’t realistic. If we couldn’t measure how many lives had been saved, could we even have this as our goal?

A better approach to impact measurement

By creating a more rigorous theory of change, identifying the logic and assumptions it contained, and collating the evidence underpinning it we started to see a way forward.

Working backwards, we knew that plenty of clinical research already exists to confirm first aid’s role in increasing survival and improving recovery (we didn’t need to prove this).

Research tells us:

• Most health incidents happen in the home — a large majority of which are witnessed by a bystander.
• A family member, loved one or bystander’s intent to perform a behaviour, is a strong predictor of how they will behave (given the brief nature of our intervention means we had no practical way of measuring actual behaviour, so intent acts as a proxy measure).

We then did a rapid realist review which really helped us understand how we can increase intent to provide first aid by influencing beliefs that exist within communities. Put together, this gave credibility to our approach. It also suggested what we need to focus on measuring the factors that influence intent, and what we could reasonably measure (contained in the blue box), namely:

• First aid confidence / self-efficacy
• Understanding of and belief in the difference first aid can make
• Sense of responsibility to help in a health emergency.

We’re now starting to develop ways of measuring all of the factors that influence intent to provide first aid – not just skills and confidence, which we hope will lead to insights that can improve the work we do in communities across the country.