How to evaluate the impact of care and support services

Octavia is a not-for-profit organisation which provides thousands of people in London with homes, support and care. It offers a wide range of support to isolated, older and vulnerable adults living in its homes and the communities it operates in. This guest blog by Neil McCarthy, Assistant Director of Care and Support at Octavia, shares details of NPC’s recent evaluation of Octavia’s outreach, befriending and activities service.

What is the best recipe for a successful impact assessment framework? Take some clear objectives and a common understanding of how your work impacts on people’s lives, mix thoroughly with clear definitions and concrete evidence, then add a good measure of forward planning.

In 2019, Octavia commissioned NPC to evaluate and develop a theory of change for our outreach, befriending and activities service. Fast forward two years and our first foray into theory of change has taught us a lot—from helping to determine our objectives, right through to helping us translate our data into action.

Octavia is a not-for-profit organisation providing thousands of people in central and west London with homes, support and care. The work of our outreach, befriending and activities service is an exemplar of how we empower isolated, older and vulnerable people to live well and connect with others. We regard this to be increasingly critical work, as the effects of loneliness and isolation are known to be emotionally damaging, eroding self-esteem and affecting physical health. You don’t need to look very far to see the impact they have had on individuals during the pandemic.

Learning from a theory of change

Going into the first theory of change workshop, in our minds our service objective was simple—to combat loneliness and social isolation in our local communities. Led by NPC’s facilitators, initial conversations with our frontline teams, managers and senior leaders helped us to dig deeper and go beyond this initial understanding, by reflecting on our intended outcomes and how we go about achieving them. For example, through creating connections and social activity, and supporting better access to health care and activities, our work empowers people by improving their confidence and resilience and helping them to address any issues they are experiencing.

Once you are clear about the purpose of your theory of change, it’s important to think about who you want to involve in the process and why. For us, involving frontline staff at this stage made a big difference. As our eyes and ears on the ground, they brought many useful insights to the discussion through their first-hand experiences with service users.

Measurements

With a theory of change now clearly defined, the challenge lay in the measurement. To obtain a full picture of our impact, we collected quantitative data through initial and follow-up surveys and qualitative data through one to one interviews. The quantitative survey drew on questions that existed in other tested surveys, such as the Short Warwick Edinburgh Mental Wellbeing Scale. The qualitative methods enhanced this data with rich insights that can only be gained from one to one discussions. A pioneering approach, this method of data collection is currently being promoted at a national level through the All-Party Parliamentary Group on Loneliness Inquiry. Supported by the British Red Cross, the data collected will be used to secure commitments from the government on tackling loneliness.

Choosing a set of questions for the surveys took time and there was much debate over how many questions should be included. Those related to self-esteem were reported as the most difficult to get across to service users by staff carrying out the surveying. Drawing on our experience, we recommend that other organisations use language to describe outcomes that service users can readily understand and relate to.

Through our evaluation with NPC, we now know that our service is achieving its intended outcomes and having a positive impact on the lives of some of the most isolated people in the communities we serve.

The research culminated in a report which illustrates the benefits these outcomes can yield in fostering connections, improving mental well-being, improving physical health and reducing loneliness amongst service users. It also highlights the effectiveness of group work in supporting people to feel more socially connected.

Taking on board recommendations

The research involved more than 500 people receiving over 2,500 hours of befriending from Octavia’s service. Octavia calculated the social return on investment (SROI) from the service as 1:6, so for every £1 spent on providing the service, an equivalent of £6 was achieved in terms of wider social value.

We are now taking on board NPC’s recommendations, including how we can better use technology to connect with service users, staff and volunteers, and how we can reduce digital exclusion. Moving forward, we will be offering a blended service of face to face, telephone and online contact to enable connectivity between individuals.

For us, this evaluation was never intended to be a stand-alone process, we want to make evaluation part of the way that we work. The rigour of analysing how an activity is expected to work, how you will know if it is working and how you can use this information to adapt your offer is an essential part of service delivery. We are now applying this discipline to other projects—harnessing the value of our theory of change work to inform our upcoming communities’ strategy and other care and support services.

For those looking to fully embrace impact measurement, starting with a discreet project is a great way to familiarise yourself with the process before applying it to work on a larger scale.

As an organisation with a strong social purpose, obtaining definitive proof of how we are achieving our aims is central to fulfilling our vision and purpose. Demonstrating impact is an essential tool for ensuring more is done better, for the benefit of society.