Evaluating and learning during Covid-19

The Building Connections Fund was launched in December 2018 as the first ever government fund dedicated to reducing loneliness in England, with NPC as an evaluation partner. Over 120 organisations have been awarded grants to work with people from all age groups and backgrounds, from young care leavers to the elderly, LGBT+ and those with disabilities. Typical services included face to face befriending, mentoring or peer to peer support, advice and signposting, providing safe spaces in the community and arts leisure activities such as music, gardening and painting. In this blog, Elizabeth Parker shares how our evaluation approach is adapting to Covid-19.

What was our planned evaluation approach?

Before Covid-19, we were working with a really committed group of grant-holders to capture evidence and understand how the fund tackled loneliness.  We were collecting longitudinal quantitative data in the form of surveys; and these were going to be supported by qualitative interviews with service users and staff. The research was underpinned by the theory of change that provided a framework for linking the activities to the intended outcomes and longer-term impact.

Covid-19: What now?

When the lockdown hit, our first thought was ‘what now?’. We wanted to make sure we could use the valuable data we’d collected, but it was also clear that it no longer made sense to continue as previously planned. The dramatic lifestyle changes imposed by social distancing meant survey responses about loneliness and well-being could no longer be fairly compared to any follow ups after March 2020. Moreover, services themselves were changing dramatically. The virus has forced a major shift to digital and telephone services as grant holders adapt to keep pace with an ever-shifting context and changing needs.

We’re taking a two-pronged approach to these challenges:

• Firstly, we want to make best use of the data we collected pre-Covid-19. We’re analysing it and will publish as a stand-alone research report. Watch this space!
• Secondly, thanks to the flexibility and forward thinking of the funders (DCMS, The National Lottery Community Fund and Co-op Foundation) we are undertaking a developmental evaluation to learn from the current situation and capture emerging findings.

What is developmental evaluation?

Developmental evaluations are well suited to complex, dynamic environments like the one we find ourselves in, where projects are adapting quickly to meet changing needs. We hope this approach will allow us to track, record, and provide real-time feedback to grant-holders about emerging challenges and urgent needs so we can inform decision making.

We are facing an unprecedented situation, which means we have no existing studies or research to show us the best path to take. We hope that by adopting a transparent and collaborative approach we can help support grant-holders and others in the same boat to share useful insights as quickly as possible.

There are no pre-defined tools or methods for developmental evaluations. By their nature they must be adaptive to suit the context. Rather, developmental evaluation is a way of approaching the challenge of evaluating social innovation through guiding principles.

Building off the work of Michael Patton and others, the core principles underpinning our approach are:

• Supporting the needs of grant holders in real time (not creating an additional burden at a difficult time)
• Facilitating wider peer-to-peer learning for grant holders
• Rapid dissemination of findings (e.g. more appropriate to be live collaborative guidance notes, blogs, short briefing notes or recordings of meetings; less likely to be long reports)

Developmental Evaluations keep the end objectives of the Fund in sight, and we are planning our data collection activities and analysis on a month by month basis. So far, our approach has included online surveys, interviews and collaborative learning sessions. Looking ahead we also hope to work closely with a small group of grant holders in a focused and supportive way to understand what they are doing, how they have adapted and what they have been working on to reduce or prevent loneliness.

Raising the profile of loneliness

At NPC we think that tackling loneliness and developing the evidence base has never been more important. In response to Covid-19 there is increasing government funding to address issues of loneliness and social isolation and we hope that our research can inform this, and other related initiatives that follow.

The situation is a first for all of us and we are keen to adopt a collaborative and open approach. To stay updated with our research, drop me a line at elizabeth.parker@thinknpc.org.