Introducing a joint evaluation programme for the youth sector

NPC and the Centre for Youth Impact are leading the learning and impact strand of the £40m Youth Investment Fund (YIF), a joint programme supported by government funding from DCMS and National Lottery funding from Big Lottery Fund.

YIF is funding around 86 open-access youth providers for three years (open-access meaning services that don’t require a referral). We are working them to design an evaluation approach that will capture the value of their work, and support learning and improvement. We’re calling this work the YIF Learning Project. This blog outlines five ways we are confronting the challenges of evaluation in open-access settings.

Moving away from blanket outcomes measurement

Measuring and understanding outcomes is a central focus of our work but we are explicitly moving away from blanket measurement—ie, trying to capture every outcome for every young person for every organisation. Instead, we are focusing on high-quality targeted measurement with a sub-sample of the YIF cohort.

This shift responds to many of the challenges in open-access settings including fleeting or irregular engagement, defining generalised outcomes for individualised provision and the long-term nature of many impacts.

As well as this targeted approach to outcomes, YIF Learning will place increased emphasis on the experience of young people and the quality of the provision they receive. And crucially, we are aiming to link the data on outputs, user feedback and quality to outcome data. That way, we can understand not only whether the provision is having an impact on young people, but why.

Focusing on user experience

Young people often feel overburdened with sometimes obscure-seeming surveys. In response, our YIF Learning approach asks young people questions on the elements of delivery that are most relevant and meaningful to them—namely their experience of services (eg, feelings of safety, respect and positive challenge).

We are working with Keystone accountability to develop a set of standardised questions to collect regular, light touch feedback—perhaps 3-5 questions once a month. And we are working with providers to process and act on this feedback, and tell young people what has changed as a result.

Improving as well as proving

Practitioners often report a sense of dislocation between a focus on impact and their everyday work. Part of our commitment to ‘going with the grain’ of provision is a focus on the quality of youth work practice—focusing on those elements of provision that are most relevant and meaningful to youth workers themselves.

So the YIF Learning Project is drawing on an established quality improvement framework from the US—the Youth Program Quality Assessment—which relies on youth worker peer observation. This framework is not a critique of existing quality assessment frameworks but a complement to them, focusing on relationships, environment and ongoing practice improvement.

Understanding young people’s journey through services

Most providers collect detailed attendance data. But many tell us that they use this for monitoring overall service demand rather than truly understanding the way that individuals engage with their services.

By using existing data and trialing new digital methods such as Yoti we aim to build a much more nuanced picture of what young people do with their feet ie, how often they attend, for how long, and how they move through provision. We can use this as a proxy for their levels of engagement and ‘exposure’ to interactions.

Pushing on with shared measurement

Arguably the greatest opportunity presented by the YIF is the potential to collect shared data across 86 grantees for three years. This offers a rare (probably unique) opportunity to build an evidence base across a huge diversity of open-access provision (detached/building-based; structured/unstructured; universal/targeted). And, by comparing the results across different types of provision, we will be able to really understand the strengths and weaknesses of different services.

This is a shortened version of the blog Matthew recently posted on the Centre for Youth Impact website.