Measuring the multiplicity of impact

As part of our three-year learning journey with the £40m Youth Investment Fund, we’ve been tasked with assessing the potential economic value of the youth programmes it’s funding. In preparation, I’ve been reading previous economic evaluations of youth programmes.

It’s got me thinking about what we can all learn about value and impact from the process of economic analysis—whether we’re economists or not.

A striking observation, when you’re reading economic evaluations of similar programmes, is the variation in the methodology evaluators use. This isn’t just the technical aspects of how value is estimated (e.g., discount rates, deadweight and attribution—which can have very substantial implications for results). At a more intuitive level, the variation occurs because there are so many things you could value. After all, social programmes have wide reaching and complex impacts on our lives. The result is that different economic evaluations often include different sorts of impacts in their estimations of how much ‘value’ programmes generates, even when the programmes have lots in common.

Let me illustrate with the example of a youth club.

Youth work creates intrinsic value for participants—feeling better about yourself, having more self-confidence, getting on better with people. It’s possible to estimate what these improvements to subjective wellbeing are ‘worth’ to people experiencing them. But these attributes are not just intrinsically valuable, they might have long lasting impacting on the lives of young people and their communities.

What if young people grow up to be happier and more satisfied with life as a result? What if these skills mean young people do better in school, grow up to earn more money in the future, or be more active participants in their community? As well as changes in how people feel, what about changes in how they behave? If access to a youth club means a young person is less likely to become involved in anti-social behaviour or risk their health, these offer immediate, and potentially long-term savings to the public purse, as well as a happier life.

The diagram below aims to capture some of the complex interactions in how value is generated by youth programmes, based on the literature we’ve reviewed so far.

In short—social programmes create a complex web of impact. So when we come to estimate the value of a programme (or rather, the value of its impact), we have a lot to choose from.

This is important because unless evaluators are all adopting exactly the same approach, there’s a risk of inconsistency when comparing benefit-cost ratios for different programmes—which might incorporate different factors in the ‘benefits’ side of the equation. There are ways to try to get around this—for example, the SROI methodology aims to bring some consistency to evaluation approaches, and HACT focuses on valuing wellbeing. But even among the dozen papers I’ve reviewed on youth clubs, they take a variety of different approaches to build an estimate of the value of youth work.

A useful step would be for all economic analyses to clearly articulate who receives value and how they receive it, as well as how much. This sort of clarity is needed if you want to compare the £X benefit for £Y cost of different programmes. Without it £X could include a range of different benefits to a range of different people (and distribution, as well as magnitude, matters).

Despite the challenges of measuring and comparing all this impact I don’t want to sound pessimistic. It’s an important reminder of the fact that only when we get serious about measuring something, do we begin to understand the reach it is having.

For more insights like this and to never miss a blog, sign up to our newsletter.