The value of assessing ‘value for money’

Is it helpful to assess whether charities provide value for money (VfM)?

In my experience, there are three broad responses to this question. For some, the obvious answer is yes as such assessments can support the aim of directing resources towards activities that create the most value and take resources away from those that create least value.

But for others the answer is no, not because they disagree with the aim of allocative efficiency but because they believe that the methods we have for defining and assessing what value is, and for whom, are not up to the task and / or can be manipulated to suit political ends.

The third response is to be unsure of making such a sweeping judgement, appreciating the points of view of both the yes and no camps.

This is the first of two blogs where I will provide my response to the question above, using NPC’s recent experience of conducting a VfM assessment of a £187m grant programme—which funded over 8,200 organisations—as my guide. This assessment was, as far as we are aware, the first of its kind and we hope the approach we applied can be further used and improved upon in the future.

What do we really mean by VfM?

Before I talk about that analysis, I should first be clear about what we mean by VfM. We each may have an intuitive understanding of VfM, but misunderstanding can occur because, annoyingly, economists use the term to mean several different things, some of which are not intuitive.

The National Audit Office has identified five ways of assessing VfM: economy, efficiency, effectiveness, equity, and cost-effectiveness. But a common and different approach to VfM, laid out by HM Treasury’s Green Book, is cost-benefit analysis (CBA).

The table below shows the differences between these six ways, using an example of a charity that provides support to clients with special needs to help them stay living in their home rather than in an institution. Three points are worth noting.

First, effectiveness and equity might seem out of place, because they are not directly concerned with money or economics but, arguably, they reflect the turf-encroaching behaviour of the economics profession. Secondly, the difference between efficiency and cost-effectiveness may appear subtle. Technically the former concerns the cost per output, while the latter is cost per outcome. However, when providing support to people, the difference between outputs and outcomes can sometimes be hard to distinguish. Thirdly, and most importantly, being clear on the type of VfM assessment required is important because for any given charity the results of each assessment may be different. A charity can provide VfM under one assessment, but not under another. Most notably, as any good Executive Director knows well, a charity can do well on economy by cutting costs, but doing so may undermine its ability to provide a good quality service and hence may score low on cost-effectiveness or cost-benefit analysis.

Our approach to VfM for the Coronavirus Community Support Fund

NPC’s recent analysis, in partnership with Ipsos UK and the Tavistock Institute of Human Relations, of the VfM of the Coronavirus Community Support Fund (CCSF) provides a real-life example of the difference between these different types of assessment. The funding for the CCSF came from the Department for Digital, Culture, Media and Sport (DCMS), and was distributed by The National Lottery Community Fund (TNLCF). The fund was a £187m grant programme that funded over 8,200 organisations in the voluntary, community and social enterprise (VCSE) sector between 2020 and 2021, with the dual objectives of:

• increasing community support to people disproportionately affected by the Covid-19 crisis, through the work of civil society organisations.
• reducing temporary closures of essential VCSE organisations.

Ipsos UK was appointed to deliver a process, impact and VfM evaluation of the CCSF, as well as activities aimed at facilitating grant-holder learning. Commissioned by TNLCF, NPC led the VfM analysis, which drew on data collected through the process and impact evaluation strands—namely data collected from VCSE organisations at the grant application stage, an online survey issued to all grant-holders following the end of their grant, and a programme of research (interviews and case studies) conducted with organisations who received funding.

This analysis covered three of the ways of assessing VfM mentioned above: economy, efficiency, and cost-benefit analysis (assessment of the effectiveness was part of the impact evaluation led by Ipsos UK). In summary, we cautiously concluded that the CCSF provided VfM on the basis of economy and cost-benefit analysis—with a benefit-cost ratio between £1.86 and £1.38—but we were more confident that it provided VfM on the basis of efficiency as we had more robust data to draw on. The full results can be found here.

More importantly, this analysis was, as far as we are aware, the first attempt to conduct such a broad analysis of VfM of a large—in terms of both the value and number of grants—portfolio of grants, that covered a variety of different services and outcomes. The pandemic highlighted a lack of data and evidence on the impact of civil society as a whole (beyond individual charities). There is not currently a clear understanding of what the sector looks like at any one time, the value it brings from a public services perspective, and its role in preventing the need for more acute services (or conversely, the value lost when an organisation closes or a group of organisations close). The impact and VfM strands of this evaluation were designed to help address this gap in evidence on the value of the work of civil society.

Judging economy was, in part, subjective. The CCSF was launched in the unique context of the Covid-19 pandemic, in part to prevent significant disruption to the charity sector, and there was no objective benchmark against which to determine whether more money was spent than needed. We judged the programme was economical because: the evaluation concluded the programme was effective; the cost of £187m to the government was offset by savings of £7m from un-furloughing staff and preventing furloughs; and most grants were small and there was no evidence of any excess expenditure.

Assessing efficiency was relatively straightforward as—as can be seen from the definition in the table above—we only had to know the value of the inputs (for CCSF, the amount of spending on the activities and the outputs the grants supported) and the number of outputs (for CCSF, the number of beneficiaries supported and number of volunteers and staff recruited and retained). Grant-holders had these figures at hand. From these we calculated spending by beneficiary and activity (e.g. mean spending of £165 / median £38 for each person who was provided material support and mean of £420 / median of £200 for each person who was provided personal care) and benchmarked these against comparable figures from a non-covid period.

Estimating the cost-benefit ratio of the CCSF grants was harder as we had to find out both the outcomes for beneficiaries of the 8,200 grant-holders as well as estimate the value of those outcomes to the public sector and to the beneficiaries themselves. This is difficult enough to do well with a single programme, during stable times, and even harder for over 8,200 grant funded programmes during a volatile period.

Our solution was to develop a formula that we could apply at scale and also to collect from the grant-holders an estimate of the number of their beneficiaries that benefitted from one or more of 13 standardized outcomes, such as: ‘short-term needs were met (e.g. financial, food, clothing, shelter)’, ‘physical health was better’, ‘people were better protected from harm, violence, or abuse’ and ‘people felt less lonely’. We estimated the value of these outcomes based on evidence of comparable programmes and using feedback provided by the grant-holders. The full methodology and sources of evidence are described in the report.

The formula is:

Value = Scale (i.e., no. of beneficiaries who experience one or more of 13 ‘standardized’ outcomes as reported by grant-holders) X Unit value (in monetary terms) of the standardized outcome X Change per beneficiary in units of the standardized outcome (expressed as a %), the estimate of which was informed by research literature X Duration, the period over which outcome is experienced

This approach inevitably required a trade-off between accuracy and parsimony, but in the end we believe the results provide a reasonable estimate of the value of the services provided, despite the generalisations made. Using this approach we estimated the net benefits of CCSF funded programmes were valued at £402m. We estimated that the costs of resources used to create these benefits were between £216m and £291m (as the £187m of CCSF funds were supplemented by other sources).

Assessing the VfM of the CCSF was a significant challenge and we hope the approach we developed here for estimating benefits is used and improved upon in the future. But we are not blind to the challenges in assessing VfM at a small scale as well as this large scale. I will discuss these small and large scale challenges, and the implications for our starting question above, ‘is it helpful to assess whether charities provide value for money?’, in a follow-up blog in few weeks’ time.