Why assessing impact remains crucial

Towards Balanced Evaluation

There is an awful lot going on at the moment. Internationally the war in Ukraine proves that peace is precarious and the terrible floods in Pakistan remind us of the ever more urgent climate crisis. The Black Lives Matter movement, and the Women and schoolgirls’ protest in Iran – both sparked by brutal murders – show how far we have to go to achieve fair and inclusive societies.

Here in Britain, we have a new King, and a new Prime Minister – who is reversing everything about debt and deficits that her party seemed wedded to. We’re entering a severe cost of living crisis in which, for the first time in generations, people now fear the coming winter.

So many people – in Britain and all over the world – are not living as good lives as they could.

So, with all this happening, why go on about impact and its assessment and use?

Surely the answer is clear – our resources need to be used in ways that enable us to do our very best to make lives better. This we can surely all agree upon.

And that is the impact agenda.

To use resources well we need to apply all our knowledge – from all sources.

There are two strands to this – top-down and bottom-up.

Top-down is about professional expertise, analysis, data, and research. Bottom-up is about lived experience, community knowledge, and user involvement.

Both must be used to help us achieve impact, to be able to assess that impact – the difference we make – and to understand where that impact came from. And both need to be inclusive of a diversity of voices, backgrounds, and perspectives.

Certainly, we will always need hard impact data to convince hard-nosed and hard-pressed people in the Treasury or the local council to fund charities. Budgets are tight and opportunity costs are real not theoretical. But funders and charities need to do this too where they can in any case.

It’s not easy. If you’re a business running 35 product lines, you regularly check if you would make more money with just 20. You have easy metrics and a simple bottom line (even if ESG and the SDGs are thankfully making it a bit harder!). But it’s very hard in the non-profit sector as it’s not tradeable, there’s no market price and no simple metrics to decide, for example, between funding frontline services or research or advocacy.

Pretending there is a magic metric to decide between different causes is as dangerous as it is impossible – whatever the effective altruism crew may say. That is why charities will never be like the private sector however much some from business, the city and private equity backgrounds say they should be.

But that’s not a reason to abandon the impact agenda and impact measurement.

We have always believed that the strongest argument for understanding impact is to learn and improve, to make better decisions about the use of our resources. It’s not to judge and attempt to ‘prove’ our impact.

All this is very much wanted by charities – and I do not think that is just because charities think funders want to see it.

By a long way, the most popular resources on our website are our guides to theory of change and impact advice, as charities and funders try to get a grip on what they are trying to do and where their priorities should lie.

We know we need to shift power in our sector – to be guided and led by the people we seek to serve. This is especially true for funders who are often a long way away from the frontline and for whom the power dynamics really do kick in and distort decisions.

Some claim this means we have to drop everything we have known as it is all top-down, including the impact agenda and ways of assessing impact.

I think that is fundamentally wrong and even dangerous.

Yes, things do need to change – and they are changing. But to truly shift power in our sector we need to blend the top-down with the bottom-up – to develop Balanced Evaluation – not destroy the very concept of assessing impact.

Right now, we’re in danger of the two strands splitting.

As we get stuck in opposing camps, we caricature, dismiss, and hurl abuse. That we must and will resist. We all lose if we don’t – and the people we want to help will lose out the most.

Assessing impact remains crucial for shifting power. Without the accountability that comes from reliable impact assessment, activity will be dictated by the most powerful voice rather than what actually helps.

Today I am going to explain what I mean by Balanced Evaluation and where we need to take this impact agenda.

 

Let me start with the first strand: the top-down.

NPC was set up twenty years ago to bring rigour and analysis to our sector. We wanted to know how funders and philanthropists can do the most good. Who are the charities creating great outcomes effectively? What works? How can charities and funders set a strategy most likely to achieve their mission? How can we measure any of this?

This has got us a long way. After 20 years, the concept of ‘Impact’ is now very much embedded in the sector (even if there is sometimes resistance to the word as it does not gel with everyone). There is now, I would argue, more thinking of what to do to achieve your aims and how to assess that and think these things through, and there is less reliance on just heart and anecdote – crucial though these are to our sector.

Together with others, we’ve brought some measure of accountability to the sector – to donors, to grant makers and philanthropists, to charities and non-profits, and to the people we are trying to help – something otherwise rather absent.

Theory of Change, a powerful tool to help think through all these things, that we played a part in popularising, is very much about this. Social investment, outcome-based payments, and impact investing all link to this in some way.

Of course, there have always been people uncomfortable with the whole impact agenda as not understanding the sector and its motivations. And a more justified fear that we begin to treasure what we can easily measure, rather than what actually matters – although advances in measuring things like Wellbeing are making that a less profound critique.

There have been more practical issues about the capacity of the sector to do any of this at the same time as doing their day job of helping people, not least given the financial models that most operate under and the fact that the public still tend to donate based on emotion and stories.

There are also more technical issues about appropriate metrics and the ability of the data we can easily collect to tell us what we actually contributed compared to what would have happened anyway.

In our 20 years, we have been responding to all these issues. We are 100% clear that we do need some way of deciding the best way to allocate resources, whilst avoiding the trap of doing what looks rigorous but is in fact too inaccurate and burdensome to be of any practical use.

We have for instance virtually dropped doing simple Social Return on Investment (SROI) calculations, at best trying to guide those who ask for them to do break-even analysis instead. The thought sequence that goes into SROIs should be on everyone’s mind, but a ‘one pound in, fifteen back’ isn’t really getting us anywhere. The numbers are almost always flaky and based on too many assumptions. No wonder we now see what some academics have called SROI inflation. And no wonder that other organisations like Social Value UK are also downplaying these approaches.

And when we have done what looks like more conventional Cost Benefit Analysis or SROI work for larger organisations – for instance for the National Lottery Community Fund and the £200m of DCMS money they dished out via their Coronavirus Community Support Fund – we are very careful to be clear about the limitations of the work and to do sensitivity analysis around the assumptions that must be made.

We are much happier pushing for really clear and accessible data sets that allow for good analysis, and for charities to be able to make use of. This is why we want the government to be more open with its data. Government administrative data is a fantastic way to follow people we’re helping over time and compare them with credible control groups in a cheap and easy to use way. That is why we’ve worked so hard to get Data Labs going.

The award winning Justice Data Lab, now run by the MoJ, that helps work out whether programmes by charities working with prisoners really did reduce reoffending, is the jewel in our crown so far. Soon (we hope) there will also be an Employment Data Lab run by the DWP that does the same for helping people into work.

We have also appealed to smaller charities not to overdo the data collection but to focus on five types of useful data that they can collect. Indeed, we have published a blog on why charities should collect less impact data.

We have also emphasised the importance of qualitative data – not just quantitative. Such data helps us understand what caused the outcome in a way that purely quantitative data rarely can.

We have also argued against aiming to climb some kind of evaluation ‘ladder’ towards the pinnacle of a randomized control trial (RCT) which is often neither possible nor proportionate – or even that helpful when it comes to social policy where an intervention that works according to an RCT in one area at one time may not work in another place at another time. Better to do the level of evaluation you are at really well, than try to push up the ladder and do it badly.

There have also been some important challenges to our thinking on the impact agenda in recent years. What for instance does the increased use of a system change lens – a core part of our work these days – mean for it? Does the traditional version of the impact agenda end up being too much about how to do the ‘help’ better, rather than how to change the system?

Our thinking about how to do theories of change in a system change setting, and how to assess impact in that context, are starting to get a handle on these issues. We want to see much more sharing of data and insights amongst funders especially – our current work on Open Philanthropy is all about that.

So there is a lot of change going on in this top-down world.

 

But is this enough?

The answer is a resounding no. We need to push on the bottom-up strand.

We need to embrace and involve lived experience, community knowledge, and user voice into how we achieve and assess impact. To understand what works, we need the input and expertise of those we are trying to help.

I don’t want to push the analogy too hard, but it is interesting to look at what happened to the New Public Management agenda that once swept the public sector about how to get services that delivered better for citizens. In some ways it was analogous to the early impact agenda, although it was even more dirigiste with targets and league tables and other features like outsourcing, choice, and contestability.

There are many things to be said about NPM in its purest form. Some of the perverse incentives and behaviours it encouraged outweighed any good it did. But in short, it was far too top down.

Most crucially, despite being all about improving services, NPM did not listen enough to the people who actually used the services – choice mechanisms only partly addressed this. In the New Labour period you can see this realisation happening as the fairly simple NPM-style public service reform of the early Blair era gave way to a more subtle if less clear approach by the Brown premiership days.

Was the same going on amongst charities and funders? Did we get too carried away with metrics, impact, evaluation?

When we recently asked stakeholders how they thought the sector was changing, one respondent said, “stories and narratives based in lived experience” are what the world wants now. That, the respondent suggested, meant NPC’s time is up – along with the rest of the impact agenda.

A provocative thought that deserves consideration.

There is a lot that is right in the critique of old-style impact assessment.

As Elliot Stern in a paper for the then Big Lottery Fund, Bond, Comic Relief, and the Department for International Development said in 2015, too often:

The language of ‘impacts’ implies a passive ‘voice’ for beneficiaries or so-called target populations: outsiders administer treatments to those who face problems which are themselves often defined by outsiders.

Our own Tris Lumley recently wrote:

[Conventional] tools get captured by the existing power dynamics; they flow with the contours of money, access, and influence. They don’t break through existing power structures, so they don’t really shift the power to where it could be.

Some see this more inclusive agenda as being the antithesis of the impact agenda. Stop measuring stuff – or even trying to – just let the community or the users decide.

Such a thought can be seen emerging in the eager adoption by some funders of trust-based philanthropy. While a lot of the trust-based philanthropy movement is welcome – we have advocated for unrestricted funding for ages, and it took off (a bit) during the Covid crisis – the downplaying of impact and impact assessment is concerning.

So we must be careful here.

That is why Balanced Evaluation is about being inclusive of users and communities, but blending these methods with top-down techniques, not throwing out top-down approaches all together.

Balanced Evaluation is fundamental to our ability to effectively use our resources and the assets and knowledge of the people we want to help.

Because there are two things we want to achieve.

The first we can call ‘productive efficiency’ – can we do what we are doing better? That is very, very, important. Imagine if we could get funders, and charities five or ten percent better. An awful lot more good would be done. Traditional impact measurement techniques are a big help with this.

But the other we can call ‘allocative efficiency’ – are we spending our time hitting the right thing, the things that really do make people’s lives better? There is no point in plotting the fastest route to a destination that nobody actually wants to go to!

For instance, we may be funding a charity to plant some wild meadows in the local park. And we find it works and some people sort of liked it. But is that what the community wanted of that park? Are we measuring the wrong thing and was the project misdirected in the first place?

Or dealing with homelessness. What matters most to the clients themselves? Fast service? Help with benefits? Or decent Wi-Fi? Did we ask them?

To do that we need to listen to them, find out what they think, get them involved so the programme works, and its effects have a chance of being sustained. That goes for communities and users of all types.

Lived experience allows better design and evaluation and therefore better allocation of resources and more impact.

In an NPC event on the role of diversity, equity, and inclusion in evaluation, it was argued that:

It’s important to ‘work with’, rather than ‘do things for’, people. Organisations should make their communities part of the decision-making process, rather than paternalistically handing down plans from on high.

The conclusion was that:

Evaluation will be more effective if it has buy-in from the people involved in their services. …This shift in accountability is not only respectful of your communities, it is likely to make programmes more effective, impactful, and relevant.

It’s also why Tris Lumley concluded:

 What we need to do is put the right tools in the right hands; to create structures with the people they’re supposed to benefit, so that they can be truly empowering and liberating.

That’s why we’ve incorporated user led approaches into our fundamental approach, encouraging our clients and the sector to think harder about these issues, bringing these ideas into all of our work. We have a major program in this area and hope to produce useful guides and ideas for the sector.

There are also big issues of diversity here. Are all voices – minoritized voices especially – being heard, being involved, being put at the core of your work? We are actively involved in the Equitable Evaluation Collective to help make progress here.

As Sanjukta Moorthy, a Planning, Measurement, Evaluation and Learning consultant, said in a recent NPC blog on shifting power in our sector:

When designing your M&E strategy, consider where your community’s voice is present. Who has built your strategy, and what biases are those groups bringing to it? … What assumptions are you making about the work and its success? Opening up this space when designing your M&E strategy for a programme…ensures that your work is based on peoples’ realities and that your solutions are more likely to be successful.

 

These two agendas, top-down and bottom-up, needn’t be in conflict.

So how do we bring them together in the best way?

There will be and can be no definitive answer from me today, but instead some strands of thinking. Clearly it depends on the issue, the question to be asked, risk appetite, urgency, scale and so on.

We must not stop asking the question “did something change and was that a result of what we did?” (and was it cost effective?).

But we need to broaden who is included in the “we”. It’s not just charities and governments who should be asking, and be asked, what works, but individuals and communities as well.

Data matters, but the crucial thing is that users, the community, are involved in framing the questions that we use that data to answer and are an integral part of the evaluation work.

So evaluation techniques must not be chucked in the bin – good longitudinal data, attempts to create control groups, doing sound quantitative work are all important. To go backwards on data would be strange indeed given the enormous leaps in the amount of data available and what we can potentially do with it.

But the questions we ask need to be guided far more by those in the communities and households that services aim to help. We need more involvement, we need participatory grant making, to use peer researchers, we need to exploit the possibilities of digital and data to understand and get more feed in. We need to experiment with the kind of things we are trying out in our Open Philanthropy work where multi-stakeholder panels decide where grants should go.

Yes, too, we do want to use good quality qualitative data – stories if you like – although a dependence on them makes it impossible to really answer questions of additionality. Storytelling and narratives are always important. We have often said ‘no numbers without stories’. But we have always also said ‘no stories without numbers’.

Getting that balance right is what we need. We must stay clear of the enticing and slippery slope back to where some in our sector feel most comfortable.

Where we don’t try to look at outcomes and their achievements in a rigorous way at all.

Because there are dangers in going that way.

Let me give an example from public policy from when I worked at the Education department. There were schools that on all measures were letting children down. Yet if the council (that still had power in those days) or the department proposed changing the leadership or bringing in a different governance form, there was often massive resistance from parents. They were not wrong to have an opinion – indeed maybe there were other aspects of the school they valued more or had different ideas on what needed to change to improve things. And they needed to be listened to. But policy makers needed to have both kinds of ‘data’ to make the best decision.

Another example is the infamous Drug Abuse Resistance Education (D.A.R.E.) program in the US, that instinctively sounded like it should work, was very popular with parents, got an awful lot of funding, but sadly has been shown to be ineffective.

For Balanced Evaluation we need both top-down and bottom-up. If we give up on impact assessment there will be waste, and likely scandals too when we find that no or little impact is being achieved. That may not matter to a funder with a lot of money to dish out with little due diligence or desire for metrics, but I’m sure it would matter to those whose lives have not been improved as much as they could have been.

There is indeed a danger of fetishizing top down approaches instead of seeing them as important tools in the armoury. But the same danger exists with being exclusively bottom up.

The exciting agenda is how we bring these together to get better outcomes for people. We at NPC have changed and will change. We want to work with others ploughing the same furrow – like Dartington Service Design Lab in the UK and Bridgespan in the US. But we will also be careful of the pendulum swinging too far.

I hope you will have heard some of the potential solutions to achieving Balanced Evaluation being talked about through many of the sessions at NPC Ignites this week: searching for the right balance and how to do these things well.

We will continue to work on these issues. I hope you will want to work with us.

There are exciting times ahead.