The Ballad of Ronnie (Cohen)
4 March 2019
A private equity firm and an international rock star might not be the most likely candidates for solving the question of impact measurement. Nevertheless, TPG Analytics and Bono are having a go. Their joint venture—the Rise Fund—has developed a methodology it hopes will help impact investors trying to maximise both the financial and social return of their investments.
Financial Times tables
The ‘impact multiplier of money’ (IMM) methodology draws on work from across the impact sector including longstanding approaches like social return on investment (SROI) and cost benefit analysis, as well as thinking like the interesting work NESTA has published on impact investing.
The IMM methodology is sensible as far as it goes, and it has some good features—like the offset between scale and depth, the need for the causal linkages to be strong to be useful, and for outcome data to be good quality. Applied well it can also give a sense of the magnitude of impact to help make judgements between extreme cases. But there are issues that cannot be dodged.
The number of assumptions that have to go into the calculation are legion and make the results uncertain. All along the line, different assumptions could be made with different end results. The methodology does not evaluate the impact of the specific investment, but estimates it based on an evaluation of a similar programme, which hopefully was rigorous enough to be lent on that much. The additionality of any programme (i.e., the difference it made compared to what would have happened without it) and of the investment that went into it is also a key metric that is hard to get at but is crucial to make these calculations meaningful.
As the IMM team themselves admit, ’for all the rigour that may lie behind a given IMM calculation, it is possible that some other analyst will rely on a different, equally valid anchor study that leads to a quite different number’. And the fact the methodology depends on value judgement about what different sorts of impacts are ‘worth’ (judgements that are needed to translate outcomes into dollars) makes this as a far less objective process than the results would imply.
Lies, damn lies, and single figure impact rankings
The simple output of the methodology is likely to be misleading. IMM tries to rationalise impact into a single figure, and this necessarily overlooks the fundamental nature of the problem. There is an emerging consensus in the impact investing sector of the weaknesses of applying of such a formulaic approach. Impact is complex, multi-faceted and in some cases intangible. It means different things to different people. In short, this is complicated stuff which doesn’t meaningfully lend itself to a straightforward, single figure result.
Expressing ‘impact’ as a single number akin to ROI risks glossing over the complexity of what comprises it both in terms of what that impact really is and how it’s been calculated. Single metrics often hide important effects. Lives saved, for example, ignores the quality of those lives. Is a longer unfulfilled life better than a richer shorter one? IMM also carries an inherent bias towards short-term measurable impacts at the expense of longer-term harder to measure impacts.
This isn’t to say there isn’t a place for single figure impact estimates. But these are most useful when comparing the cost effectiveness of similar programmes, aiming to achieve similar outcomes.
Another issue is that the methodology is at risk of being hijacked for chest-thumping. The push for a simple metric, combined with the underlying desire to do good, be seen to do good, and the nature of competitive markets, will lead to ratio inflation, where measurable impacts are subtly manipulated, sometimes unknowingly, to produce inflated figures that diverge from the underlying reality. We have seen this with the application of SROI. Thus the methodology contains the seeds of what undermines it. This risk can be addressed by third party auditing and applying good practices, but the incentives for investors to do either are weak.
Finally, the amount of effort that would be needed to use this methodology to look at each and every single investment means that we think it unlikely to be influential and mainstream enough to divert major quantities of impact investing into more socially impactful areas, companies, and projects. It could possibly be a cost-effective methodology if investors pool resources to fund independent and rigorous evaluations of different types of programmes and investments but it is hard to see that happening in a fiercely competitive market.
At NPC, we’ve been thinking about how to measure impact for a decade, and in more recent years have been helping impact investors, including the KL Felicitas Foundation, Social Investment Scotland and Propel Capital, better understand the impact of their portfolios or pipelines. In addition to the challenges to IMM noted above, we have also identified other issues to deal with, such as:
- How to compare the relative value of outcomes, which are more/less important for different people and communities. All too often those most affected by investments (besides the investors themselves) are not at the table when these relative values are being compared.
- How to account for the distribution of impacts, not just the size of impact. Maximizing impacts typically ignores their distribution. For example, is more wealth combined with greater inequality a good thing? Economics and psychology suggest there is not a simple trade-off to be made.
- The difficulty of accounting for the quality of an intervention, where relationships and environment can be as important as the mechanism.
- Ensuring how impact is conceptualised and measured does not create unintended negative consequences—for example, by ignoring long-term but difficult to measure negative side-effects.
- Dealing with ambiguity and confusion about what we really mean when we talk about ‘outcomes,’ ‘impact,’ and related concepts.
- Developing measurement frameworks that work in practice—that are good enough to steer capital in the right direction and avoid impact wash, but not so complex or expensive that will not be used or used properly.
These sources of complexity are addressed in some of the existing approaches. For example, SROI tries to get at the relative value of outcomes, by asking people what they value. The Impact Management Project acknowledges the fuzzy nature of outcomes, by exploring a shared lexicon. Our contribution is the Impact Risk Classification (IRC). This recognizes but sidesteps the problems of assessing the impact of investments altogether by comparing the impact management practices of organisations, rather than comparing diverse outcome results. This is based on our experience that sound impact management is usually a good proxy for impact achieved.
The intention of the IMM is great—helping channel more private capital towards projects that create social and environmental good, as well as profit. In Bono’s words: ‘Capitalism isn’t immoral, but it is amoral and it needs direction… If capitalism is to be a force for good we have to be able to measure when it’s doing good and when it’s doing harm.’ There’s lots of fantastic work going on to get us there, but for now—sorry Bono—I still haven’t found what I’m looking for…
See more of our work on impact investing here.