Project Oracle: transparency and good ideas

One of the most interesting findings from our (NPC’s) recent survey of charities’ practices of and attitudes towards measuring impact was the very strong support for the idea that charities should be transparent in reporting their impact. Over 70 percent of the respondents agreed or strongly agreed with the statement, ‘Charities should be encouraged to report failures or negative results.’ So why does this not happen in practice? Why don’t charities’ websites include stories of ineffectual programmes and disappointing results? Why are these so rare?

The answer, I think, is obvious. Many chief executives of the charities we work with are torn between wanting to be transparent in the results of what their charity does and so contribute to the sector’s knowledge of “what works”, while also wanting to show their charity in a good light. This creates a dilemma if results are disappointing. The conventional wisdom is to attract funds and motivate staff and beneficiaries, you need to demonstrate good results. Any charity that puts its head above the parapet and reports poor results or evaluations is at risk of losing funds, staff morale, and stakeholder support. So typically—with some exceptions—they don’t. Poor results are hidden.

The biggest problem is that, as a consequence, other charities fail to learn. Poor results provide valuable information about what does not work, or does not work effectively. At the launch of Project Oracle, Professor Nick Tilley of UCL repeated a very apt phrase he had heard from a police officer. The officer despairingly noted that the police “are doomed to succeed.”  If all programmes succeed, then everything works, and anything goes. And there is no progress. Because poor results are hidden, charities are left to repeat each other’s failures.

What is less obvious is a way out of this dilemma. But Project Oracle’s Evidence Competition provides the beginning of an answer. The solution is to create a culture where adding to the collective knowledge of what works and what does not is as important—and is equally rewarded—as the credit a charity gets for showing it is successful.  If charities and funders make this shift together, we would suddenly see an opening up of data and evidence as to what programmes and interventions help people the most. The impact would be significant. To help initiate this shift, Project Oracle is running an Evidence Competition. Charities that work with young people in London, who register for Project Oracle, and complete a self-assessment of their evaluation and monitoring practices, are invited to submit past evaluation reports or ideas for future evaluation projects into a competition. In addition to a receiving a prize valued between £2,000 and £5,000, the winning charities will be held up as being great examples of transparency and rigour in reporting results, regardless of what the results actually say. This helps to shift the incentives towards learning. By itself this competition cannot change the culture of reporting impact. But it is a good start and will also generate good ideas for monitoring and evaluating results. The closing date is December 31st, so check out the site.