How to rebalance data for the 21st century

This blog shares the thoughts of Jara Dean-Coffey and Bonnie Chiu, two social sector leaders who recently spoke at an NPC and Charities Evaluation Working Group event on rebalancing data for the 21^st century. This free event was part of our 2020 annual conference, NPC Ignites. Our speakers discussed their work on trying to shift traditional third sector evaluation and measurement practices towards a more equitable approach.

What needs to change?

Both speakers stressed that for centuries, research and evaluation has been used as a tool for control and oppression. In the social sector, evaluators and funders still act as neutral and objective figures who can determine the validity and worth of programmes without bias.

Jara Dean-Coffey, Director of the Equitable Evaluation Initiative, spoke to us about the need to redress this unequal power dynamic, encouraging us to rethink how evaluation and research can either reinforce inequalities and harmful structures, or seek to address them. Since 2019, over the course of five years, the Equitable Evaluation Initiative is aiming to build a field of practitioners who are committed to embedding the Equitable Evaluation Framework (TM). The Equitable Evaluation Framework (TM) helps to challenge fundamental beliefs about evaluative work, and to realign the design and implementation of evaluations as to reflect evolved definitions of evidence, rigor and validity, embrace complexity and be in service of equity.

Bonnie Chiu is the Managing Director at The Social Investment Consultancy, and an advocate for diversity and inclusion across the social sector. Bonnie spoke about the need to decolonise the measurement and evaluation space, and to reckon with our colonial history as a field.

‘There are so few people of colour working in research and evaluation. It’s not accidental, it’s because there have been so many centuries of how research has been used as a tool of command and control and colonisation, where people of colour are subjects, not at the decision-maker table.’

Changing mindsets before behaviours

Both Jara and Bonnie reflected on the importance of recognising and understanding the current paradigm for evaluation and research before launching into more practical action. In order to reconstruct the philosophy and assumptions that underpin practice, we first need to recognise what those look like.

The Equitable Evaluation Framework (TM) encourages practitioners to recognise and reflect on some commonly held principles—such as the idea that evaluators or funders are objective experts who are able to determine the merit and worth of something. All too often, the frame of reference for determining this merit and worth is centred around the experiences of privileged groups, such as those who are white, male or able-bodied. Historically, as Jara pointed out, evaluation has not been about creating change, and evaluators have seen themselves as removed from change happening on the ground.

‘[Evaluation] does not have a history of being about change. And so, everything that has come to this point is built on this idea of neutrality and objectivity which perpetuates its origins.’

We need to get better at recognising our own privilege and power, and get comfortable with complexity and uncertainty, before we start thinking about concrete outputs. Bonnie similarly reflected on the need for us to understand our own complicity—both as individuals and as a collective group—in perpetuating inequality.

‘Evaluation as an activity is not neutral … if we are making money from this work, we are taking resources, and it is our responsibility to examine our own biases and where our practices and behaviours might have unintentionally or intentionally perpetuated inequalities.’

Participatory approaches and power dynamics

Current approaches to research and evaluation tend to reinforce unequal power dynamics between funders or evaluators, and participants. Both Jara and Bonnie feel that we need to work harder to build long-term, trusting relationships with participants by being honest and transparent, and by giving participants ownership over the evaluative process.

Currently, the process and design of evaluation frameworks tends to be top-down and funder-driven, and Bonnie encouraged us to move away from this approach towards more participatory methods.

‘I strongly believe that people who are experiencing the problems themselves are the best ones to actually determine the solutions, and too often we have this very elitist, top-down approach to determining social change.’

Instead of knowledge and experience being extracted from users, participants should have ownership over this process, for example by being paid to be peer researchers. Plus, evaluators should use their power to ask this of funders, even if it’s not laid out in the terms of reference. Users should also have more power throughout the evaluation process, for example when setting the research framework or through governance structures.

Findings should be shared back with users; currently, research is mostly conducted by and for a white audience, and the field needs to be better at engaging Black, Asian and minority ethnic (BAME) groups. To do this, we need to consider what structural barriers might be in place—for example, inaccessible academic language; inequity of funding, which means fewer BAME-led organisations have the resources to engage in research and evaluation; and a lack of transparency around commissioning processes. We also need to diversify talent in the evaluation space—currently there is no clear career path into the field, and we need to consider how to make the space a safe and enabling one.

Policies, guidelines and standards should also be written in a way that explicitly encourages equitable approaches to measurement and evaluation, and we should think more about what we can learn from alternative methodologies, such as feminist or intersectional approaches, or international approaches that may have different or fewer ‘rules’ on how evaluation should be conducted.

Creating change towards equity

If we want to create change in the social sector and embed anti-racist practice in our work, we need to consider the history of measurement, reflect on our current frameworks, and then start to shift the mindsets and behaviours that shape the research process. Jara’s final prompt for funders applies to all of us who need to consider how to best use our power to effect change:

‘If you are a funder—how can you think about evaluative practice as part of your strategy, to effect change that is towards equity and is anti-racist? How can you reimagine how it serves your mission, and who might you bring along with you, and what might be possible?’

Did you miss our annual conference? You can watch our session on rebalancing data for the 21^st century here. All of the other proceedings from our annual conference were also recorded and you can continue to purchase access to them here.