Whose data is it anyway?
19 September 2018
Spark Inside deliver coaching programmes in prisons to encourage rehabilitation, reduce violence and lower reoffending. Since 2012, they have worked with over 1,000 people living and working in prisons across London and the South East.
Their CEO, Baillie Aaron, will be speaking at NPC Ignites and here she sets out some of the challenges of evaluating programmes with incarcerated people and how they have used co-design to try to overcome them.
At Spark Inside, we take evaluating our impact seriously: both of our programmes are evaluated by independent researchers, who are specialists in social impact.
Why we prioritise evaluation
To us, programme evaluation is not about veneer, optics or fundraising; it’s about understanding how our programmes work, how they are received and how they can be improved to better meet the multiple needs of our stakeholders. We exist to meet the needs of people in prison: we work for and with them in everything we do.
Evaluations are a charity’s way of being accountable to its participants. No matter how well intended our programmes are, we should never just assume we know what our impact is. We recognise that our programmes can have unintended consequences and inadvertently cause harm to our participants, and we’re committed to investigating this and being held accountable for our impact through robust evaluations.
Why we involve our participants in designing our evaluations
We undermine people in prison when we decide what metrics we use to define and measure their success. Imposing a directive theory of change goes against our organisation’s ethos and our coaching approach: we believe people in prison are the experts of their own lives, with the power to change their futures.
Turning ‘success’ into measurable outcomes is complex—we know success looks different from person to person. The coaching relationship, which is fundamentally person-centred, requires that our participants will work with their coaches to define their own successes, according to their own values, and plan their journeys accordingly. This raises the question: how do we design impact measurement to reflect all of the outcomes that are possible in an individualised, tailored programme? How do we ensure we encapsulate success to ensure we have a meaningful theory of change?
Our theory of change has been, and will continue to be, an iterative process: a feedback loop. With each evaluation, we are working alongside our participants and evaluators to ensure our impact measurement better reflects their journeys.
Some organisations in the criminal justice sector see reoffending rates as the most meaningful impact measure, which is bolstered and reinforced by some funders and commissioners who focus only on reductions in reoffending. We believe the sector’s focus on this simplistic measure ignores the reality of people changing their lives and moving away from crime. We are proud to have co-developed a theory of change with people in prison, that acknowledges that reoffending is a symptom of other causes: poor wellbeing and resilience, low self-esteem and confidence, impulsivity and lack of decision-making skills.
How we implement co-design in evaluations and the lessons we’ve learnt
Being conscious of our programme’s potential to cause unintended harm taught us how our evaluation could cause harm too.
For our Hero’s Journey life coaching programme, we used to work with academics who wanted to measure impact comprehensively and produced a long questionnaire covering attitudes about offending to problems faced with addictions, money and employment. This was reviewed and trialled by our Youth Advisory Board, made up of young Londoners who had been directly or indirectly affected by the criminal justice system—including young people who had just left prison. We implemented their feedback, that we should cut down the length of the surveys and change the language we had used throughout.
However, it wasn’t enough. Our coaches, who were responsible for giving out the questionnaires to participants, were rarely handing them out because they perceived these questionnaires to be causing harm to participants. The 65 questions in the survey were intrusive, focusing on criminogenic factors. They required participants to divulge sensitive information about their past, thereby reliving their trauma. In addition, many of the questions had already been asked by other organisations or criminal justice officers, so were seen as unnecessary repetition.
When we commissioned The Social Investment Consultancy (TSIC) to evaluate Hero’s Journey, they redesigned and drastically shortened our questionnaires so that they focused on short statements about how our participants were currently feeling, rather than intruding on their past mistakes and trauma.
After these changes, our questionnaire response rate has grown to 84%, which we see as a great success.
But we know it’s still not good enough. Our programme team recently conducted a survey of our participants’ views on our evaluation methodology in HMP Wandsworth. We learned that those with low literacy levels were too embarrassed to ask for help. Some felt wary of us asking the same questions with different words – caused by using two validated scales with similarly worded questions.
Most interestingly, many felt suspicious about our motivations in conducting the research; they didn’t know where the information would go or whether or not it would be used against them in the future. This scepticism and concern about repercussions from our evaluation raises questions about its validity, as participants may have self-reported more positively in the pre-questionnaire.
As a result of these findings, we’re working on how we can safeguard our participants against repercussions and provide them with extra support filling out the questionnaires. Going forward, we will more closely embed co-design into our evaluation process – we’ve just won funding to establish in-prison advisory groups, to take more ownership of our impact.
All of our programme evaluations are publicly available to read and download on our website, and we’re working on making them more accessible to our participants and people who live and work in prison. In addition, we’re recruiting a trustee with a strong participatory research background, to ensure that we’re held accountable for our impact at the highest level of governance in our organisation.
To find out more about Spark Inside’s work, and impact, please visit their website. Baillie will be discussing the issues raised in this blog on 10 October as part of NPC Ignites. Book your place now.