people planning

Evaluating systems change

By Ellen Care and Ben Fowler 21 May 2021 4 minute read

This joint blog by Collaborate and NPC has been posted on both organisation’s websites. It details our work developing a maturity model for Save the Children UKs Early Learning Communities programme.

Since autumn 2019, Collaborate and NPC have been working together as learning and evaluation partners to Save the Children UK’s Early Learning Communities (ELC) programme. Our partnership combines Collaborate’s systems change knowledge with NPC’s evaluation expertise, to develop practice in the emerging field of systems change evaluation.

The ELC programme aims to make a sustainable difference to the lives of children in four places (Bettws, Feltham, Margate and Sheffield) through a multi-sectoral, multi-agency approach. The programme seeks to better align and connect local activities, and identify how to improve the performance of local systems and improve early years childhood development for those who live in the area. It also aims to share learning about what is most effective in creating systems change. Since 2018, the ELCs have been bringing together local partners and stakeholders, listening to families, building relationships, and collaboratively developing theories of change and local strategies to support children and families. More recently, they’ve started to test new ways of working, both in terms of direct interventions with children and families, and by helping partners from the statutory and voluntary sectors work together. Crucially, the work done by the ELCs helped the local systems to better respond to the pandemic, enabling them to provide the necessary support for the families in their area.

A developmental evaluation approach

Evaluating systemic change is difficult for various reasons. Systems have lots of different elements, so change is not always obvious and happens at different times and at different speeds. Change isn’t linear, so can sometimes seem to be going backwards as well as forwards. This means understanding the impact of the ELCs on children’s speech and language and emotional development could take years. Even when change is observed, it can rarely be attributed to specific activities or causes. As a result, we have chosen to take a developmental evaluation approach that focuses on helping the ELCs to learn and adapt as they go. We are mainly focusing on how the whole system is changing rather than directly evaluating specific interventions or services.

Our challenge has been to find ways to understand the progress made by the ELCs when they are in their early stages. A key part of our response has been the development of a systems change maturity model, capturing ten systems change conditions which we believe will enable collaborative, long-term and sustained change. While we cannot yet know the impact for individual children and families, we can assess the strength of these conditions as foundations for achieving this impact in the future.

The maturity model was developed by Collaborate, in partnership with NPC, Save the Children UK and the ELCs, building on ideas and frameworks in Collaborate’s reports such as Building Collaborative Places and From the Margins to the Mainstream. We also referred to similar tools developed by the NHS for integrated care systems, the Early Intervention Foundation for maternity and early years, and Sheffield Hallam University and Save the Children UK’s own work on Children’s Communities.

The model

The maturity model was developed by Collaborate, in partnership NPC, Save the Children UK and the ELCs. Follow the link below to go to a PDF of the maturity model.

A one page summary version of the maturity model is available here.

The maturity model sets out four levels of maturity, with indicators of what progress looks like at the different levels for each of the ten systems change conditions. Our evaluation methodology includes interviews, focus groups and document reviews, to enable us to look for evidence of these indicators in the ELCs. This helps us assess their current level of maturity and identify particular areas of practice that need to be strengthened.

Below is one example of the indicators for the systems change condition ‘Trusted, collaborative relationships.’

Example of the indicators for the systems condition ‘Trusted, collaborative relationships.’

As with all diagnostic tools, they inevitably involve some judgement and crucially we intend to use the model to aid understanding and drive learning and improvement rather than performance management—though we do hold internal meetings to ensure we are using the model consistently across the different ELCs. We are not trying to find flaws in the ELCs or measure them against pre-determined expectations. We’re also not seeking to directly compare progress in one ELC to another, because we know that their journeys will be driven by different local contexts.

For all these reasons, the conversations that the maturity model generates are as important as, if not more important than, the judgements reached. We recognise that progress will be uneven, so exploring why some systems change conditions are more advanced than others and what has contributed to this will help the ELCs to shape and refine their approach.

We started to use the maturity model with the ELCs in spring 2021 and as a result have made a number of improvements to it. In particular, these relate to how best to manage the inevitable overlap in the systems change conditions and identifying deepening understanding of what different levels of maturity mean in practice. We’ve also learnt a lot about our evaluation methodology and how to make the best use of our limited time with practitioners on the ground, in order to balance the breadth and depth of information that the maturity model seeks to cover.

Overall, we found that the model provided a useful framing for our research, an effective way to tell a compelling story about progress so far in the ELCs, and it identified opportunities for where they might go next. It also highlighted opportunities for learning across the four ELCs.

We think the model would be relevant to other systems change endeavours, including those beyond the early years sector, and we hope that it contributes to the development of wider thinking about how systems change happens and how it can be evaluated.

Look out for a future blog which explores how the Covid-19 pandemic has affected systems change progress in the ELCs. If you would like to reference or use the maturity model, please attribute it in the following way: Early Learning Communities Systems Change Maturity Model. Developed by Collaborate, in partnership with NPC and Save the Children UK. Accessed via this blog: https://collaboratecic.com/since-autumn-2019-collaborate-and-npc-have-been-working-as-learning-and-evaluation-partners-to-c319125bf9b5.

To find out more about this work and the maturity model, please contact Dawn Plimmer on dawn@collaboratecic.com and Ben Fowler on Ben.Fowler@thinkNPC.org. If you want to find out more about the Early Learning Communities themselves, please contact Sarah Crosby on s.crosby@savethechildren.org.uk

Systems have lots of different elements, so change is not always obvious. This blog shares how @CollaborateCIC, in partnership with @NPCthinks and @SaveChildrenUK, developed a model to measure systems change progress: Click To Tweet

Categories:

Footer