Tackling the tricky question of attributing impact

A question we often get asked is ‘how do I know that the difference I see in young people’s well-being is due to my programme, or due to other causes?’ This question is not new for researchers and one that you will find comes up in any survey or research that you do.

The simple answer is that you can never be 100% certain. In their laboratories, scientists put enormous effort (and money) into isolating different factors or conditions in their experiments. This allows them to focus their experiment on what they want to test.

Unfortunately, social scientists don’t have this luxury. Because they work with people, it is impossible to recreate laboratory conditions.

Instead, the nearest thing to recreating lab conditions is to use a ‘control group’. This is a comparison group, drawn from a population with similar characteristics and selected at random, which allows you to say ‘what would have happened’. By comparing the results of your experiment with what happens to the control group over the same period of time you can isolate the impact of what you are testing.

It is always worth thinking about whether you could run a control group within your survey. However, it may not be possible or desirable. For one, it is expensive and it is not worth doing unless you do it properly. And second, if you work with a particularly vulnerable group it may not be something you want to do as it might mean not providing a service to a comparison group that you know really needs it.

Without a control group your results will still give you a good sense of the difference you make to young people’s lives. When presenting your results it can be useful to talk about ‘contribution’ rather than attribution—recognising that there are many influences on young people’s lives and your work is one of them.

• NPC’s Well-being Measure is a fully validated online survey-based tool that measures how children feel and can be used by organisations working with young people aged 11 to 16 to show the difference made to their lives. It automatically analyses your results against our national baseline so your results are in context and comparisons can be made.
• Our Data Labs project aims to open up government administrative data to the not-for-profit sector to help organisations better understand the impact of interventions. The Justice Data Lab model reports back not only aggregate re-offending rates of the people a charity has worked with, but also s a re-offending rate for a statistically-matched control group, which shows more robustly if the intervention has made a difference.