2. Qualitative data
Hearing from service users about their experience and perceptions in their own words.
In our guide to Understanding impact, we explore how to use your theory of change to build a measurement and evaluation framework. In this closer look we explain in more detail how to use qualitative research to understand where you are making a difference.Case studies should capture a wide spectrum of experiences of service users experiences or professionals, not just the cases in which the project worked best. Click To Tweet
Qualitative research nearly always involves talking to service users and stakeholders about their experiences and perceptions, most importantly, it allows people to answer questions in their own words. Qualitative discussions are more free-flowing and can give greater insight into participants’ views; identifying not only what they know, think and do, but also why.
Qualitative research can be used on its own but is best deployed as part of a mixed-methods approach, combined with numerical data. We like the maxim: “No numbers without stories, no stories without numbers”. Qualitative research with service users is particularly valuable, but the approach is also suited to interviewing other stakeholders, including staff, volunteers, partners and funders, community groups and people working in other services. In our experience, people tend to prefer engaging in research in this way as there is greater scope for them to express themselves.
The following are some of the potential benefits of qualitative research charities:
- While numerical ratings of a service user’s experience can tell you how your service is doing, qualitative research can provide more information to help you improve it.
- Quantitative data can show that your service caused a change, but in-depth interviews or focus groups can help you discover how that happened, helping you adapt and refine your project for other contexts and user groups.
- Statistics can be a powerful way to show impact achieved, but well-researched case studies can show a richer, more personal story of change.
- Qualitative research is particularly helpful when you are aiming to understand the experience of particular subgroups of service users, or those who have achieved particular outcomes, in order to learn why.
- Qualitative research is the best way to get feedback on how to improve.
In this section, we give an overview of the main methods for qualitative research (see our paper Listen and Learn for more detail).
Qualitative interviews are frequently used in social research. They differ from questionnaires in that the answers are not pre-categorised as they are in questionnaires and the interviewee is given the space to expand on their answers. Interviews are normally used to give more detailed information on a particular topic and to explore the topic from a person’s own viewpoint. Qualitative interviews are more appropriate to understand how and why change might have happened and are generally not easy to use as measures of impact.
When interviewing you should:
- Prepare a topic guide rather than a questionnaire.
- Use open questions (such as ‘How…?’, ‘Why…?’, ‘In what way…?’).
- Avoid suggesting answers or biasing responses.
- Approach sensitive topics carefully, appreciating the impact on interviewers.
- Ensure all interviewees are asked the same question while allowing the interviewer to probe interesting answers.
Observation involves the systematic observation and recording of behaviours and interactions of people in their environment. Observation is used to generate detailed descriptions of certain behaviours or traits and can be useful in recording information that is otherwise difficult to obtain with other research methods. When carrying out observation, it is important to:
- Have a theory about what you are observing.
- Decide on your focus.
- Systematically record what you have observed.
- Use analysis to identify themes and patterns of behaviour to generate any conclusions.
Like interviews, observation is best used to understand a topic or issue in detail and may not be the most efficient tool to measure impact quantitatively, though it can be done.
A focus group (or discussion group) is a type of research method where a group of people are asked about their thoughts, attitudes and beliefs towards a topic of interest, whether that be a product, service or social issue. Questions are posed to the entire group and people share their thoughts in an interactive manner. Notes are taken at focus groups and later analysed to look for trends and themes.
Focus groups are very useful for understanding why a service leads to change for people, but because of the subjective nature, the possibility of bias (focus groups can be dominated by the strongest voices), and because of the small numbers involved, they are not appropriate as quantitative measures of impact.
Ad hoc, informal data and feedback
Ad hoc data and feedback arise from day to day interactions with your beneficiaries or other stakeholders. These interactions may contain stories and feedback on how your work may have positively impacted someone or something. Some of these anecdotes may be hard outcomes, such as a change in legislation or someone moving into employment. These can be recorded systematically. Other anecdotes may be ‘soft’, for example, someone telling you how their confidence has improved or how they made new friends on your course.
Such feedback is useful and is one piece of evidence that can be used to test your theory of change, but it should not be used as the only or main source of impact data.
A spreadsheet can be used to capture systematically any ad hoc data on your impact. Recording e-mails or oral comments, texts, and so on, can help an organisation see trends in these stories and build up a picture of impact over time.
Case-based approaches are a useful tool for in-depth understanding of how changes occur in context. The aim is to build up enough information to create a rich description of how the project has or has not affected the individuals, reasons for change, and other factors that are important.
Case studies need to be chosen carefully by charities to reflect the aims of what you are trying to understand. Importantly, case studies should capture a wide spectrum of experiences of service users experiences or professionals, not just the cases in which the project worked best. To aid credibility, charities can choose a small sample of cases randomly or according to certain criteria. Case studies can be solely qualitative, but it is often useful to get quantitative and secondary data too, which tends to support the case as it makes it a more insightful piece of analysis.
Case studies are often more effective when not anonymised, so getting permission from the individuals or an organisation being written about is advised. It is often good to get consent early, depending on how sensitive the case study is. We also recommend you send a draft copy of your case study to selected individuals, so they can verify or correct any information.
There is often no fixed template, as it needs to be written in a manner that is engaging to your audience. However, we advise that you think about your theory of change as a form of template for a case study, as it allows you to go in-depth from your activities to outcomes and impact. It also lets you explore your change mechanism and discover how your activities led to your outcomes.
Read our full guide to using your theory of change to build a measurement and evaluation framework.
How to turn your theory of change into a plan for measurement, the five types of data you will need to pay attention to, and how to prioritise what to measure.
Mechanisms – A missing component in your theory of change?
A realist approach to evaluation, which stresses how contexts interact with mechanisms to achieve outcomes.