The cycle of good impact practice: Surveys

1. Develop your questionnaire

You may be able to use an existing questionnaire, or you may need to develop your own. In some cases, you may want to use a combination of both.

Existing questionnaires or surveys designed and tested by professional researchers are referred to as ‘validated’. They have benefits and limitations. Benefits include: they have been statistically tested to ensure high quality data, they can often allow you to compare your results with other organisations, and may save you time and give you more credibility with funders or other stakeholders. However, a key limitation is that they have not been designed with your service in mind, so you may find you need to adapt the survey for your users and service. If you do choose to adapt a validated tool, you will not be able to compare your data with others and the evidence relating to the measure’s validity is no longer applicable.

If writing your own questions, it is often helpful to use both closed and open questions.

Closed questions collect quantitative data. They are quicker for users to answer, and the results are easier to analyse. Types of closed questions include: choosing from a list of options; ranking options in order; and selecting a point on a scale.

The advantages of closed questions:

• Simple and quick to answer.
• Tend to get a higher response rate as they require less effort to answer.
• Can help users to recall relevant information.
• Can convey specific areas of interest.
• Easier to analyse responses.

Open questions collect qualitative data in the form of descriptive information; for example, by asking ‘What do you feel you need support with?’ They enable users to answer freely and in more detail, while providing insights you perhaps hadn’t anticipated.

The advantages of open questions:

• Allow users to answer freely and in more detail.
• Less likely to lead users to a suggested response.
• Require users to think carefully about their answers.
• Can provide insight into users’ true feelings and views, including those you didn’t anticipate.

When designing your questions, the following points can help to ensure good quality responses:

• Keep your questions simple, focused, and easy to understand. Remove unnecessary words, use non-technical language, and keep your sentences short. Be specific and avoid words that are open to interpretation; for example, use ‘daily’ or ‘weekly’ rather than ‘often’ or ‘usually’.
• Keep your survey short. It is beneficial to keep to a maximum of 15 questions if possible. Ideally it should take the respondent around 5 minutes for online surveys, 10 minutes for paper or phone, and 15 minutes for face-to-face, but this will depend on the nature of your relationship with the respondents and how much time you think they would be willing to spend responding to your survey.
• Avoid leading questions. These are questions that prompt or encourage a specific answer; for example, ‘Why were you satisfied with this service?’
• Ask one thing at a time. For example, split ‘Did you find the session helpful and interesting?’ into two questions, because “helpful” and “interesting” are not the same thing.
• Focus on the objectives of your survey. It can be tempting to take advantage of the opportunity to gather information that is not related to your immediate objective. For example, you may want to ask about other aspects of your service, test interest in an event or project, or gauge opinion on a particular issue. This will make your survey longer and less appealing to participants, so should be carefully considered.

2. Choose how to deliver your survey

The benefits differ depending on whether you conduct your survey online, on paper, or in-person/over the phone:

Online

• No printing or postage costs.
• Takes less time to issue, administer, and analyse the results.
• More flexibility in terms of layout and design.
• Useful for discussing sensitive topics as users can complete in private and in their own time.
• May be easier to fulfil accessibility requirements (e.g. using large font or translating into other languages).
• Reduces the risk of responses being affected or biased by an interviewer.

Paper

• Available to users who may have difficulty completing an online survey.
• Useful for discussing sensitive topics as users can complete in private and in their own time.
• Reduces the risk of responses being affected or biased by an interviewer.

In person or over the phone

• Available to users who may have difficulty completing an online survey.
• Interviewer can offer support if the respondent is struggling.
• Speaking to a person may generate more thoughtful responses.
• Speaking over the phone may offer a greater sense of anonymity compared with an in-person interview (which can result in more accurate responses).

3. Design your survey

The way you design your survey will influence how people respond to it.

Provide participants with clear instructions. This could include:

• Why you are conducting the survey and how the information will be used Information about your organisation and any associated stakeholders.
• Any potential benefits for the participant; for example, a chance to influence services, or an incentive for completing it.
• Any potential discomfort or risks; for example, the need to share sensitive information, personal data or documentation.
• Instructions for navigating the survey; for example, a requirement to answer all questions.
• How long it will take to complete the survey and when the survey will close.
• Contact details in case participants have any questions.

Top tips for effective survey design:

• Ensure your survey is accessible and easy to read. It can be helpful to think about formatting, font colour and size, and spacing. Avoid including too much information in one section. Consider if any of the respondents are likely to have accessibility requirements.
• Organise your questions in a logical way. Group similar questions together. It may make sense to place important questions at the beginning and profile questions – for example, age, gender, and location – at the end.
• Thank respondents for their participation at the end. You may want to consider whether you plan to share the results with them, and add this to your thank you message.
• Test your survey with a small group of people before rolling it out. This will provide insight into how easy it is for participants to respond, and how useful the results are, so you can tweak it.
• Make sure it goes to the right people. It’s critical that your survey reaches the people you want to hear from, or else the results may be distorted.

Adapted from content from NCVO.