Concentrating on better data from a small representative group, rather than lots of poor-quality data from lots of people.
Reporting your response rate
An advantage of both random and stratified approaches is that they enable you to calculate a response rate. This is a simple calculation of the proportion of people you selected for the research who then actually took part. It is one of the main ways to assess the quality of research because a low response rate greatly increases the risk of bias. If you can quote a response rate it will add greatly to the credibility of your findings.
A good response rate is generally over 50%, while 70-80% is exceptional because the majority of people have been interviewed and the risk of bias reduced. But the overall response rate is not the only measure of success, you also need to consider differential response; whether some groups responded more than others. For example, a 50% response rate in the general population looks good, but if only men took part and no women did then the sample is clearly biased. The section below on checking for bias looks at this issue further.
Checking and correcting for bias
No matter how good your sampling approach is, you should always do what you can to check for bias. Remember, the aim is to ensure your sample is as representative of your target population as possible, so you should take whatever you know about the population and compare it to the sample. For example, if you know that half of your service users are men and half are women, then you will want the same proportion in your sample. Similarly, if you know that half of your service users worked with you for three months and half for six months then this should also be reflected.
The more you go through the process of checking your sample against the population (and report this), the more robust your research will appear.
If you do identify a bias, you may decide to correct this by doing more interviews with groups who are under-represented. An alternative approach, applicable to quantitative studies only, is to weight the data. This is a statistical adjustment of the data that corrects biases. At its simplest, it’s quite easy to do. For instance, if the split between men and women in your population is 50/50, but in your sample it is 75/25, then you could correct this by giving men in the sample a weight of 0.67 and women a weight of 2 to bring it back to 50/50. The drawbacks are that technically it reduces statistical confidence in the findings. You will also need more advanced software like SPSS or Stata to do it. Weighting by more than two variables starts to get very complicated.
Research should not rely on speaking to those who are easiest to engage, as this produces biased results. Rather, if any sampling is involved, it should be based on a systematic approach to selecting people and careful comparison of the sample against the population you want to generalise.
Sample size is important, but not as important as ensuring the representativeness of the sample. This applies as much to qualitative research as it does to quantitative.
Finally, you should explicitly address the issue of bias when reporting your findings. Being upfront about concerns or limitations, and how confident you are in your findings, is an important part of being credible.