Sample Sizes and Precision

For a quantitative survey, one of the most commonly debated questions between clients and survey providers is, “What should the sample size be?"

There's no simple answer to this one. But the key to bear in mind is that the larger the sample, the more precise the results.

Statisticians define precision by the width of a confidence interval, commonly the so-called 95% confidence interval. (In the press, this is sometimes reported as the margin of error, or sampling error). In 2009, a survey of 1,305 British adults found that 40% of people had never watched "Strictly Come Dancing". The 95% confidence interval was a little under +/-3%, which means that we can be 95% certain the true number is somewhere between 37% and 43%.

Non-statisticians are often surprised to find that relatively small samples can give quite accurate estimates of the views of large populations. For example, a sample of 384 is theoretically adequate to measure the views of the UK population to within +/- 5%.

The reason why some opinion polls turn out to be inaccurate is not usually because the sample size is too small, but because the make-up of the sample does not adequately reflect the make-up of the population, i.e. certain groups in the overall population may be over- or under-represented in the sample.

Organisational surveys differ from surveys of the general population in that it is theoretically possible to survey everybody, or almost everybody. In this case it could be argued, as some statisticians do, that the margin of error is zero. However, we do not subscribe to that school of thought. We recommend that for attitude surveys conducted in organisations, sampling error should be calculated in the normal way.

In organisational surveys, the question of sample size is often turned around, because the entire organisation, or an entire department, is surveyed. Here, the sample size is not controlled by the survey provider, but is simply the number of people who respond.

The question of interest then becomes "Given the number of respondents we have, what is our margin of error?". And the corollary is of course "How can we ensure the highest possible response rate, so as to reduce the margin of error as much as possible?"

However, high response rates are desirable in organisational surveys for another reason as well. A 20% response rate from a population of say 5000 would provide a sample of 1000 employees. Even though the margin of error would be quite small, the results in this case would be suspect because of 'non-response bias'. That is, the 20% of employees who did respond are likely to be different in important ways from the 80% who did not, and hence unrepresentative of the organisation as a whole. Low response rates in organisational surveys limit the conclusions that can be drawn from the results, and every effort should be made to ensure high levels of participation in the survey process.