A Question of Bias

Research results that are free of bias are both desirable and unattainable. The challenge is to identify the potential sources of bias and then minimise and account for their effect.

Non-response bias

It is very rare for a voluntary survey of any size to attract a 100% response. Research has shown that the strongest driver of voluntary response is the expectation that something will happen as a result, so the response rate can itself be a useful indicator of people's views.

As the response rate falls from 100% so an ever-larger proportion of the original sample becomes a source of bias. If the response rate drops below 50% more than half the people are unrepresented in the results. If the non-respondents are evenly spread across the range of views there is no problem. However, if the non-respondents have some factor in common – especially particular positive or negative views – the final results could be heavily biased one way or the other.

At high level we might reasonably assume that non-response bias is fairly evenly spread – that people failed to respond for lots of reasons and that, overall, their views are similar to the people who responded. With smaller groups this assumption about non-response bias becomes increasingly risky. In a group of 30 people, 9 non-respondents could easily share a very small number of reasons for staying out.

Demographic information such as splits by grade or gender or length of service or function can help to identify particular and obvious groups of non-respondents, but this is only a crude indicator.

The only way to reduce non-response bias is to maximise the opportunities for people to complete the survey and demonstrate top-level commitment to taking action on the results.

Proximity

It has long been recognised in research that proximity is a huge source of bias.

Where respondents feel themselves to be part of a smaller and smaller population they become more and more likely to adapt their responses to some “felt-acceptable” norm (e.g. they will tend to reflect their liking – or otherwise - for their boss, or will suppress their true opinion for fear of retribution).

Furthermore, leaders “in proximity” may be motivated to try and influence the results – something that is easy to do in small groups. The consequence of this is that at higher levels the data become unreliable and the whole survey becomes a measure of popularity levels rather than of climate effectiveness. In short, taking measurement down to low-level renders the higher level figures unreliable.