We recently asked readers to pose their research questions on the B2B Marketing LinkedIn group. One question came back more than once. “In a B2B context how can I ensure a survey is reliable?”. A big question but let me give it a shot.
Reliability boils down to whether the right questions have been asked to enough of the right people.
Survey questions need to be unambiguous, unbiased and asked in a standardised, objective format. Beware of three classic mistakes here:
- The ‘double barrelled’ question including two mutually exclusive items, e.g. ‘how satisfied are you with the quality and speed of response?’
- The leading question containing implicit assumptions, e.g. ‘how much do you expect material costs to increase?’
- The weighted scale which biases towards a favourable answer, e.g. ‘Are you extremely, very, quite or not at all satisfied?’
The next challenge is to ensure the survey has accessed those whose opinions really matter. Ask yourself:
- Have the major organisations in the market been surveyed?
- Are key sub-segments of the market represented relative to their importance?
- Can you be sure that individuals surveyed are really budget holders or influencers?
And finally the perennial challenge. How many interviews is enough? One approach is to use a test of statistical precision known as the ‘margin of error’. This calculation indicates the degree to which answers might vary were the same survey to be repeated 100 times, e.g. a 3% margin indicates that in 95 out of 100 identical surveys the result would vary by no more or less than 3%. Aiming for a low margin of error is the ideal approach but is not always practical in B2B surveys where there is a small pool of potential participants to draw on who are not always accessible. An alternative then is to use the margin of error alongside four rules of thumb when specifying a target number of interviews:
- What level of precision is adequate to make decisions?
- Will additional interviews add sufficient value to justify additional cost?
- Will it be possible to analyse differences between key segments?
- Is the ideal number of interviews actually practical?
Find out more about Circle Research’s B2B research services here.
Enjoyed this post? Subscribe and receive new posts by RSS