There’s horse meat in my survey

By Andrew Dalglish - 2nd April 2013

What you see isn’t always what you get.

Horse meat in our Findus Crispy Pancakes.  Sub-prime mortgages hidden within bundles of triple-A rated loans.  Dodgy surveys parading as fact.

This deception – whether intentional or accidental – has far-reaching repercussions.  In the case of faulty research it can lead to sub-optimal decisions and missed opportunities.  And it’s an easy mistake to make.  Statistics carry a sense of authority and it seems counter-intuitive to challenge the statement that ‘customers told us’.  But challenge we should.

Next time survey numbers are presented as conclusive fact, here are three questions you might want to ask.

Who was surveyed? 

The power of surveys is that by speaking with just a proportion of the target market, the likely opinion of all can be deduced.  However, this is only true if two criteria are met.

First, the survey must comprise sufficient responses to justify the label ‘reliable’.

Second, it is essential that those responses reflect the market structure.  Here a lot of surveys fall down.  In their quest for capturing as many opinions as possible, they fail to ensure that more weight is given to proportionately larger (or more important) groups.

No really, who was surveyed?

Dig a bit deeper and sometimes you’ll find the horse meat – survey respondents who look legitimate, but aren’t.

So ask who was invited to participate in the survey.  Was it a controlled, vetted list or could anyone respond?

Ask what steps were taken to screen out inappropriate respondents.  Was everyone posed non-leading questions to determine their role and level of authority?

Ask if any checks on identity have been conducted post-survey.  Are we confident that people are who they claim to be?

Who wasn’t surveyed?

Just as important is to understand who might have been excluded from the survey and whether their absence may distort the picture.  The principle consideration here is methodology (online, telephone, etc.) as not all survey techniques provide an equal opportunity for the target market to respond.

Tuck in!

Enjoyed this post?  Subscribe and receive new posts by email or RSS

Connect with Andrew on Twitter or Google+