Defending Survey Results

Recently we've had several articles about the need for better questionnaire design, and a better focus on generating quality data and logical conclusions. Far too many news outlets and researchers are going about their data collection incorrectly and disseminating the results without the proper amount of oversight. These organizations cannot actively defend their work, and refuse to do so.

That's why we were pleased to see the following article released by ABC News . The polling firm was panned by critics for the results of a recent political survey. While some companies simply ignore the critiques, ABC News decided to fight back, explaining how and why all of their survey methodology is valid and should be backed as a valid questionnaire.

Some of its strengths: It uses previously released data to show how and why there are reasons to trust the results of their current study. It randomized the questions and the question wording to make sure that order was not creating bias in the results. It explained why each question was chosen and how it was asked in a logical and unbiased manner.

ABC also gladly releases their datasets and questionnaires to the public, so that if others want to argue the question wording that can do so easily. Regardless of your thoughts on the actual article, it does act as an important reminder – the company had a clear and identifiable response to scrutiny. They were willing and able to back up and justify all of their decisions – something that many other research firms cannot and/or will not do.

That shows dedication to the research, and while it's still possible that they did introduce some biases or that their data isn't meaningful in the end, at least they can stand behind it knowing that were not reckless with any aspect of their data collection. Any company that engages in any type of research, no matter how trivial, should be able to do this. Because as a company, if you cannot defend your survey against scrutiny, then your data has less value, because it's possible you introduced problems that make your conclusions completely inaccurate.

Any time you decide to run any sort of study, make sure that every decision you make can be defended. You can also contact us about your questionnaire design, and allow us to provide you a consult to make your survey defensible.