Questionable Survey Methods in News

Continued from Part 1 An unsophisticated forecaster uses statistics as a drunken man uses lamp-posts – for support rather than for illumination. ~Andrew Lang Sometimes the problem with a survey isn’t the survey itself. Sometimes the problem is the how the results are utilized. It’s possible to create even poorly designed surveys that can provide you with interesting or relevant information, but it has to be made known that those results may not be applicable to a wider population, or useful enough to draw broad conclusions.

Several recent news stories were released about findings from the National Sleep Foundation. A large number of respondents were reporting unhappiness with their total amount of sleep, along with considerable technology use before bed. The results of the survey were published by the foundation as relevant, making several claims including: People are using more technology and getting less sleep.

Technology may be causing more sleep deprivation. These claims would be interesting – if they were accurate.

However, experts claim that the data is fundamentally flawed.

Previous years, the National Sleep Foundation used a phone survey and a randomized sample to collect their data. This year, they changed their methodology. Instead of the same methodology, they foundation also used “online opt-in sampling” to the study.

Not only that, but they only used that type of sample for half of the respondents – meaning they collected data through two different methods, one of which is a severely flawed way of getting data that can be considered applicable to the population at large. From the article : "There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population," the report stated. The National Sleep Foundation appears to have later admitted that there may be some issues with the methodology, but not after several news reports were released about how technology “is” causing poor sleep.

The likelihood that readers of those articles also read articles about criticisms of the study is slim. If this issue illustrates anything, it is that poor survey methodology can have greater implications than simply how they affect your own business. Flawed surveys create flawed “facts,” because today’s society and journalists rarely check to ensure the information they’re reporting is actually information, and not the result of a company’s willingness to release data that may not be representative of the larger population.