Facts From Bad Surveys

Bad research is everywhere in our society. One of the most famous examples is the Dust to Dust survey by CNW Market Research. It was the famous study that concluded that dollar per dollar, the Toyota Prius Hybrid costs more per mile than the Hummer H3 gas guzzler.

Nearly everyone has heard of this study, and many people still believe it’s true. However, the research that went into this study was flawed in many ways: The researchers estimated the Prius would die after 100,000 miles (many last beyond 300,000). The researchers admit this was nothing more than a guess – they had no evidence that cars only last this long.

The researchers estimated that the Hummer would last 300,000 miles (many last less than 100,000). Once again, this was based on a guess. There was no research to back up this claim.

The researchers included the costs of research and development (the Hummer used research conducted by other companies while the Prius involved technology recently invented), and then it divided that number by the total number of cars on the road (the Hummer has existed since the 1980’s). There are so many flaws with this research design that it’s become a laughingstock in the scientific community . Yet despite these flaws, this study is widely believed to be true, primarily because it is often cited by those against the use of Hybrid cars and presented as “fact.”

Many of these same problems occur with bad survey research. Often researchers introduce questions that alter the answer – sometimes this is done on purpose, sometimes by accident – or they use samples that are not representative or not chosen at random. Then the findings of these surveys get released to the media, and quickly become a popular topic, with news reports of some of the findings of the studies without any additional details given that would highlight these flaws. Quickly this information becomes “fact” in popular culture, leading to poor decision making and an ignorance of truth.

This brings up an interesting question. Are the researchers to blame for providing these false results? Is the media? It’s a question that cannot necessarily be answered definitively, but it’s clear that the abuse of research is a serious problem.

Researchers need to be more ethical when it comes to how they conduct their surveys, no matter how badly they want a specific result, but the media needs to hold researchers accountable rather than look for their next big hook. The public at large cannot be expected to critically think about surveys they have no information about. The media and researchers have a responsibility to make sure the information being released to the public is accurate, and when either fail they need to be held accountable.

Continued