Bad Polling and Bad Survey Research

As a survey research company, we often see examples of bad data collection and research. It is, unfortunately, everywhere. The mainstream media, blogs, and even research journals far too often report on data that is scientifically inaccurate, either because the data was collected incorrectly or the conclusions that they draw from that data are ridiculous.

So it's great to see journalists finally calling out these organizations that are reporting bad data – a practice that should probably happen more often since it makes a good story. Here's an example in the Kennebec Journal. The article discusses three different polls/surveys, all released in just a few weeks' time, all with considerable methodological errors.

The first survey was conducted by a bias research group and a sample size of only 48 people about a political issue that the group was committed to fighting. They did not release their questions, or the demographics, or any other information about how the survey was conducted, and the dataset has been kept a secret from the public.

The second survey was an internal study about the election campaign of a Maine Republican, hoping to show how they were likely to unseat the Democratic incumbent. The political campaign conducted the survey in house, and supposedly only released a small subsection of information with few additional details.

The journalist showed something else with the third survey, and was commenting more on the state of the media than one the survey itself. The survey was large, conducted generally correctly, and released all of its information. While it may not have been a perfect survey, it was well conducted and contained valuable information with well worded questions. By the media, it was used as only one small additional data point in a sea of worse research, as though it was equal to the other studies.

It's the fault of both the media and the organizations that release this research that so many people are being led astray in their beliefs. It's good to see articles like this that discuss the importance of better quality research, and it's important that as a company you make sure that you're not contributing to all of this incorrect data. Also, be critical of everything you read unless you see evidence that it was conducted properly, and if you see an article or a study with methodological flaws or incorrect conclusions, contact us and we'll dedicate an article to it.