Combining Survey Results

Giving yourself a larger sample from which to draw conclusions can have a lot of uses. Problems with sample size are the boon of even the best of researchers, and even if you have enough of a sample to consider your research results conclusive, a larger sample is still preferred for a greater level of accuracy and confidence. So when you have two surveys run presumably simultaneously (or close) that share some questions (although not all), you may be tempted to combine those results into one larger dataset and using that larger set to draw your conclusions.

There are ample times when this may occur, but an example in business would be running two separate surveys about different features of a potential new product, while also including general customer satisfaction questions in the survey (perhaps as a baseline) that are identical. You may want to calculate the mean of all of the satisfaction scores separately for a different type of analysis.

There are several problems with simply combining data and assuming it is useful for analysis. A few include: It’s possible that multiple respondents have completed both surveys, which means that their data is doubled and affects the survey results. It’s possible that the answers to some questions, or the layout, or the introduction, or the way the survey was conducted, the length of the survey, etc., may have changed the way respondents answer otherwise identical questions. The idea that one can take results to identical survey questions and combine them is a nice one, but from what we know about research, there is no way to guarantee that the data in the larger sample is going to be more accurate when you are using multiple sources to analyze it.

The International Survey of Statisticians released a useful document about how to effectively handle these samples. Ultimately you will have to decide between pooling samples or combining samples, while correcting for auxiliary issues with the data. When the right adjustments are made, the sample may be able to provide you with an accurate result. Using multiple studies with similar questions and combining the data is a nice thought, but it does take some additional analysis to ensure that you are accounting for problems that may occur from this method.