Survey Confidence Explained

We’ve had several articles of late about confidence intervals (aka “error margins”) and how they can often be incorrect interpreted or misunderstood when it comes to analysis. Yet they are one of the most important parts of survey research, because they are providing you with a range designed to allow you to say, with confidence, where the data of your overall population is likely to lay. But there is one other error that can occur with understanding confidence intervals, and that error can be introduced when you created your survey.

Imagine you ask a market research question about a specific feature of a product. You run the research with a large sample, and receive an overwhelming response that the feature will be well received, with a very low margin of error and 99% confidence. You then release the feature and it flops considerably.

What could have gone wrong? It’s possible that you were unlucky, and the results fell into that 1% lack of confidence. It’s possible that the needs of your customers changed by the time the feature was released.

It’s possible that you ran the math wrong on your survey (unlikely). But it is also possible that the problem was actually within your survey. It’s possible that the question you asked was worded in such a way as to provide a different result than the actual feelings of the population.

Researchers often forget that the error margins and confidence interval calculations are all related to the question, not necessarily the true feelings of the community. That is, with XX% confidence, the researcher can say that their population will answer the question in that specific way.

However, if the question isn’t doing a good job of asking what it was intended to ask, then the results you’re receiving don’t have any real-world practicality.

The benefits of your analysis are limited to the ability for the question to reflect the true feelings of those you’re surveying. Often it does, when the question is clear enough to the consumer, and they can easily picture what you’re asking in order to respond accordingly. But on occasion, the question is not measuring what you think it is measuring, and as a result the results of your survey are going to be different than what you expect when you make decisions based on your survey results.