How to Pre-Test a Survey Questionnaire

Pre-testing a survey is the only way to make sure that it is going to deliver the data you were hoping to receive. While there may not be such thing as a survey without a single response error, there are ways to make sure that respondents are providing the responses you expect, and that the survey is working correctly. Many survey researchers fail to pre-test their surveys.

But without pre-testing, it is difficult to find out if your survey has any logic problems, if the questions are too hard to understand, if it is prone to central tendency and habituation, if it has too high a dropout rate, or if it has any response bias. Using professional survey software with built-in analytics makes pre-testing much easier. There are many issues that affect the quality of your data, and pre-testing is the only way to make sure that your survey is getting you as close to perfect answers as possible.

Researchers pre-test for a variety of reasons. At its core, pretesting is designed to make sure that people understand the questions, and that there isn't anything in the data that indicates the information is inaccurate. This can be a difficult process – how can you know if the answers you're receiving are accurate? What may seem like an anomaly may be honest answers, and what may look like sensible answers may be incorrect responses. In general, researchers look for the following warning signs in pre-test data:

1. Skipped Questions – One of the most telling signs that a question may not be understandable is when it's skipped. Questions that are confusing or unclear are the most likely to be skipped on a survey.
2. "Don't Know" Answers – Another common response when a question is unclear is the "don't know" or default answer. If respondents are unclear what a question is asking, they may select the response that most closely resembles "don't know" or "neither agree nor disagree."
3. Data Differences by Position – If identical questions get different responses based on where they appear in the survey, something may be wrong. For example, a question that shows up late in the survey getting a different response when it shows up early.
4. Hesitation Times – Some survey platforms track the time it takes to answer each question. Long pauses may indicate respondents are struggling to understand or answer.
5. Debriefing Respondents – It's common to debrief a small number of respondents before sending the survey to your full sample. Ask them directly if they had any issues with question clarity or the survey flow.
6. Question Type Testing – Try different question formats asking the same thing. If you get different answers for different styles of questions, one format may be clearer than another. Beyond these six methods, having someone trained in survey research review the questionnaire can catch issues that data analysis alone won't reveal. A confusing question is often obvious to an experienced eye.

While pre-testing is important, no method is perfect. There's an inherent assumption that problems will reveal themselves, and this simply may not always be the case. Researchers should be careful about believing too strongly that all survey problems have been addressed after pre-testing.

However, following best practices and having experienced researchers review your questionnaire will go a long way. Pre-testing remains an essential part of survey research. Without it, problems may go unnoticed, and you risk wasting time and resources on a flawed survey.

The good news is that modern survey software makes pre-testing easier than ever – with real-time response tracking, skip logic testing, and built-in analytics to help you catch issues before your full launch.