Survey Pretest Benefits

Pretesting is the way that survey researchers find if their surveys have any serious issues. The goal with pretesting is that the data will indicate if there are any problems with the survey, either because the responses will not be clear, there will be hesitation, or there will be something else within the data that tells you the survey is broken. Also, if the test takers are participating respondents, they’ll also let you know if there is anything with the survey that may cause a problem.

One can see a problem with this method right away – people are subjective measures of quality, and the data isn’t always going to indicate a problem. That is why in many research papers they advocate using sensitive testers to take the pretest survey (sensitive to what the survey should look like).

But even that may not be perfect. As described in the Public Opinion Quarterly: “Yet even if interviewers were extensively trained in recognizing problems with questions (as compared with receiving no special training at all, which is typical), conventional pretesting would still be ill suited to uncovering many questionnaire problems. Certain kinds of problems will not be apparent from observing respondent behavior, and the respondents themselves may be unaware of the problems.” Another issue comes from aspects of the survey that may not be reported.

For example, if the introduction causes a bias, if the descriptions before a section change the opinions of people, etc. Also, if the survey introduces a closed loop with its logic, the survey takers may not have any idea. Data collection can certainly help (if you have no data for a question, there’s a chance that the survey is skipping the question), but it’s certainly not a way to solve the problem.

Unfortunately, there is no direct solution. The best thing you can do is educate yourself on understanding how to write and use survey questions, how to debrief people that are testing your surveys for you, and introduce multiple eyes to see if other people see something that you don’t see.

But always be open to the chance that your survey has a problem. Evaluate and re-evaluate your survey. If you have any funding, see if you can also run the survey changing the order of the questions results in any differences in data too, in order to see if your survey length is affecting your results.

Pretest your survey, always, but also make sure that you never assume your pretest was successful in uncovering every problem.