2 Types of Survey Pretesting Samples

Pretesting a survey is designed to catch errors. Sometimes these errors can be seen in the data. Other times the errors can be reported, when a respondent tells you that the question was unclear or notices that the question can be misunderstood.

In general there are two (arguably three) types of samples that you can choose from: Undeclared Undeclared pretesting involves using a sample that has no idea they’re being sampled for a pre-test survey. The idea is that they will take the survey as though they are normal respondents, and you will be able to view anything you weren’t expecting with the data. This method of collecting data is useful, because it helps you catch some errors that may not occur if the respondent is motivated to complete it in order to help you find flaws in the survey (for example, problems with habituation).

However, in many cases it’s hard to know what’s a problem with the survey, and what’s the actual data from the respondent. Participating Participating pre-tests involve a sample that knows they’re taking part in a pre-test. These individuals take the test and pay attention to any issues, and then they report these issues to you at the end of the survey.

These can be far more useful for finding out when a question is confusing and if it may lead to incorrect results. This method uses the same general sample (customers, employees, etc.) but does tell them ahead of time that they are part of a pretest. You may also want to consider using researchers and experts in survey methodology to pretest your survey as well, because they may be able to notice issues that have occurred in their own experience.

This type of pretest is useful for catching errors with leading questions, how a question is supposed to be understood, etc., but it is not the best way to find out if the survey is too long or if the order the questions are asked affects their data, and other test taking issues.

Both types of pretests have their own benefits and weaknesses, so if you have the financing available, it’s a good idea to start with a participating pretest, then follow that up with an undeclared pretest once some of the issues have been fixed. However, experts argue that a participating pretest may be more important, because participants may spot errors that are not as obvious in the data.