Should You Split Test Your Surveys?

Even the best researchers sometimes make the mistake of assuming a very small change in their survey will not cause a noticeable difference in the results. The truth is that respondents are often hard to understand, and many of their answers can be changed by even the slightest alteration to your survey. Something as minimal as a font change has the potential to drastically alter their answers, and while not every change is going to cause these differences, researchers need to be aware that it is possible for even the most minor of changes to cause considerably different results.

That is why when you have two different surveys – either because you are uncertain which version to use or you have changed an old survey – you may want to consider split testing. Split testing involves running both surveys simultaneously and comparing results to discover if there are any differences within answers and/or which survey provides more accurate results. Split testing has several benefits: It allows you to make sure that a change did not alter the responses.

If the responses are the same for a question between two surveys and your sample sizes are both large enough, it can be assumed that the change did not have any pronounced effect. It allows you to see if one survey is more effective at collecting data than another survey. There are several types of analyses you can employ to compare survey responses and find out if a survey is better.

It allows you to spot differences before you draw conclusions. If two surveys with similar questions appear to have very different results, you can figure out why the results were different and make changes accordingly. Some researchers also use split surveys to test drop out and response rates, since survey design can affect how many people are both taking your survey and completing it.

Split testing can be very beneficial when you have more than one type of survey, but it does come with some caveats. First, it is obviously much more expensive, as you need to have large samples for both surveys running independently. Second, it may be difficult to gauge whether an answer in one survey is more reliable than an answer in another.

If you ask identical questions on two surveys, and you get different results, it can be hard to know which survey’s results are more accurate. Third, the surveys must be run simultaneously. If you run the surveys in separate time periods, time as a variable may play a role.

Still, split testing can be a very important and very useful way to test two surveys. It’s costly, but those that can afford it will be much less likely to draw inaccurate conclusions.