Testing via Question Randomization

Survey question randomizers are tools that use randomization algorithms to either place a survey question in a random place within the survey or randomize what questions the individual receives at all. Questions may be randomized individually or as blocks, and have a variety of benefits to researchers looking for a way to counter question biases and habituation. But that is not their only uses. Question randomization is also a great tool for question testing.

Your data is only as good as your survey. If your survey isn't measuring what you think it's measuring, then you're not receiving any useful results. So it's often in a researcher's best interests to essentially test their survey. You can do this with question randomization.

Order has an effect on your survey, but ideally you want to have the questions in an order that makes sense. If you're curious whether or not your survey order is having an effect on your results, you can keep one survey static and run a simultaneous survey with the questions randomized and see if the results are any different.

Another method you can try is randomizing blocks of questions, where the blocks themselves have something in common that you're trying to test. A great example is question wording. It's possible that you are unsure if the wording of your question is providing you with the right answers.

You can then create two separate blocks of questions, each with the different wording, and run them together. Respondents will not be subjected to both blocks. Rather, they'll be subjected to a single block, and you can compare the results of the one block against the results of the other.

This will help you see if there are any issues that arise from the wording, as well as see if dropout rate increases at the chosen questions.

Finally, you can also test your survey changes using question randomization, using the blocks example above. Any time you change a survey – even a single word – you may be changing the way your survey is responded to. With randomization you can run the old survey and the new survey simultaneously, and see if the results are different, how, and what you can do to account for future survey results. These are a few of the ways that survey question randomization can help you test your survey, while still running a survey at the same time.