Validating Survey Data Quality

Often we talk about why it is important to cut down on questions in order to reduce survey length. Long surveys make dropout more likely, and also reduce the amount of time that respondents spend thinking about each question. Ideally your survey is as short as possible while still providing you with a relevant amount of data.

But there are some cases when adding a question or two is a good idea, and can be used as a way to ensure that the data you are collecting is accurate. An example is with questions that are commonly referred to as “check questions.”

While this term changes depending on the researcher, a check question can best be described as a question that ensures that the data you just collected is going to be accurate. Let’s use the following example: Say you add an audio file to your survey, and you ask questions based on that audio file. You then find, however, that depending on the browser, speakers, etc., the audio may be either corrupted or play poorly, making it difficult for the respondent to hear.

When that happens, there is a good chance that the questions they answer are not going to be as accurate as the questions answered by someone with access to the full and complete audio file. You may not know if you want to exclude this from your data, but you do know that you want to have a record of the respondent’s experience so that you can know if it is worth using their data in the dataset. You can do this by adding one of these questions, such as: Q: Did the audio file play correctly? Now, if the person was unable to hear the audio file, you’ll know.

You can then adjust your dataset accordingly or keep their data if you find that the information is still useful without it. You can also use it to compare and contrast browsers, so that if someone doesn’t answer that question, but seems to have the same computer/browser as someone else, and their data looks like an outlier, you’ll have a sign that they may not have heard the audio file correctly. These types of check questions still add length to the survey, but they can be very valuable in ensuring that you are getting the right data into your dataset.