Open Question Sample Sizes

Small samples are one of the primary reasons for bad decision making in business. You complete a study, you see some type of interesting result, and you change your business model based on that result even though the sample was too small for you to make conclusions. It is for that reason that companies need to make sure that their surveys use large enough samples to get meaningful data.

Most researchers know that. Really all trained researchers know that.

But what even the most skilled researchers often forget is that when you ignore a large part of your sample for some reason – even if it’s a good reason – you become prone to small sample problems once again. This is the reason that researching based on demographics is a bad idea. You can cut down a list to Caucasian males over 50 and see how they feel about your products (for example), but if you do that your sample is going to be much, much smaller, and not large enough to make any definitive judgments.

We run into this problem once again when we code open ended questions. Most researchers take people’s answers to open ended questions and put them in categories or give them codes to help discover if there are any trends in the data. This is very useful, because it gives you data to use for analysis rather than requiring you to read each answer individually and hope to get something interesting.

But when you use this information, you may be making judgments on small samples, and small samples are not necessarily meaningful. Open ended questions are the most likely to be skipped. If you have 1,000 respondents to your survey, and only 50 fill out an open ended question, and 20 of those people have similar answers, that’s interesting and useful, but not necessarily something you should be making business decisions with.

That is not to say it doesn’t have a purpose. Indeed, when you see a repeating theme in open ended questions, you probably need to pay attention to that information. Researchers can learn a great deal from text boxes in a survey, because the data they share is data that did not require the researcher to pre-plan the question, and may show a great deal of insight that should spark further research.

But it’s still important to be wary of sample size issues with text boxes in surveys. Its useful information, but it may still be a small sample, and you need to be careful about what you do as a response to the information in the survey.