What a High Response Rate Doesn’t Tell You

Response rates are an important – and often forgotten part of survey research. For your research to be valuable, you have to try to get as high a response rate as possible. A “random sample” assumes that randomly choosing from the population will get you an approximation of what the public believes.

But if you don’t have a high response rate, there may be something in common with those that haven’t filled out the survey that prevents you from getting relevant data. For example, retail companies that put surveys at the end of their receipts are more likely to get negative responses than positive responses, because those that had a “fine” experience have no incentive to fill out the survey. Similarly, perhaps you field a telephone survey and don’t call cell phones – youth are unlikely to answer landline phones and may be unwilling to respond to your survey anyway, so your response rate will decrease and you won’t have an important demographic included in the results.

That’s why a high response rate is so important. The higher your response rate, the more likely you are receiving accurate information from your sample – and the less likely those that do not respond have something in common that affects the accuracy of the results of your research.

Some researchers, however, take response rate a bit too far. They look at a high response rate as a guarantee that they are getting accurate answers.

But there are things that response rate doesn’t tell you. For example: Successful Randomization If your sample wasn’t successfully randomized, then the response rate doesn’t have as much as an effect. It’s still better to get a good response rate than a bad response rate, but if there are individuals missing from your sample that have something in common, then a good response rate isn’t going to find them.

Incentive Bias What you use to get that high response rate may also impact the answers you receive. For example, a person that is excited to get your incentive may rate your company higher than a person less excited about the incentive. The language you use to convince people to respond may impact their opinions as well.

Employees at a company that requires a survey may be too motivated to answer positively, and someone that takes a survey because they feel they have to may not be incentivized to read questions in full. There are countless ways that you, as a researcher, can affect the answers of your survey based on how you attract the high response rate in the first place. Poor Questions One of the most common problems with data has nothing to do with the sample at all.

It has to do with the questions. Poorly written questions – or questions that do not provide the answers the researcher was looking for – are going to create meaningless data. Even the best response rate cannot make up for those data issues.

Data Collection Issues Similarly, there may be issues with the data collection itself. AAPOR writes, “Self-reported stress and depression were quite high immediately following the terrorist attacks of September 11 th , but declined rapidly in the following two months. A survey that took two months to complete would have missed that trend.” Even with a high response rate, the data may not have been meaningful.

Anything that impacts your data collection efforts and the accuracy of that data is going to put your analysis at risk. Data Analysis Problems Finally, even the best data is meaningless without those that are qualified to analyze it. All of the data in the world is only as good as the analyst, otherwise you may be drawing incorrect conclusions or missing out on something important simply because your surveys were not correctly analyzed.

Similarly, the value of response rate is relative to the randomization of those that complete the survey. If the responses you receive are still random (i.e. those that were more or less likely to fill out the survey were roughly as randomly selected as the survey sample itself) then the response rate should be irrelevant. The problem with response rate is that those that do/don’t fill out the survey tend to have something in common. If they have nothing in common, then the response rate is unimportant provided you’re able to attract at least the minimum to be confident in your analysis.

None of this is designed to decrease the importance of a good response rate. A high response rate is important and will always be important.

But if you focus too heavily on response rate, you may forget other very important features of well-designed surveys. Survey best practices are extremely important, and without them your data becomes far less valuable. If you want to learn more about surveys or you’re ready to get started with your survey research, sign up for an account at SurveyMethods or call us today to find out more about our survey research services.