Pre-Defining Survey Expectations

Researchers highly recommend that before you collect data you have a hypothesis. There's a type of data analysis that involves collecting large groups of random data and then mining the data for items that have meaning, but in any large survey it's possible to have results that look meaningful, but are really an effect of random chance. That's why it's often recommended that you have some idea of what you're trying to discover before you run the survey. Using your knowledge about survey data, you may want to consider setting parameters for what is a "success" or what is meaningful to changing your business.

Let's say you are considering the idea of adding a new feature to your product but you don't want to pre-define expectations. You run a customer survey and find that 30% of all customers have a very favorable view of the new feature. What does this tell you? You may decide that it was a success, but what were your expectations? What number did you want to reach to consider it a success? At this point, your decision would likely be subjective, based on whether or not you wanted to complete it in the first place.

Now, let's say you had planned beforehand that only if the number reached 50% would you consider it a success, based on your knowledge, expectations, and the cost of implementing it. Since only 30% were very favorable, you would choose not to install the feature. Your answer would be made for you, and there would be no subjective considerations when deciding whether the feature is worth implementing.

This method may not work for everyone. You may not have any idea what would be a success or failure before you start the survey. Maybe you find that 0% is very favorable but 100% is favorable? Had you defined a success by very favorable and stuck with it, you wouldn't have initiated the change, despite it clearly having value.

If you have little survey experience or expectations, choosing to have set numbers beforehand may not work either. But it is something to consider, especially if you hope to take the bias out of analyzing the results of your survey. It's not always a good idea to try to decide what you want to do only after you have received the data.

It may be better to have an idea beforehand and see if the results indicate you should make that change.