Why Satisfaction Data Fluctuates

When you have a large sample and collect data over time, you expect to see a flow of results. While it may go up or down, the amount it changes should not be very high, and your hope is that you see changes occur gradually in the direction you’re hoping to reach. When you see a big jump, however, it could still be good news.

Great satisfaction programs, for example, could result in a big jump in data. But what happens if you run the survey again later, and the result is back to where it started – far more than just regression would have you expect? What could have been the cause of the blip in the data?

Date Dates can affect the data. Surveys run immediately after employee bonuses are paid are going to have different results than surveys run in the middle of the year. If you chose a date that coincided with some big change in your company, you may have inadvertently biased the results.

For example, if you recently finished a big employee event, satisfaction levels are likely going to be much higher than if the survey was run during the more “regular” part of the year. Statistical Noise It’s possible, even with large samples and controlled questions, to still have statistical noise cloud your results. It is the same reason that scientific studies at major universities are expected to be replicated, and why most analyses use confidence intervals – it is still always possible that your survey yielded results that are not an accurate reflection of the population.

Satisfaction Adapts While not that common, satisfaction scores themselves to do not always have direct meaning. For example, if your employees tended to score a 7 in satisfaction, and then you initiate fantastic employee satisfaction efforts, run the survey, and get a 9, that’s a successful change.

But over time, employees will get used to being satisfied, and may go back down to a 7 the next time you run the survey. They are still equally as satisfied, but they have a new “baseline” of satisfaction. It means that as score of 7 is more valuable, but it also means that it has a different meaning than it had when you first ran the survey.

The above list represents just a small fraction of the possible reasons for a blip in your data. If you see such a blip, you shouldn’t let it dishearten you. Instead, you should continue your efforts and explore the blip further to see if there is anything interesting you can learn from the result.