More Baseball Data Lessons

In a previous article, we looked at how baseball is prone to fluctuations in the data, and how dangerous it is for teams to make any judgments about a player’s capabilities based on a small sample. If teams made decisions about a single month’s worth of data, they would be constantly cutting their best players in favor of players that have no business being on a major league roster. Small samples lead to dangerous data.

As a company, you have to make sure you don’t let small samples affect your decision making either. But for some researchers this can be difficult, because they don’t realize they’re working with a small sample. One of the best examples is with demographic data.

Demographic data is perhaps one of the most misunderstood parts of sample size. Say you run a survey and your sample is sufficient for analysis. Then you notice that women between the ages of 30 and 35 love your product more than anyone, so you decide to switch your marketing strategy to target women between the ages of 30 and 35.

But wait – by focusing on only that demographic, you have now cut your sample dramatically. From a sample of 3,000, you may now have a sample of only 40 or 50 – not nearly a large enough sample to draw conclusions. If you focus your marketing efforts on that demographic, you may be making the wrong decision.

Researchers make these mistakes all of the time. If the size of your demographics would not have been a large enough sample to run analyses with your initial survey, then it is not a large enough sample to make definitive judgments about what a demographic thinks of your product. If you’re curious you can explore it further, but it is simply not useful enough to draw conclusions

Unless you are positive that you are not going to be prone to these types of errors, you may simply want to leave that information off the survey altogether. Demographics themselves may be interesting in a large sample, but parceling the data further will far too often make the sample too small to make judgments, and as we’ve seen from the baseball example, making decisions based on these small samples is almost always going to lead to bad decision making. Related Blog Part 1