It is, and always will be, possible to find data that supports your theory. Data is not perfect. And though many of us treat it that way, it is this imperfection that will end up tricking us into doing things that aren’t to our benefit.
For example, if I want to prove my point that a specific page on your website needs to be updated, I could show you the page’s bounce rate, exit rate, or conversion value. I might show you that the bounce rate for your product page has gone up 20% over the last 30 days. So clearly something is wrong.
But what if during that time the traffic sources also changed drastically? Or perhaps the bounce rate on the entire site went up by 20% because of some change in consumer behavior we haven’t factored into our analysis.
Successfully data scientists and analysts get paid good money for a reason. It’s because they are able to tell when the data is lying to them. They have trained themselves to do everything possible to keep their own bias and subjectivity out of their work. They let the data speak to them instead of starting off with a guess about what the data will say.
The best analysts know that not all metrics are created equal. They weed out the metrics that don’t matter to find the ones that truly show us what’s happening with the business. For some companies, the average pages per visit on your website might matter a great deal. And for others, we could care less.
It’s about identifying the right data for the right purpose.
Trust the data. But that doesn’t mean you shouldn’t question it. The key is to make sure you are trusting the right data.