Statistical Significance, Explained

What is statistical significance?

Wikipedia defines it this way. In statistical hypothesis testing, statistical significance is attained whenever the observed p-value of a test statistic is less than the significance level defined for the study.

In simpler terms, statistical significance is the point at which we can confidently conclude that the results of a test we are running are real, and not just a coincidence.

Why does statistical significance matter?

As marketers, we should love testing. We should test everything.

Some of the most common tests marketers do today include pricing, email subject lines, website (conversion rate optimization), and advertising copy.

Most tests are simple AB tests. We test one version directly against another, and we compare the results. But if you don’t measure for statistical significance, those results might lie to you.

For example, if you don’t have enough visitors to your website to achieve a statistically significant result, the “winner” of your test may be the winner for any number of reasons and not necessarily because of the changes you made between that version and the other.

When you do achieve a statistically significant result, you know with the utmost confidence that the changes you made directly resulted in the improvement or decrease in performance.

Here is a quick calculator for statistical significance that you can plug your test results into.