Statistical Significance and Sample Size

A test showing a promising lift on day two feels like a result worth acting on immediately, and acting on it immediately is usually a mistake. Early data in any test is noisy - a handful of visitors behaving slightly differently by chance can create the appearance of a meaningful difference that simply hasn't had enough data yet to average back out. Statistical significance and sample size are the tools that separate a real, trustworthy result from a number that just happened to look good for a moment.

What Statistical Significance Actually Tells You

Statistical significance answers one specific question: is the observed difference between two variants unlikely to be explained by random chance, at a defined confidence level, typically 95%? It does not mean the difference is definitely real in some absolute sense, and it says nothing about whether the difference is large enough to matter for the business - those are separate questions this lesson comes back to. A result reaching statistical significance is a necessary condition for trusting a test, but it isn't the same as trusting it automatically just because a platform's dashboard shows a green checkmark.

Sample Size Has to Be Calculated Before the Test Launches

The amount of traffic a test needs to reach a trustworthy result depends on the page's current conversion rate, the size of the lift being tested for, and the confidence level required - and it should be calculated before the test starts, not checked informally as data comes in. As a rough guide, a page converting at a few percent typically needs a few thousand visitors per variant to detect a meaningful difference reliably; a page with less traffic than that needs either a longer run time or a shift toward qualitative diagnostic methods, like the session recordings and heatmaps covered earlier in this playbook, rather than formal A/B testing. Calculating this upfront, before committing to a test, avoids the common failure mode of launching a test that was never going to reach a valid sample size in a reasonable timeframe.

Why Checking Early and Stopping Early Is the Most Common Mistake

Watching a test's results daily and calling it the moment a difference looks significant - sometimes called peeking - is one of the most common ways a real-looking CRO result turns out to be false. Early data fluctuates randomly enough that it will cross a significance threshold temporarily at some point in most tests, purely by chance, even when there's no real underlying difference between the variants. A result read at that moment looks like a win; the same test, run to its pre-calculated sample size, frequently shows the apparent lift regress toward nothing as more data arrives. This is exactly why setting up a statistically valid A/B test means committing to a sample size and run time before launch and holding to it, rather than treating the test as done the first time it looks promising.

Run Time Also Needs to Cover a Full Weekly Cycle

Sample size isn't the only requirement - a test also needs to run long enough to capture a full weekly traffic pattern, since behaviour on a Tuesday afternoon and a Saturday evening can differ meaningfully depending on the business. A test that reaches its sample size in three days but only covers weekday traffic risks a result that doesn't hold once weekend behaviour is included. In practice, this usually means each test needs two to four weeks minimum, even on sites with enough daily traffic to reach statistical sample size faster than that on paper.

Statistical Significance Isn't the Same as Practical Significance

A test can reach statistical significance with a lift so small that implementing it isn't worth the engineering effort or the risk of disrupting a page that's already working reasonably well. Understanding statistical versus practical significance means asking both questions before acting on a result: is this difference likely real, and is it actually big enough to matter for the business. A confirmed but tiny lift on a low-traffic page may not be worth the implementation cost, while the same lift on a high-traffic, high-value page might clearly be.

What's Next in This Playbook

With the mechanics of trustworthy testing covered, the next lesson moves to measurement at a wider level - how to judge CRO performance honestly through to closed-won revenue, not just at the point of a form fill or a test's declared win.