Type I Error vs Type II Error: What's the Difference?

A Type I error, or false positive, is concluding a variant made a real difference when it actually didn't - declaring a test result a winner when the outcome was really just chance, and shipping a change that does nothing for the business.

A Type II error, or false negative, is the opposite mistake - failing to detect a real effect that genuinely exists, calling a test inconclusive or a tie when the variant actually was better, and leaving a real improvement unshipped.

The probability of a Type I error is set directly by the significance threshold chosen before the test runs - at the common 0.05 level, roughly 5% of tests on a truly ineffective change will still falsely show a winner purely by chance. The probability of avoiding a Type II error is called statistical power, with 80% power - accepting a 20% chance of missing a real effect - as a common target. The two are in direct tension: tightening the significance threshold to cut Type I errors (requiring stronger evidence before declaring a winner) makes it correspondingly harder to detect real effects, raising the Type II error rate, and loosening it does the reverse. The one lever that reduces both simultaneously, rather than trading one off against the other, is a larger sample size - which is why underpowered tests, run on too little traffic to detect anything but a very large effect, are one of the most common sources of unreliable A/B testing results.