Writing a Real, Testable Hypothesis

"Let's try a different headline" is not a hypothesis, no matter how confidently it gets proposed - it's a guess that happens to have a test attached to it. The diagnostic work from the previous lesson exists specifically to prevent this: a real hypothesis connects a diagnosed problem to a proposed fix and a measurable, predicted outcome, and writing it out explicitly, before any variant gets built, is what separates a test worth running from one that's really just an opinion with extra steps.

The Four Parts a Real Hypothesis Needs

A properly structured hypothesis names four things explicitly, in a format close to: "Because [diagnosed problem], changing [specific element] to [specific change] will result in [predicted outcome], measured by [specific metric]." Each part does real work. The diagnosed problem ties the hypothesis back to actual evidence from heatmaps, session recordings, or funnel data, rather than a hunch. The specific change and predicted outcome make the test falsifiable - it can actually fail, which is what makes a positive result meaningful. The specific metric removes any ambiguity about what counts as success before the test starts, which matters because it's much easier to find a metric that happened to improve after the fact than to predict the right one beforehand.

Why a Vague Hypothesis Wastes the Traffic Spent Testing It

Every test consumes real traffic and real time - typically two to four weeks to reach a valid sample size, covered in more depth later in this playbook. A vague hypothesis risks spending that traffic testing a change that was never connected to an actual diagnosed problem, which means even a positive-looking result doesn't reliably explain why it worked or whether the underlying issue is actually fixed. A specific hypothesis, by contrast, produces a result that's interpretable either way: if the checkout redesign doesn't move checkout completion, that's informative too, because it rules out a specific explanation for the original drop-off and directs the next round of diagnosis somewhere else.

One Variable at a Time, in Almost Every Case

A hypothesis should generally test one change against a control, not several at once. Changing the headline, the button colour, and the layout simultaneously in a standard A/B test makes it impossible to know afterward which specific change actually drove any difference in the result - a win could be entirely attributable to one element while the other two did nothing or even worked against it. Testing several elements deliberately, to understand how they interact, is what multivariate testing is built for specifically, and it requires meaningfully more traffic to reach a valid result than a single-variable test, since it has to isolate the effect of more combinations.

The Success Metric Has to Match the Diagnosed Problem

It's tempting to judge a test by whichever metric looks best afterward, but the metric has to be decided before the test starts, and it has to match the problem the hypothesis was actually built to solve. A hypothesis aimed at reducing cart abandonment should be judged on cart-to-checkout completion, not on a secondary number like average session duration that could shift for unrelated reasons. This is also where the distinction between micro-conversions and macro-conversions matters - a micro-conversion like a button click can be a useful diagnostic signal, but the hypothesis's actual success metric should usually be the macro-conversion the whole funnel exists to produce, not a smaller proxy that can move without the real goal moving with it.

Watching for a False Positive Before Trusting a Result

Even a well-built hypothesis can produce a misleading result if the testing setup itself is flawed - a tracking error, a segment imbalance between variants, or a test that simply hasn't run long enough. Running an A/A test occasionally - splitting traffic between two identical versions of a page - is a useful sanity check on the testing infrastructure itself, since a meaningful difference between two identical variants signals a setup problem worth fixing before trusting any real A/B test result that comes out of the same system.

What's Next in This Playbook

With a properly structured hypothesis in hand, the next lesson applies this directly to landing pages - how to structure, write, and test a page against a single conversion goal, rather than one page trying to serve several competing purposes at once.