False Positives in A/B Testing: Why Tests Win Then Fail

The Test Graveyard

Many CRO programs have a test graveyard: a list of "winning" tests that were implemented and then showed no measurable improvement (or even degradation) after implementation. This is demoralizing and leads organizations to question whether A/B testing works at all.

It works — but false positives from methodological errors are common and predictable. Understanding them helps you design better tests and interpret results more skeptically.

Why False Positives Occur

1. Peeking and Early Stopping

The most common cause. When you check results daily and stop the test as soon as p < 0.05, you've inflated your actual false positive rate dramatically.

Why: Statistical significance fluctuates during a test. In any sufficiently long test, the p-value will cross the 0.05 threshold at random points — even when there's no true effect. Stopping at the first crossing captures that random fluctuation, not a true signal.

In simulation: stopping 10 sequential tests at first significance with no true effect → 30%+ are "winners." Your 95% confidence level is actually ~70%.

Fix: Set end date before starting and do not check results until then. Or use sequential testing frameworks designed for early stopping.

2. Multiple Metric Testing

Testing many metrics simultaneously without correction. If you run a test and check 10 different conversion metrics, you'd expect 0.5 false positives by chance at p < 0.05.

Fix: Pre-specify one primary metric. Treat secondary metrics as exploratory. Apply Bonferroni correction if you must evaluate multiple primary metrics.

3. Insufficient Sample Size

Underpowered tests (insufficient sample) can reach significance through random chance — particularly if variance in conversion rate is high. The "win" reflects noise in a small sample, not a true effect.

Fix: Calculate required sample size before starting and collect it fully before analyzing results.

4. Seasonal and External Effects

A test that runs during an abnormal period (sale event, news story, competitor outage) can produce results that reflect the period, not the treatment. Variant B might "win" simply because a significant promo email happened to go out during the variant B period.

Fix: Run tests across full business cycles. Watch for anomalies in traffic patterns during test periods.

5. Novelty Effect

Returning users may engage more with new page elements simply because they're new — not because the elements are better. Novelty effect typically fades after 1–2 weeks.

Fix: Run tests long enough to see novelty effect normalize (minimum 2 weeks). Segment analysis by new vs. returning visitors — if "winners" are driven entirely by returning users clicking the new element, suspect novelty.

Regression to the Mean

A different phenomenon: tests that show extreme results (very high or very low conversion rate during the test period) tend to revert toward average over time. If you caught the variant during an unusually good period, its "win" overstates its true effect.

Fix: Look for the effect size to be consistent throughout the test, not just in a particular window.

How to Validate Before Full Implementation

For high-stakes tests (major design changes, pricing changes), run a post-test validation:

1. After calling the winner, run the winning variant as an "A/A test" (comparing itself to itself in two segments) for 1–2 weeks to confirm the baseline is stable
2. Run the test again on a different time period to replicate results
3. Implement in phases (10% traffic initially) and monitor real conversion rate against pre-test baseline

Connect your testing platform to GA4 for ongoing post-implementation monitoring. Our CRO team designs statistically rigorous testing programs that minimize false positives. Contact us to audit your testing methodology.