A/B Testing in Product Analytics: Setting Up and Analyzing Experiments

Why A/B Testing in Products Differs from Marketing Tests

Marketing A/B tests (landing pages, email subject lines) typically have large sample sizes and short test durations. Product A/B tests often deal with lower traffic volumes, longer user journeys, and metrics that take days or weeks to fully manifest (especially retention and revenue). This makes product experimentation more statistically complex and more prone to premature conclusions.

Defining Your Experiment

Every experiment needs four things defined before launch:

1. Hypothesis: If we do X, metric Y will change by Z because of mechanism M
2. Primary metric: The one metric that determines whether the experiment won or lost (usually a conversion or activation metric)
3. Guardrail metrics: Metrics that must not regress — even if primary metric improves, the experiment fails if guardrails are violated (e.g., don't improve activation by breaking a downstream feature)
4. Sample size and duration: Calculate before starting using a sample size calculator

Sample Size Calculation

Before running any A/B test, calculate the required sample size. You need to know: baseline conversion rate, minimum detectable effect (the smallest improvement that's worth shipping), statistical significance threshold (usually 95%), and statistical power (usually 80%). Run a sample size calculation to determine how many users each variant needs before you can trust the results.

For product experiments with conversion rates around 5-20%, you typically need thousands of users per variant to detect 10-20% relative improvements with confidence. If you don't have enough traffic, you can't run meaningful A/B tests — focus on qualitative research instead.

Running the Test

Use a feature flagging tool (LaunchDarkly, Split.io, or built-in tools in Amplitude or Mixpanel) to assign users to variants. Assignment should be random, based on user ID (not session ID), and stable — the same user should always see the same variant throughout the test. Track the assignment event in your product analytics alongside all subsequent events so you can segment analysis by variant.

Analyzing Results

Wait until you've reached your pre-calculated sample size before looking at results (or run Bayesian sequential testing if you must peek early). Check primary metric, guardrail metrics, and segment results by user cohort (new vs. returning, plan type, acquisition channel) to check for heterogeneous treatment effects. Use BigQuery for the statistical analysis if your product analytics tool's built-in analysis is insufficient.