A/B testing

In short

• An A/B test shows version A to some users and version B to others, at random, and compares one chosen metric.
• Decide the hypothesis, the metric, and the sample size before the test starts.
• Tests need a lot of traffic. Most early products learn faster by talking to users.
• Stopping a test as soon as one version looks ahead is the most common way to get a wrong answer.

Also called: Split testing, AB testing, Experimentation, Multivariate testing.

How an A/B test works

Version A is usually the current design, called the control. Version B, the variant, changes one thing: a headline, a price layout, the order of steps in sign-up. Visitors are assigned at random to one version and stay in it. Because the only systematic difference between the two groups is the change, a large enough difference in results can be put down to the change.

What makes a good test

• A clear hypothesis. For example: showing real availability before sign-up will increase completed bookings, because people see value first.
• One primary metric. The number the decision depends on, such as completed sign-ups. Track a few others only to make sure nothing breaks.
• A planned sample size. Calculated before the start from your current conversion rate and the smallest improvement worth detecting. Free calculators do this.
• Whole weeks. Running for complete weeks avoids results skewed by weekday and weekend differences.
• A meaningful change. Small tweaks need enormous traffic to measure. Bigger changes show clearer results.

When not to A/B test

A/B testing needs volume. A page with a few hundred visitors a week and a low conversion rate can take months to produce a reliable answer, if it ever does. For most early products, that time is better spent elsewhere.

Five people trying a new sign-up flow will often show you the problem in an afternoon. A test is the right tool once the product has steady traffic and you are choosing between two reasonable options. In the last week of a Choie build, the product goes live and Choie watches real people use it, fixing what gets in their way. A problem that plain does not need a test.

Tools and setup

Many product analytics tools now include experiments and feature flags, and dedicated platforms exist for larger programs. Feature flags let engineers ship both versions in the same release and switch who sees what without a new deployment, which is useful on mobile apps where releases go through store review.

Common places to test once traffic allows are the landing page headline, pricing page layout, sign-up steps, and onboarding. On a live product, Choie's ongoing monthly work takes one priority at a time, which makes it easier to change one thing and measure the result.

Common mistakes

• Stopping early. Ending a test when the result looks good, before the planned sample size.
• Testing too many things at once. Five changes in one variant leave you unsure which one mattered.
• Picking the metric afterward. Looking at twenty metrics will turn up a winner by chance.
• Ignoring novelty. Returning users sometimes click on anything new, then settle back. Longer tests reduce this.
• Testing with too little traffic. An inconclusive test is not a sign that the two versions are equal.

Common questions