# A/B testing

A/B testing is comparing two versions of a page, screen, or feature by showing each to a random half of users at the same time, then measuring which one performs better on a goal chosen in advance.

Also called: Split testing, AB testing, Experimentation, Multivariate testing.

## In short

- An A/B test shows version A to some users and version B to others, at random, and compares one chosen metric.
- Decide the hypothesis, the metric, and the sample size **before** the test starts.
- Tests need a lot of traffic. Most early products learn faster by talking to users.
- Stopping a test as soon as one version looks ahead is the most common way to get a wrong answer.

## How an A/B test works

Version A is usually the current design, called the control. Version B, the variant, changes one thing: a headline, a price layout, the order of steps in sign-up. Visitors are assigned at random to one version and stay in it. Because the only systematic difference between the two groups is the change, a large enough difference in results can be put down to the change.

**Diagram: The steps of an A/B test.**

1. Hypothesis (What you change and why it should help)
2. Metric and size (One goal, a sample size, a run time)
3. Run (Random split, no peeking)
4. Decide (Ship it, drop it, or learn and retest)

_The decisions made before launch protect the result. Choosing the metric afterward almost always finds a winner that is not real._

## What makes a good test

- **A clear hypothesis.** For example: showing real availability before sign-up will increase completed bookings, because people see value first.
- **One primary metric.** The number the decision depends on, such as completed sign-ups. Track a few others only to make sure nothing breaks.
- **A planned sample size.** Calculated before the start from your current conversion rate and the smallest improvement worth detecting. Free calculators do this.
- **Whole weeks.** Running for complete weeks avoids results skewed by weekday and weekend differences.
- **A meaningful change.** Small tweaks need enormous traffic to measure. Bigger changes show clearer results.

> **Do not peek and stop.** Results swing a lot early on. If you stop the moment B looks ahead, you will often ship changes that did nothing. Wait until the planned sample size is reached.

## When not to A/B test

A/B testing needs volume. A page with a few hundred visitors a week and a low conversion rate can take months to produce a reliable answer, if it ever does. For most early products, that time is better spent elsewhere.

| Situation | Better approach |
| --- | --- |
| Low traffic, early product | [User research](https://www.choie.co/glossary/user-research) and usability sessions |
| Testing a whole new idea | A [prototype](https://www.choie.co/glossary/prototype) shown to target users |
| Obvious usability problem | Fix it, then watch the numbers |
| High traffic, a specific question | A/B test |

Five people trying a new sign-up flow will often show you the problem in an afternoon. A test is the right tool once the product has steady traffic and you are choosing between two reasonable options. In the last week of a Choie build, the product goes live and Choie watches real people use it, fixing what gets in their way. A problem that plain does not need a test.

## Tools and setup

Many [product analytics](https://www.choie.co/glossary/product-analytics) tools now include experiments and feature flags, and dedicated platforms exist for larger programs. Feature flags let engineers ship both versions in the same release and switch who sees what without a new deployment, which is useful on mobile apps where releases go through store review.

Common places to test once traffic allows are the [landing page](https://www.choie.co/glossary/landing-page) headline, pricing page layout, sign-up steps, and onboarding. On a live product, Choie's ongoing monthly work takes one priority at a time, which makes it easier to change one thing and measure the result.

## Common mistakes

- **Stopping early.** Ending a test when the result looks good, before the planned sample size.
- **Testing too many things at once.** Five changes in one variant leave you unsure which one mattered.
- **Picking the metric afterward.** Looking at twenty metrics will turn up a winner by chance.
- **Ignoring novelty.** Returning users sometimes click on anything new, then settle back. Longer tests reduce this.
- **Testing with too little traffic.** An inconclusive test is not a sign that the two versions are equal.

## Common questions

### How long should an A/B test run?

Until it reaches the sample size you calculated before starting, and for at least one full week, ideally two. The right length depends on your traffic and the size of the change you want to detect.

### What is the difference between A/B and multivariate testing?

An A/B test compares two versions that differ in one way. A multivariate test changes several elements at once and measures each combination, which needs far more traffic to reach a reliable result.

### Can I A/B test with low traffic?

You can, but results are likely to be unreliable or take months. With low traffic, usability testing and user interviews usually answer the same question faster. Choie's one-week [coded prototype](/services/coded-prototype) is built for that stage: after delivery you can put it in front of your own users and see where they hesitate.

### What does statistical significance mean in an A/B test?

It means the difference between versions is unlikely to be due to chance alone, given the sample size. It does not tell you the change is large or worth shipping, so look at the size of the effect too.

Related: [Product analytics](https://www.choie.co/glossary/product-analytics), [Landing page](https://www.choie.co/glossary/landing-page), [Usability testing](https://www.choie.co/glossary/usability-testing), [User activation](https://www.choie.co/glossary/user-activation), [User research](https://www.choie.co/glossary/user-research).

HTML: https://www.choie.co/glossary/ab-testing