What is A/B testing and how do you run one properly?

An A/B test shows two variants to randomly split audiences simultaneously and measures which performs better on a defined metric. Random assignment and concurrent running are what make it causal rather than correlational.

How to run one properly:

• Start with a hypothesis, not a hunch: "moving the form above the fold will increase submissions because mobile users are not scrolling to it". A hypothesis makes the result informative whichever way it goes.
• Test one meaningful variable at a time, or you cannot attribute the outcome.
• Calculate the sample size in advance from your baseline rate and the effect size worth detecting. This determines how long to run, and it is the step most people skip.
• Run for full business cycles — at least one or two complete weeks — because behaviour varies by day of week.
• Do not stop early because it looks good. Peeking and stopping at the first significant result is the most common way A/B tests produce false wins.
• Measure the metric that matters. A button colour that lifts clicks but not purchases has achieved nothing.

Note: Two honest points. Most tests produce no significant difference, and that is a normal result rather than a failure. And small sites often lack the traffic for statistical significance on small changes — in that case testing bigger changes, or relying on qualitative research, is more sensible than running underpowered tests and trusting the output.