Yes. You can start learning on Learnr for free. Learnr Pro unlocks higher daily limits and extra features.

Learn A/B Testing

Design experiments that give answers you can actually trust.

Free to start. Built from a single prompt. No credit card.

A/B testing is a randomized controlled experiment for products and marketing: you split users at random between a control and a variant, measure a chosen metric, and use statistics to judge whether the difference is real or noise. The math is approachable, but most failed tests break on design: a vague hypothesis, the wrong metric, too small a sample, stopping early when the result looks good, or a randomization bug that sends unequal traffic to each group. Learnr builds an A/B testing course around your product or marketing context, teaching the statistics and the experiment design together in short lessons.

How to learn A/B Testing, step by step

Write the hypothesis before building anything. State the change, the user behavior you expect to shift, the primary metric and the minimum effect worth acting on. Add guardrail metrics, such as refunds or page load time, that must not get worse.
Learn why randomization matters. Random assignment balances everything you did not measure between groups, which is what lets you read the difference as caused by the change. Pick the unit carefully, usually users rather than page views.
Calculate sample size up front. Use a sample size calculator with your baseline conversion rate, minimum detectable effect, significance level and power. Small effects on low baseline rates need far more users than most people expect, so check feasibility first.
Read results without fooling yourself. Learn what a p-value does and does not mean, and report a confidence interval for the effect size. Avoid checking daily and stopping at the first significant result, which inflates false positives, unless you use a sequential method designed for it.
Run the validity checks. Confirm traffic split matches the plan to catch sample ratio mismatch, run for full weekly cycles and watch for novelty effects fading. Brief Learnr reviews help you keep these checks and statistical terms sharp.
Practice on simulated and real tests. Simulate A/A tests in a spreadsheet or Python to see how often noise looks significant, then analyze a real experiment from your work or a public dataset and write a one-page decision memo.

What you'll learn

Write a testable hypothesis with a primary metric and guardrail metrics
Estimate sample size from baseline rate, minimum detectable effect and power
Interpret p-values and confidence intervals correctly
Detect problems such as sample ratio mismatch and novelty effects
Decide when to ship, iterate or stop based on results

What's inside your A/B Testing course

A sample path Learnr builds for A/B Testing. Your real course adapts to your goal and level.

1. Why randomized experiments work
2. Hypotheses, metrics and guardrails
3. Sample size, power and test duration
4. Significance, p-values and confidence intervals
5. Pitfalls: peeking, multiple tests and SRM
6. Making decisions and building an experiment culture

How long does it take to learn A/B Testing?

The core concepts and statistics take about 3 to 6 weeks at 3 to 4 hours a week, assuming basic statistics. Designing and judging experiments confidently comes from running or reviewing a handful of real tests over a few months.

What you need before you start

Basic statistics, such as means, proportions and the idea of sampling variation, plus spreadsheet skills. Knowing some Python or SQL helps for analyzing raw experiment data.

Common mistakes to avoid

Peeking at results daily and stopping as soon as the difference looks significant, which greatly increases the chance of a false win.
Testing many metrics or variants and celebrating whichever one crosses the threshold. Pick one primary metric in advance and adjust for multiple comparisons.
Running a test far too small to detect a realistic effect, then concluding the change made no difference.

How Learnr builds your course

Type your topic

Tell Learnr you want to learn A/B Testing, or get as specific as you like.

Get a personalized course

Learnr instantly builds a full A/B Testing course tailored to your goal and skill level.

Learn and level up

Move through bite-sized lessons and interactive practice, earning XP and keeping your streak.

A/B Testing questions, answered

How long should an A/B test run?

Long enough to reach the sample size you calculated in advance and to cover at least one full week, so weekday and weekend behavior are both included. Do not end it early just because the result looks significant.

Do I need a lot of traffic to A/B test?

It depends on your baseline rate and the size of effect you care about. With little traffic, only large effects are detectable, so test bold changes or use qualitative research instead.

What is the difference between frequentist and Bayesian A/B testing?

Frequentist tests use p-values and confidence intervals with a fixed sample plan. Bayesian approaches report the probability that a variant is better given a prior. Both work when designed carefully; neither fixes a badly designed test.