How it’s calculated
Two rates, confidence and power, in one formula.
- n = (z₁₋α/₂ × √(2p̄(1 − p̄)) + z₁₋β × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²
- p₁ = current rate, p₂ = p₁ × (1 + minimum effect), p̄ = (p₁ + p₂) ÷ 2
- z: the standard normal values for the confidence level and the power (95% → 1.96; 80% → 0.84)
- Test length = Total visitors ÷ Daily visitors
An A/B test's sample size is the number of visitors each version needs so that a given difference between them can be seen with statistical confidence. It is computed from the current conversion rate, the smallest difference you want to detect, the confidence level and the power.
Small differences take a lot of traffic: halving the minimum effect roughly quadruples the visitors needed.
Stopping before the computed number raises the risk of mistaking a random difference for a real one. Don't peek and decide early.
Even if the length comes out under a week, running for at least one full week balances weekday and weekend behaviour.
Limits
- It assumes the classic (frequentist) two-sided test comparing two proportions; Bayesian or sequential testing tools compute differently.
- With more than two versions no multiple-comparison correction is applied; the false-positive risk grows with the number of versions.
- If traffic is not split evenly between versions, the total needed rises.
Frequently asked questions
How many visitors do I need for an A/B test?
Copy linkIt depends on your current conversion rate and the lift you want to detect. To detect a 20% relative lift on a 5% rate with 95% confidence and 80% power you need about 8,150 visitors per version.
What do confidence level and power mean?
Copy linkThe confidence level limits the risk of finding a difference that isn't there (95% means a 5% risk). Power is the chance of detecting a difference that is there (80% means four times out of five).
How long should the test run?
Copy linkUntil it reaches the computed number of visitors, and for at least one full week. Divide the total visitors by the daily visitors to estimate the length.
Let’s plan your tests together.
We review your account together in 30 minutes and tell you where to start.







