How to use it
- Enter visitors and conversions per variant; the first row is the control. Use Add variant for more versions.
- Choose the confidence and test type. Without a specific reason, use 95% and two-sided.
- Read the verdict, the confidence interval of the lift and the p-value; if the split looks wrong, fix the data first.
How it is calculated
- Significance: pooled two-proportion z-test, with the p-value for the chosen one- or two-sided test.
- Rate intervals: Wilson; difference interval: Newcombe hybrid score; relative lift interval: log ratio. Small samples and rates near 0% never produce negative bounds.
- Several variants: each is compared with the control and p-values are Holm-adjusted.
- Sample ratio check: chi-square test against the planned split; SRM is flagged when p is below 0.001.
- Visitors still needed: assuming the true lift equals the observed one, the visitors per variant for 80% power.
Frequently asked questions
When is an A/B test result statistically significant?
When the p-value is below 1 minus the confidence level (0.05 at 95%) and the confidence interval of the difference excludes zero, the gap is unlikely to be random noise. Also check that the planned sample size was reached and the test ran for whole weeks; a “significant” result seen by stopping early is often false.
Is the p-value the probability that B beats A?
No. The p-value is the chance of seeing a difference at least this large if the versions were actually identical. It is not the probability that B wins; judge business value from the lift and its confidence interval.
Should I use a one-sided or two-sided test?
Use two-sided by default so a worse variant is also detected. Use one-sided only if you decided before looking at the data that you will ship B solely if it is better and do not care how much worse it might be; never switch after seeing the results.
Why do p-values go up when I test several variants?
Every extra comparison is another chance of a fluke. With several variants the calculator applies the Holm correction to control the overall false-positive rate, so each variant needs stronger evidence.
What is sample ratio mismatch (SRM)?
If you planned a 50/50 split but the visitor counts differ clearly (chi-square p below 0.001), the split, a redirect or the tracking is usually broken. The conversion comparison cannot be trusted until the data issue is fixed.
XunL Experiments: edit pages visually, assign visitors consistently, confirm accepted inquiries on the server and get reports with confidence intervals and a clear decision status.