A/B testing with low traffic: what is worth testing in a small store
How much traffic an A/B test needs, what is worth testing in a small store, how to avoid conclusions from luck and what to do when volume is too low.
Vasile Dingă
.comparison-cards { display: flex; flex-direction: column; gap: 1.5rem; } .comparison-card { background: #fff; border-radius: 16px; padding: 20px; border: 2px solid #efefef; } .comparison-header { display: flex; justify-content: space-between; font-size: 16px; margin-bottom: 12px; } .comparison-row { display: flex; justify-content: space-between; padding: 8px 0; border-top: 1px solid #efefef; font-size: 14px; color: #1f2937; } .comparison-label { font-size: 14px; width: 30%; } .comparison-value { width: 65%; text-align: right; } @media (max-width: 600px) { .comparison-row { flex-direction: column; align-items: flex-start; } .comparison-value { text-align: left; margin-top: 4px; } } A/B tests sound like the perfect solution: show two versions of a page, see which one sells more and keep the winner. The problem is that a small store with a few thousand visits a month rarely has enough orders for a test to say anything reliable. The result looks clear after a week, and after a month it disappears. Below I show you how to tell what you can test, how much traffic you need, how to avoid drawing conclusions from luck and what to do instead when the volume is not there. Why low traffic changes the rules An A/B test compares two conversion rates. With few orders, differences appear and disappear by pure chance. If variant A has 12 orders and variant B has 18, B looks 50% better. In reality, at such small volumes a difference like that often shows up even between two identical pages. That is why you do not look at percentages, but at how many orders stand behind them. A practical rule I use: if you cannot collect a few hundred orders per variant in a reasonable time, a classic A/B test on conversion rate will not give you a trustworthy answer. Sample size intuition How many visitors you need depends on two things: the conversion rate you start from and how big the difference is that you want to detect. The smaller the difference, the more traffic you need, and the growth is…