"Let's just try it and see" is how most small stores make changes — swap a headline, tweak a button, ship it, watch the dashboard for a week, decide it worked. The problem isn't the instinct. It's that you can't tell a real improvement from normal week-to-week noise without a process for it. That's all A/B testing is: a process that keeps you from fooling yourself.
You don't need a data team to run one. You need six steps, done in order, without skipping the boring ones.
Key takeaways
- Test one variable at a time, tied to a real decision — not a backlog of random ideas.
- Decide your one success metric and how long you'll run before you look at a single result.
- Estimate the sample size you'll need before starting — it tells you if the test is even feasible with your traffic.
- Run for fixed, pre-committed full business cycles. Checking early and stopping when it "looks good" is the single most common way these go wrong.
- A confident "no difference" is still a useful result — it retires a bad idea for good.
Most first tests are about one page. On a typical homepage, three elements carry most of the weight, and each is a candidate for a single-variable test:
1
Pick one variable, tied to a real decision
Test something you'd actually act on either way — a checkout field you suspect is causing drop-off, a price presentation, a headline you keep debating internally. Change exactly one thing. If you change the headline and the button colour and the layout at once, a win tells you nothing about which change caused it.
2
Write down your hypothesis and one success metric first
Before you build anything: "I believe [change] will improve [metric] because [reason]." Pick the one metric that decides the test — usually conversion rate or revenue per visitor for the page in question — and commit to it in writing. Deciding the metric after seeing the data is how you end up "finding" a win in whichever number happened to move.
3
Estimate the sample size you'll need
Use a free A/B test sample size calculator (search "AB test sample size calculator" — several reputable ones from testing platforms are free to use). Feed it your current baseline rate for the metric, the smallest improvement worth detecting, and it'll tell you roughly how many visitors per variant you need. If that number would take you eight months to reach at your current traffic, that's real information: either test a bigger change (bigger effects need smaller samples to detect), pick a higher-traffic page, or accept you're making a judgment call, not running a statistically clean test — and that's fine, as long as you're honest about which one you're doing.
4
Run for a fixed, pre-committed duration
Decide the run length before launch, and make it at least one full business cycle (usually a full week, ideally two) so you're not comparing a Tuesday to a Saturday. Then don't look at results and decide mid-test — see the callout below for why that matters more than almost anything else on this list.
5
Read the result — significance and sample size together
Check whether the difference is statistically significant (most testing tools will tell you this directly, commonly using a 95% confidence threshold) and whether you actually hit the sample size from step 3. A result that clears the significance bar on far fewer visitors than you calculated you needed is more likely to be noise than a real effect — treat it as a lead worth retesting, not a final answer.
6
Decide, document, and move on
Ship the winner, or don't — and either way, write down what you tested, what happened, and why. A clear "no difference" result is a genuine outcome: it stops the same debate from resurfacing in three months as if nobody had checked.
Plan
BuildThe single biggest mistake: peeking early
Checking results daily and stopping the moment it "looks significant" is the most common way a well-designed test still gives you a wrong answer. The math behind statistical significance assumes you decided your sample size and stopping point in advance — checking repeatedly and stopping at the first good-looking moment inflates your odds of a false positive far above the 5% you think you're accepting. Set your duration and sample size target before you start, and don't call the test until you hit both.
Where this simplifies reality, on purpose
A real statistics team would layer in more than this: guarding against running many tests at once and treating any one "win" as significant purely by chance (the multiple-comparisons problem), and more rigorous approaches to deciding when a test can stop early without inflating false positives. This guide leaves those out deliberately — the six steps above catch the mistakes that actually sink most small-store tests, and adding more machinery than that usually just gives founders a reason to skip testing altogether.
If you're not sure whether a specific test you're planning has a realistic chance of reaching a clean answer with your traffic, get in touch — a short conversation is usually enough to tell you whether it's worth running as a formal test or just worth shipping as a confident call.
