A/B tests – how they work and why it’s worth running them
Changes to websites, online stores or mobile apps are often based on intuition, aesthetics or team opinion. Unfortunately, as the data shows,most of these decisions don’t translate into better business results. That’s exactly why A/B testing and conversion rate optimization (CRO) have become the foundation of modern digital product development.
Avinash Kaushik, one of the most respected web analysts, once said: “80% of the time, you’re wrong about what your customers expect from your website.“
” That line perfectly captures why it’s worth running data-driven tests.
A/B tests let you make data-driven decisions, reduce risk and build pages that genuinely answer user needs. As a CRO agency , we run dozens of tests like this every month, and in this article we’ll tell you how and why.
What are A/B tests and how do they work?
Let’s start with the basics. An A/B test is a research method where you compare two versions of a webpage (version A and version B) to see which achieves better results in conversion rate or other important metrics.
The system randomly assigns users to one of the versions – version A (the original) or version B (the test variant). Data is then gathered on their behavior: conversions, average order value and other metrics.
At the end we analyze which version of the page performed better. If the difference between variants is statistically significant, we can say with strong confidence that the change genuinely produces an effect.
So an A/B test isn’t just about comparing conversion rates – it’s a statistical test aimed at establishing causation between the change introduced and user behavior.
Why is it worth running A/B tests?
On average around 80% of ideas meant to raise a website’s business results don’t deliver the expected outcome. Conversion optimization and A/B testing let you make decisions based on data instead of intuition.
With A/B tests:
- You reduce the risk of costly mistakes
- You only ship changes that actually work
- You learn how customers really behave on your site
- You iteratively improve UX and the effectiveness of your sales funnels
- You can detect even minor purchase barriers
Small changes that lift the conversion rate by even 1-2% can translate into hundreds of thousands of PLN in additional annual sales,especially in e-commerce businesses with heavy traffic.
It’s not for nothing that tech giants (Amazon, Booking.com, Airbnb) run tens of thousands of tests a year.
An optimization and A/B testing strategy
Conversion optimization is a process, not a one-off action. A/B tests are a cycle of learning and improving – every iteration produces new findings that feed into the next tests.
A proper website optimization process consists of these steps:
- Research and analyzing user behavior
- Identifying problems and growth opportunities
- Forming hypotheses
- Verifying those hypotheses (A/B tests are the best method for establishing causation)
- Shipping the changes you have evidence lift business metrics
- Repeating the process continuously and building an ever better-performing site
CRO isn’t only A/B tests. It’s an entire process of researching the user experience, identifying purchase barriers, and shipping solutions that improve the user experience and raise the conversion rate.
A/B tests alone aren’t enough – what’s critical is understanding what and why you’re testing. That’s why before running any tests we gather data, analyze it, and build hypotheses grounded in facts rather than intuition.
How do you run an A/B test step by step?
1. Define the test’s goals
Before launching a test, we precisely define what we want to achieve:
- Raising the conversion rate (purchases, sign-ups, newsletter subscriptions, say)
- Improving engagement metrics (time on site, say)
- Lowering the bounce rate
2. Identify the elements to test
Based on data from Google Analytics and other analytics tools, we choose the elements that could have the biggest effect on user behavior.
3. Build a hypothesis
We form a clear hypothesis: “Moving the add-to-cart button higher will lift conversion by 3%, because more users will start noticing the page’s most important action button”
4. Set the test duration and sample size
Before starting the test, we calculate the minimum sample size and set the test’s duration based on the MDE (Minimal Detectable Effect), so the test has enough statistical power and is trustworthy.
The test should run for at least 2 full weekly cycles, though, to account for fluctuations in user behavior.
We write more about this here: https://digispot.io/how-long-should-an-a-b-test-run/
5. Launch the test and monitor the results
While the test runs we regularly check progress, but we don’t draw hasty conclusions before reaching statistical significance.
6. Analyze the results
Once the test ends we run a statistical analysis to check whether the difference between variants is statistically significant.
7. Ship the winning variant
If the tested variant achieved better results in a statistically significant way, we make the change permanent.
The technical side of A/B testing
Test types
We can run different kinds of tests:
- A/B – the classic two-version test
- A/B/n – a test of multiple variants at once (A, B, C, say)
- Split URL – testing pages at different URLs, used for larger layout changes
- MVT (multivariate testing) – testing multiple elements at once
Client-side vs server-side
Technically, an A/B test can be run in 2 ways:
- Client-side – changes load in the browser. Faster to ship, but may not aggregate all the data.
- Server-side – changes are implemented directly on the server. More reliable, especially for tests that affect the backend.
A/B testing tools
You can use tools like these to run A/B tests:
- Google Optimize (now requires migration)
- Convert.com
- Varify.io
- VWO (Visual Website Optimizer)
- Optimizely
Choosing the right tool depends on your site’s traffic, budget and technical needs.
Analyzing and interpreting A/B test results
One of the most common mistakes in interpreting A/B tests is expecting every change to produce a positive result.
On average only 20% of A/B tests produce statistically significant gains — which means the other 80% show no effect, are inconclusive, or produce statistically significant drops. That isn’t a problem, because every test delivers valuable findings.
We talk about statistical significance when the p-value (a key figure in statistical analysis) falls below a set threshold, usually 0.05. That means the difference between the test variant and the control is large enough that the probability of that difference occurring by chance is very low.
What does an inconclusive test mean?
An inconclusive test means there’s no statistically significant difference between the test variant and the original. It means we don’t have enough evidence that the observed effect is real rather than the result of random fluctuation.
An inconclusive test doesn’t mean the change definitely doesn’t work in reality. It may be effective, but the effect may be too small to detect at the current sample size.
Examples of effective A/B tests
Why do brands use A/B tests?
A/B testing and CRO are an investment, not a cost. Through them we find better versions of a website and get strong evidence about which changes lift conversion and which don’t.
Testing is an effective way of limiting risk. Instead of shipping changes blind, A/B tests let you check whether a given modification genuinely improves results before it’s rolled out to every user.
Summary – effective, data-driven A/B testing
A/B tests are a fundamental tool in the arsenal of any marketer or UX designer who wants to make decisions based on data instead of guesswork. A well-planned, well-run test lets you identify a page’s most effective elements, lift the conversion rate and optimize the user experience.
Remember that even a small change to a CTA button, headline or page layout can significantly affect your results. The key to success is systematic testing, analyzing the results you get, and shipping the changes that genuinely work.
Define your goals, build sensible hypotheses, run rigorous tests and analyze the results – that’s the only way to make decisions that actually improve your website’s results.
Recommended sources and further reading on A/B testing
📌 Trustworthy Online Controlled Experiments – a book written by Ron Kohavi
📌 Ruben de Boer’s newsletter and courses
📌 CXL courses – practical training in testing, UX and CRO strategy
📌 The LinkedIn of Adrian Jaskulski, co-founder of Digispot: https://www.linkedin.com/in/adrian-jaskulski/
