Web Strategy

A/B Testing on WordPress: Running Experiments That Give Trustworthy Answers

A/B Testing on WordPress: Running Experiments That Give Trustworthy Answers

“Let’s A/B test it” is one of the most common suggestions in website meetings. Should the button be green or orange? Should the headline mention price? Should the form have five fields or three? Instead of arguing, the idea goes, let the visitors decide. A/B testing can settle these questions, when it’s done properly.

It’s a good instinct. Testing changes against real behaviour is far better than guessing. But I’ve seen many small businesses run A/B tests that produced confident-looking answers that were essentially random. They declared a winner after a week, rolled out the change and saw no improvement, or even a decline.

A/B testing works, but only when it’s done with enough traffic, a clear question and some discipline about reading results. This article covers how to do it properly on WordPress, and when not to do it at all.

How A/B testing works

In an A/B test, visitors are randomly split into groups. One group sees the current version, the control. The other sees a changed version, the variant. You measure a specific outcome, such as form submissions or purchases, for each group. If the variant performs better by a margin unlikely to be due to chance, you have evidence that the change helps.

The key ideas are random assignment, a single measured goal and enough data to separate real effects from noise.

First question: do you have enough traffic?

This is where most small-site tests fail. Detecting a difference reliably requires a surprising number of conversions.

As a rough illustration: if a page converts at 3 percent and you want to detect a relative improvement of 20 percent, taking it to 3.6 percent, you typically need many thousands of visitors per variant to reach a reliable result. A page with a few hundred visitors a month would need to run the test for years.

Use an A/B test sample size calculator before you start. Enter your current conversion rate, the smallest improvement you care about and your traffic. If the required duration is longer than about six to eight weeks, an A/B test probably isn’t the right tool for that page.

What to do if you don’t have enough traffic

Low-traffic sites can still improve conversions:

  • Test bigger changes. A completely different page approach can produce large effects that are detectable with less traffic. Button colour changes produce tiny effects that need enormous samples.
  • Use qualitative research. Watch session recordings, run five-person usability tests, read customer emails and ask people why they did or didn’t get in touch. These reveal problems directly.
  • Apply proven best practices. Clear headlines, fewer form fields, visible trust signals and fast pages are well-established improvements.
  • Measure before and after. Not as rigorous as a test, but if you make a significant change and conversions jump substantially for months, that’s meaningful evidence.
  • Test higher in the funnel. Clicks on a call-to-action button happen more often than completed purchases, so tests on clicks reach significance faster. Just remember that more clicks don’t always mean more sales.

Choosing what to test

Good tests start with a hypothesis based on evidence, not a whim. A useful format:

“Because we observed [evidence], we believe that [change] will cause [effect] for [audience]. We’ll measure this by [metric].”

For example: “Because session recordings show most mobile visitors never scroll past the hero, we believe that moving the enquiry form above the fold on mobile will increase form submissions. We’ll measure enquiry form completions.”

Prioritise tests by potential impact, confidence in the hypothesis and ease of implementation. Pages with the most traffic and the most direct link to revenue, such as key landing pages, pricing pages and checkout, are usually the best places to test.

Tools for WordPress

Since Google Optimize shut down in 2023, the tool landscape has changed. Options include:

WordPress plugins. Several plugins run A/B tests directly in WordPress, creating variants of pages or blocks and splitting traffic. Some work at the block level, letting you test a single headline or button within the editor. These are convenient for smaller sites and keep everything in WordPress.

Dedicated testing platforms. Services like VWO, Convert, AB Tasty and Optimizely offer visual editors, advanced targeting and robust statistics. They’re more powerful and more expensive, and suit sites with significant traffic.

Server-side testing. For developers, assigning variants on the server and rendering different templates avoids the flicker that client-side tools can cause. It’s more work but gives the cleanest results and performance.

Analytics-based testing. Some privacy-focused and general analytics tools support split testing or can measure custom events by variant, which works with a simple variant assignment in your own code.

The flicker problem

Many client-side testing tools work by loading the original page, then using JavaScript to change it into the variant. If that happens visibly, visitors see the original briefly before it changes. This “flicker” looks broken and can bias results.

To prevent it, tools often hide the page until the script runs, using an anti-flicker snippet. That delays rendering and can hurt Largest Contentful Paint. Server-side testing, or testing tools that integrate with page caching properly, avoid both problems.

Caching and A/B tests

Page caching and A/B testing can conflict. If a cached page is served to everyone, visitors may all see the same variant. Server-side tests on cached sites need variant-aware caching, for example by varying the cache on a cookie. Client-side tests work with caching but bring the flicker issue.

Check how your chosen tool handles caching before you start, or your test might be measuring nothing at all.

A/B testing and SEO

Google is fine with A/B testing as long as you follow a few rules:

  • Don’t show search engines different content from users. Treat Googlebot like any other visitor in random assignment.
  • If variants use different URLs, add a canonical tag on the variant pointing to the original.
  • Use 302 temporary redirects, not 301s, if you redirect visitors to variant URLs.
  • Don’t run tests longer than necessary.

Following these avoids any risk of the test affecting rankings.

Reading results honestly

This is where discipline matters most.

Decide the sample size and duration in advance. Then stick to it. Checking results daily and stopping as soon as one variant is ahead, often called “peeking”, dramatically increases false positives. Early leads frequently disappear.

Run for full weeks. Behaviour differs between weekdays and weekends. Run tests for at least one, ideally two, complete weekly cycles.

Look at significance and size together. A statistically significant result tells you the difference probably isn’t zero. It doesn’t tell you the difference is large. A 0.5 percent improvement might be real but not worth the complexity.

Check your primary metric, not a dozen. If you measure twenty metrics, one will likely show a “significant” difference by chance. Choose one primary metric before starting.

Segment carefully. Looking at results by device or traffic source is useful for understanding, but small segments have even less data. Treat segment findings as ideas for future tests, not conclusions.

After the test

When a test ends:

  • If the variant won convincingly, implement it properly, preferably in the theme or content rather than leaving the testing tool to apply it.
  • If there’s no clear winner, that’s a valid result. The change doesn’t matter much, so choose whichever is simpler or better aligned with your brand.
  • If the variant lost, you’ve learned something valuable and avoided a harmful change.

Record every test, including hypothesis, dates, traffic, results and decision. Over time, that record becomes a valuable guide to what works for your audience.

A note on ethics

Tests should never deceive visitors. Testing different headlines or layouts is fine. Testing fake scarcity messages, misleading prices or hidden terms is not, regardless of what the numbers say.

Share results widely

Share test results with everyone who works on the site, including the losing and inconclusive ones. Teams that see the evidence make better guesses about the next test, and fewer arguments are settled by whoever speaks loudest in meetings. A short monthly summary is enough to build that habit.

Start with one test

Run one well-planned test at a time on the same page, so results don’t interfere with each other.

Frequently asked questions

How long should an A/B test run?

Until it reaches the sample size you calculated in advance, and for at least one or two full weeks. Don’t stop early because one variant looks ahead.

Can I test more than two versions?

Yes, but each additional variant needs more traffic. For most sites, testing one change at a time against the original is more practical.

Do A/B tests slow down my site?

Client-side tools can, particularly with anti-flicker snippets. Server-side approaches and lightweight plugins have much less impact.

What should a small business test first?

Usually the headline and primary call to action on the highest-traffic landing page, because those changes can have large effects and the page has the most data.