Most failed A/B tests were not failed by the idea. They were failed by the data underneath.
I have run experimentation programs many clients in various industries, and I own the GA4 and GTM tracking on those accounts too. That second part matters more than people expect. When the person running the test does not control the measurement, a "winning" variant can be a tracking artifact, and a "losing" one can be a real win nobody could see.
If you lead growth, ecommerce or digital, this is the short list I check before I trust any test result.
A note on numbers: the lifts below are relative changes on client work covered by NDA. I describe the industry and the result, not the client. Raw baselines and sample sizes stayed with the clients and employers, so I do not quote them. What I can show you is the method, and I am happy to walk through how I plan, run and read a test on a call.
The mistakes at a glance
- Testing before you verify the conversion event
- Skipping the sample ratio mismatch (SRM) check
- Deciding sample size and duration after launch
- Declaring a win because one number went up
- Letting consent settings and ad blockers distort the data
- Reading the test in one tool and the revenue in another
- Writing the hypothesis from opinion instead of drop-off data
1. Testing before you verify the conversion event
Why it happens: The GA4 event exists, so everyone assumes it fires correctly.
What it costs: If the "form submit" event also fires on page reload, on validation errors, or only on one device, every variant inherits the same noise. The lift you report is built on it.
What to do instead: Before launch, complete the conversion yourself on desktop and mobile. In GTM preview and GA4 DebugView, confirm the event fires once, only on success, with the parameters you expect. Then send the experiment ID and variant name into GA4 as event parameters registered as custom dimensions, so you can segment conversions by variant in GA4 and not only in the testing tool. I treat this as test QA, not analytics housekeeping.
2. Skipping the sample ratio mismatch (SRM) check
Why it happens: The testing tool says 50/50, so people trust it.
What it costs: If the split is meaningfully off, something is wrong with how visitors are assigned, and the results cannot be trusted no matter how good they look. A broken variant, a redirect, a bot filter or a tracking gap can cause it.
What to do instead: Run a chi-square test of the observed visitor counts against the expected split, on every test, before reading any result. A common rule is to treat p below 0.01 as a failed check. Context matters: 52/48 is noise on 400 visitors and a real problem on 400,000. If the check fails, find the cause and rerun. Do not "adjust for it".
3. Deciding sample size and duration after launch
Why it happens: The test looks promising on day four.
What it costs: Stopping when a number looks good is how teams ship false winners. Early peeks overstate lifts.
What to do instead: Before launch, set the sample size from four inputs: your baseline conversion rate, the minimum detectable effect worth acting on, your confidence level (95% is common) and statistical power (80% is common). Any sample size calculator will do the math. Convert it to a duration, round up to full weeks so you capture every day of the week, and write it down. My tests typically run two to six weeks, and the length is set before launch from traffic and baseline, not from how the chart looks on day four. If you need to peek, use a method designed for it (sequential or Bayesian) instead of a fixed-horizon test you stop early. I call results on frequentist statistics and watch the Bayesian view alongside it for context.
4. Declaring a win because one number went up
Why it happens: The test metric is easy to measure, and a lift feels like a result.
What it costs: Here is the one that taught me the most. On a legal-protection subscription for truck drivers, I ran a test that raised the web conversion rate. The statistics were fine. But the company also sold through an inside sales team, and that team took a hit. Total sales went down. Once we zoomed out to the whole business, we could not justify the higher web conversion rate, because the company was selling less overall. The test did not fail statistically. The business case failed.
The same thing happens with softer metrics. A clearer secondary CTA on a childcare franchise network ("Tuition and Openings") lifted CTA click-through by 13.34%. I only trusted it because lead-form conversion rose 7.23% too.
What to do instead: Pick one primary metric that is a real business outcome, then check the guardrails: other channels, phone and sales-team results, lead quality and revenue. Ask what the winning variant takes away from somewhere else. Before calling a winner I want to see that the company, not only the page, is better off.
The fix was a rebuild of the top two sections of the page. We moved the sales phone call to a step after the purchase, so the sales team stopped competing with the web checkout. Monthly revenue rose 55%, with a 28% lift in LTV per purchase and 21% in web sales. That was a rebuild, not a split test, so read it as directional. It is still the number that mattered, because it was measured on revenue and not only on conversion rate.
5. Letting consent settings and ad blockers distort the data
Why it happens: Consent setup is handled by a different team, or nobody.
What it costs: Consent configuration decides which users GA4 and your ad pixels can see. If it behaves differently across variants, devices or regions, your test and your paid media reporting can both be skewed. Ad blockers can also hide GA4 hits for a slice of visitors, and modeled data fills only part of that gap.
What to do instead: Include consent behavior in test QA. I have worked with CookieYes, OneTrust and Cookiebot. On one OneTrust setup, the consent signal fired after the page had loaded and after every other tag, so the tags that were supposed to wait for consent had already run. We had to rearrange the order things fired in and, in some cases, trigger tags off OneTrust's own consent firing instead of page load. The pieces I reach for are Consent Mode defaults set before anything else, a Consent Initialization trigger so the defaults load first, and wait_for_update so tags give the consent tool time to report the visitor's choice. Confirm in preview that tags respect consent in the right order, and that they do so consistently across variants. Know what share of your traffic GA4 cannot see, and compare against a second source.
If you work in healthcare or another sensitive category, go one step further and look at what the tags can see. On healthcare accounts I have stripped URL parameters before they reach analytics, such as location searches and values a form inserts into the URL, and I have excluded some pages from tracking altogether. This is a tracking-hygiene practice, not legal advice. Have your privacy or legal team set the policy, then make sure the tags actually follow it.
6. Reading the test in one tool and the revenue in another
Why it happens: The testing platform reports its own conversions, GA4 reports different ones, and the CRM or store backend has a third number.
What it costs: Leadership sees three answers and trusts none of them. Ecommerce teams know the version of this: GA4 purchase revenue that does not match the store.
What to do instead: Agree on the source of truth before the test starts. Reconcile the testing tool against GA4 and, where possible, against the CRM, call tracking or store orders. Some difference is normal, because tools count users, sessions and attribution differently. Explain the gap in the readout instead of hoping nobody notices.
7. Writing the hypothesis from opinion instead of drop-off data
Why it happens: Someone senior has a strong view about the hero image.
What it costs: You spend a test slot on a guess.
What to do instead: Start with where people drop off in GA4, then watch recordings and heatmaps in Clarity or Hotjar to see why. On a childcare franchise's Schedule a Tour form, that process showed a broken, high-drop-off form. I designed the rebuild, wrote the developer requirements and A/B tested old against new: +14.13% conversion rate, statistically significant.
The takeaway
A test is only as good as the question it was built to ask and the data used to answer it. If your last few "wins" did not show up in revenue, check the tracking before you blame the idea.
Want a second set of eyes on your testing program?
I work on both halves: running the experiments and fixing the tracking underneath them. If you are hiring for CRO, experimentation or analytics, or want a read on how trustworthy your current results are, book a call and send me your testing tool and GA4 setup. I'll tell you what I would check first. References are available on request.
