Your experiment finished, the dashboard shows a winner, and something still feels off. The traffic split you set to 50/50 is sitting at 47/53. Before you write up the result or ship the change, you need to answer one question: is the split you observed close enough to the split you designed, or is the test itself broken? That question has a precise, statistical answer, and getting it wrong is one of the most common ways a team ships a change based on a number that was never real. This is sample ratio mismatch, and it is the first thing a senior experimenter checks before looking at any lift.
What sample ratio mismatch is
A sample ratio mismatch (SRM) occurs when the number of visitors actually assigned to each variation differs from the split you intended by more than random chance can explain. You designed a 50/50 test; the engine delivered something that a fair coin, flipped that many times, would essentially never produce.
The important word is chance. You should never expect a perfectly even split. If you assign 20,000 visitors with a fair 50/50 rule, landing on exactly 10,000/10,000 is rare — the count wobbles around the target every time, the same way 10,000 coin flips almost never give exactly 5,000 heads. SRM is not "the split isn't perfect." SRM is "the split is so far off that randomness is no longer a believable explanation."
That distinction is why you cannot judge SRM by eye. Looking at 47/53 tells you the direction and size of the gap but nothing about how surprising it is, because surprise depends on how many visitors produced that gap. The same 47/53 is unremarkable at 400 visitors and a five-alarm fire at 400,000. You need a test that accounts for sample size. The tool for that is the chi-square goodness-of-fit test, and OptiPilot's free SRM Checker runs it for you — paste in your observed counts and expected split, and it returns the p-value and verdict.
Why an SRM invalidates your results
Randomized assignment is the entire reason an A/B test can claim causality. When you split traffic randomly, the two groups are, on average, identical in every way — same mix of devices, same proportion of loyal customers, same intent — except the change you are testing. That balance is what lets you attribute any difference in outcomes to the variation rather than to a difference in who was in each group.
An SRM is direct evidence that the randomization did not work as designed. If a mechanism pushed extra visitors into one variation, that same mechanism almost certainly pushed a non-random kind of visitor. Redirects that time out drop the impatient and the slow-connection users. Bot filters that fire unevenly strip out a specific traffic source. In every case the groups are no longer comparable, so the lift you measured is a mix of the real treatment effect and a selection bias you cannot separate out.
This is the point that trips up teams under deadline pressure: you cannot adjust your way out of an SRM. There is no correction factor, no reweighting, no "the split was 45/55 so I'll scale the numbers." You do not know which visitors are missing or why, so you cannot model the bias. An SRM means the data is untrustworthy at the source. The only valid responses are to find and fix the cause, then rerun — or, if you can prove the cause is benign and non-differential, to document exactly why. Reading the Optimizely results page is meaningless until the split checks out.
How SRM is detected with a chi-square test
The standard test is Pearson's chi-square goodness-of-fit. It compares the visitor counts you observed in each variation against the counts you expected under your intended split, and returns a p-value: the probability of seeing a gap at least this large if assignment were truly random.
The statistic is:
chi-square = sum over each variation of (observed - expected)^2 / expected
degrees of freedom (df) = number of variations - 1
For a two-arm test, df = 1. For a three-arm 34/33/33 test, df = 2, and so on. You compute the statistic, look up the p-value for that value and df, and compare it to a threshold.
The industry alarm threshold is p < 0.001. If the p-value falls below it, you treat the experiment as having an SRM and stop trusting the results until you find the cause. Some teams run stricter: Microsoft's experimentation platform uses 0.0005, and a few use 0.0001. The threshold is deliberately far tighter than the 0.05 you use for a metric result, for two reasons. First, you run this check on every experiment, so a loose threshold would drown you in false alarms. Second, SRM is a binary data-integrity gate, not an effect you are trying to measure — you want it to fire only when something is genuinely wrong. Note that a real SRM usually produces a p-value orders of magnitude below any of these thresholds, not a marginal miss.
A worked chi-square example
Take a 50/50 test. You intended an even split; you observed 10,200 visitors in A and 9,800 in B, for 20,000 total. Under a fair split you expect 10,000 in each.
Observed: A = 10,200 B = 9,800 Total = 20,000
Expected: A = 10,000 B = 10,000 (50% of 20,000 each)
chi-square = (10,200 - 10,000)^2 / 10,000 + (9,800 - 10,000)^2 / 10,000
= (200)^2 / 10,000 + (-200)^2 / 10,000
= 40,000 / 10,000 + 40,000 / 10,000
= 4 + 4
= 8.0
df = 2 variations - 1 = 1
p-value for chi-square = 8.0, df = 1 -> p ~= 0.0047
How to read that: p ≈ 0.0047 means a gap this large would happen about 5 times in 1,000 by pure chance. That clears the conventional 0.05 bar, so a naive check would panic. But it is above the 0.001 SRM threshold, so by the standard you do not declare an SRM. It sits in the uncomfortable middle — worth a glance, not worth halting for. This is exactly why the strict threshold exists: a 2% wobble at this sample size is not yet strong enough evidence of a broken split.
Now keep the same 51/49 ratio but run 10x the traffic:
Observed: A = 102,000 B = 98,000 Total = 200,000
Expected: A = 100,000 B = 100,000
chi-square = (2,000)^2 / 100,000 + (-2,000)^2 / 100,000
= 40 + 40
= 80.0
df = 1
p-value for chi-square = 80.0, df = 1 -> p ~= 3.7e-19 (effectively zero)
Identical proportions, wildly different verdict. At 20,000 visitors the 51/49 split is plausibly noise; at 200,000 it is a certainty that something is wrong. This is the single most important intuition about SRM: the ratio alone tells you nothing — the mismatch is a function of the ratio and the sample size together. It is also why staring at the percentages on a dashboard will never substitute for the test.
What causes sample ratio mismatch
An SRM is a symptom, like a fever — it tells you something is wrong without telling you what. The KDD 2019 taxonomy from Microsoft and Booking.com researchers organizes the causes by where in the pipeline the visitor count gets distorted. Walking that pipeline is how you narrow the search.
flowchart LR
A[Assignment] --> B[Exposure]
B --> C[Logging]
C --> D[Analysis]
A -.->|"faulty hash, bad IDs,<br/>uneven ramp, carryover"| A
B -.->|"redirect timeouts, flicker,<br/>variation errors, latency"| B
C -.->|"bot filters, dropped events,<br/>caching, ad blockers"| C
D -.->|"wrong trigger point,<br/>bad joins, segment filters"| D
Assignment-stage causes
The split is wrong at the moment of bucketing. A faulty randomization or hash function, corrupted or reassigned visitor IDs, or carryover effects from a previous experiment that still tag returning users. Uneven ramping belongs here too: if you launch variation B a day after A, or ramp them to different allocations, the counts diverge for a completely mundane reason.
Exposure-stage causes
Assignment was fair, but visitors are lost between being bucketed and being counted. Redirect tests are the classic offender — a visitor sent to a separate URL can close the tab before the redirect completes, so the event never fires and that visitor is never counted. Because the delay is not symmetric across variations, the counts drift. Client-side flicker and load races drop a variation when the page renders before the experiment code runs, latency in one variation lets impatient users leave first, and a variation that errors for a subset of browsers silently loses those visitors.
Logging and analysis-stage causes
The split was fine; the measurement distorts it. Bot and spam filtering applied unevenly strips a traffic source from one arm. Ad blockers and tracking-prevention block the logging beacon for some visitors more than others. Caching or CDN behavior serves stale assignments. On the analysis side, choosing the wrong starting point — counting from a downstream event that already differs between variations rather than from first exposure — manufactures an SRM out of clean data. Bad table joins and delayed log arrival do the same.
The most useful directional clue: excess visitors in the treatment arm often points to logging or engagement changes (the new variant generates more events, so more one-time visitors get recorded), while missing visitors in treatment points to exposure failures like redirects, errors, or latency.
How to diagnose an SRM
Once the check fails, resist the urge to guess. Diagnosis is a process of localization: find the segment or stage where the mismatch lives, and the cause usually becomes obvious.
flowchart TD
S[SRM detected] --> Q1{Is it in every segment<br/>or just some?}
Q1 -->|Isolated to one<br/>browser/device/source| Seg[Suspect that segment:<br/>rendering bug, bot source,<br/>ad blocker, redirect on that platform]
Q1 -->|Spread evenly<br/>across all segments| Q2{Does the split break<br/>at assignment or later?}
Q2 -->|Wrong at first exposure| Assign[Assignment layer:<br/>hashing, IDs, ramp timing,<br/>overlapping experiments]
Q2 -->|Fine at exposure,<br/>off downstream| Q3{Redirect or heavy<br/>client-side render?}
Q3 -->|Yes| Exp[Exposure loss:<br/>redirect timeouts, flicker,<br/>variation errors]
Q3 -->|No| Log[Logging/analysis:<br/>bot filter, wrong trigger point,<br/>join error, caching]
Work through it in this order:
- Segment the mismatch. Break the counts down by browser, device, operating system, day, traffic source, and new-vs-returning. If the SRM collapses into one segment — say, only Safari, or only mobile, or only the day of a deploy — you have found it. A rendering bug, a browser that rejects redirects, or a bot wave from one source will all show up as a single hot segment.
- Find the stage where the ratio breaks. Compare the split at assignment, at exposure, and at analysis. If bucketing is even but exposure is skewed, the loss is happening after assignment (redirects, errors, latency). If assignment itself is off, the problem is upstream in the hashing or ID logic.
- Check the analysis trigger. Confirm you are counting from first exposure, not from a downstream event that the treatment itself changed. This alone explains a large share of "SRMs" that are really analysis artifacts.
- Line up the timeline. Map the onset of the mismatch against deploys, ramp changes, and marketing sends. SRM that starts precisely at a code push has a precise cause.
One caution specific to how detection works: run the SRM check on the whole experiment population, not on filtered segments as a decision input. Segmenting is a diagnostic tool for finding the cause once an SRM is flagged — it is not a place to hunt for fresh imbalances to act on, because slicing enough ways will always turn up a low p-value by chance.
How to prevent SRM
Prevention is cheaper than diagnosis. The teams that rarely see SRM have built these habits in:
- Assign server-side where you can. Server-side or edge assignment removes the flicker, redirect, and client-render failure modes that cause most exposure-stage mismatches. Client-side tests are more exposed and need tighter QA.
- Bucket at the point of exposure. Assign the visitor at the moment they are actually eligible to see the change, not at some earlier step that a variation may reach at a different rate. Assign-at-exposure keeps the counted population aligned with the treated population.
- Run an [A/A test**](https://optipilot.com/data/aa-testing) first.** An A/A test — two identical variants — should show no SRM. If it does, your instrumentation is broken before you have tested anything real. It is the cheapest possible way to catch a pipeline problem.
- Put an SRM check in CI and pre-analysis. Make the chi-square check an automated gate that runs continuously, not a manual step someone remembers at the end. Add it to your pre-launch experiment QA checklist so it is verified every time.
- Be careful with redirects and uneven ramps. Prefer single-page variation over redirects when you can; when you cannot, expect and monitor a small imbalance. Ramp all variations together, and never adjust traffic mid-flight to "rebalance" counts — that corrupts the results further.
How Optimizely detects SRM automatically
Optimizely does not make you run this check by hand. Its Stats Engine A/B tests include automatic SRM detection with an Experiment Health indicator on the results page: a green Good status means no imbalance, and an SRM detected status means the test has failed a traffic check and needs investigation before you trust it.
One design detail matters for practitioners. Optimizely's detector does not use a one-shot chi-square test. It uses a sequential SRM (SSRM) method that checks traffic counts continuously as data arrives, so it can flag a problem within the first days of a launch — when stopping the test actually limits the damage — rather than only at the end. This is the same reasoning behind sequential testing and the peeking problem: a fixed-horizon test is only valid if you look once, whereas a sequential test is built to be monitored continuously. A retroactive end-of-experiment chi-square check has the opposite problem — by the time the totals even out, a transient mid-experiment SRM can be hidden entirely.
A few operational limits are worth knowing. Optimizely's automatic detection applies to Stats Engine A/B tests with Manual traffic distribution (not Stats Accelerator), running 45 days or less, with at least 1,000 visitors, and it evaluates only first decisions. It does not run on segmented results, and it does not check paused or archived experiments. So the manual chi-square check and the SRM Checker still earn their place: for Stats Accelerator tests, for segment-level diagnosis once an alarm fires, and for any experiment run on a platform without built-in detection.
Trust the split before you trust the lift
Sample ratio mismatch is the smoke detector of experimentation. It does not tell you what is on fire, but it tells you — reliably, and before you make an expensive decision — that you should not trust the numbers yet. The discipline is simple and non-negotiable: check the split with a chi-square test against a strict threshold, and if it fails, find and fix the cause rather than reasoning around it. A test with an SRM is not a weaker result; it is not a result at all.
Make it the first thing you look at, not the last. Before you read a single metric, run your observed counts through the SRM Checker. If the split holds, everything downstream — your lift, your confidence interval, your variance-reduced estimates — rests on solid ground. If it doesn't, you have saved yourself from shipping a decision built on a broken experiment.
