A/B Verdict — is your winner real?
What this does
You ran a test. One variant looks better. This decides whether that is a finding or a coin flip — and it runs the check that kills most reported winners.
Three things break an A/B decision. Most tools only catch the first two.
- Not enough trials. The gap looks real but the confidence intervals overlap completely.
- A real gap between groups. A two-proportion test catches this. Most tools stop here.
- The group label explains nothing. Variant group A "beat" group B, but inside group A the variants run from 10.5% down to 3.0%. When the spread inside a group is larger than the spread between groups, the label is not the explanation — something else is.
What you paste in
For each variant: a label, trials, and successes.
| variant | group | impressions | clicks |
|---|---|---|---|
| A1 | hook_type_A | 6253 | 647 |
| A2 | hook_type_A | 1859 | 55 |
| B1 | hook_type_B | 1101 | 112 |
| B2 | hook_type_B | 17010 | 245 |
Ad impressions and clicks, email sends and opens, sessions and signups, calls and bookings — anything with trials and successes works. A table, CSV, or pasted screenshot text is fine.
What you get back
- A Wilson 95% interval per variant, with its sample size beside it
- An overdispersion gate per group —
chi²/df, with a verdict - A two-proportion test between groups, but only if the gate passed
- One verdict line:
WINNER·UNDECIDED — need more trials·AXIS REJECTED·WRONG TEST - If undecided, the number of extra trials needed. Not a guess — the formula
A real case
Fourteen ad creatives, three hook types labelled by hand. The between-group test passed cleanly: type A beat type B by 1.84×, p ≈ 0.
Then the gate ran. Overdispersion came back 60.7, 24.7 and 12.6. Inside type A the creatives ranged 10.52% to 2.96%; inside type B, 8.15% to 2.17%. The spread inside each group was bigger than the gap between them.
The hypothesis was dead. The p-value had said go.
What it refuses to do
- It will not declare a winner on a p-value alone
- It will not score a hook or variant that has not run yet — that is a prediction, not a test
- It will not handle continuous outcomes (revenue per user, watch-time seconds). It will say so and stop
- It will not skip the gate because the gate rejected your favourite axis
- If you built the grouping axis after seeing the results, the whole verdict is labelled post-hoc and it says plainly that confirmation needs new data
Notes
Text and arithmetic only. No account connections, no data leaves the conversation, one pass in and out.
