ab-testing - marketingskills online

Got an A/B test idea, but not sure how to run it properly?

Less guessing, less wasted traffic, and a test plan your team can review before going live.

What It Does

This skill turns an A/B test idea into a pre-launch experiment control document.

It is built for product, growth, marketing, and founder teams that already have a change they want to test, such as a pricing page change, landing page message, signup flow, onboarding step, CTA, layout, or product experience.

Its job is not to suggest random CRO ideas. Its job is to decide whether the experiment is defined tightly enough, measurable enough, and powered enough to trust before it goes live.

It produces the core controls of a valid test:

• A hypothesis in proper experiment format
• Control and variant framing
• One primary metric and supporting secondary metrics
• Guardrail metrics that define what must not get worse
• Sample size calculated from the user's real baseline and MDE
• Duration estimate based on available traffic
• Traffic allocation and implementation notes
• Pre-launch QA checklist
• Decision rules for winner, loser, mixed, and inconclusive results
• Result report structure for analysis and documentation

What Makes It Different

This is not an A/B testing advice checklist.

It has built-in controls that make the test harder to fool:

1. It runs sample size math instead of hand-waving it.

When the user provides a baseline rate, minimum detectable effect, confidence level, power, and traffic volume, the skill uses the included Python sample size calculator. It calculates required sample size per variation, total sample size, expected variant conversion rate, absolute lift, relative lift, and estimated duration when daily traffic is available.

That means the plan can say, before launch, whether the test has enough traffic to detect the change the team cares about.

2. It rejects weak tests before they waste traffic.

If the required sample size is too high, the expected duration is too long, or the MDE is unrealistic, the skill flags the problem. It can recommend a larger change, fewer variants, a higher-traffic conversion point, more baseline measurement, or a qualitative decision path instead of pretending every idea deserves an A/B test.

3. It uses guardrail metrics to catch harmful wins.

A variant is not treated as good just because the primary metric improves. The plan defines metrics that should not get worse, such as refund rate, support tickets, lead quality, retention, revenue quality, downstream conversion, or user complaints.

This prevents the classic fake win: more clicks, worse customers.

4. It locks the decision method before launch.

The skill documents the hypothesis, metric hierarchy, sample size, MDE, analysis timing, and decision rules before results exist. That reduces peeking, early stopping, segment cherry-picking, and post-hoc metric switching.

5. It forces implementation and tracking checks.

The pre-launch QA checklist covers variant rendering, assignment consistency, tracking events, metric definitions, traffic split, device/browser checks, and stakeholder review. A test with broken instrumentation is not treated as a valid experiment.

How It Works

The skill starts by collecting the inputs that determine whether the experiment is testable:

1. What page, feature, flow, or campaign is being tested
2. What change is being proposed
3. The current baseline conversion, retention, or action rate
4. The minimum detectable effect worth caring about
5. Daily or weekly traffic available for the test
6. Testing tool, implementation method, and tracking constraints

It then uses the included sample-size guide and Python calculator to check feasibility. If the test is viable, it builds the experiment brief. If the test is not viable, it explains exactly why: not enough traffic, effect size too small, too many variants, weak metric definition, missing baseline, or tracking risk.

The output follows the built-in templates for experiment briefs, stakeholder updates, result reports, and test repositories. The structure is fixed enough to keep teams from improvising the methodology after the test starts.

What The Output Looks Like

Users receive an experiment document that can be reviewed before launch and used again after results come in.

The output can include:

• A hypothesis using the "Because / we believe / will cause / we'll know" structure
• Control and variant definitions with one clear variable
• Primary, secondary, and guardrail metrics
• Sample size inputs and calculator output
• Required sample size per variant and total sample size
• Duration estimate based on real traffic
• Feasibility warning when the test is underpowered or too slow
• Traffic allocation recommendation
• Tracking and QA checklist
• Pre-committed decision rules
• Result report structure with confidence intervals, effect size, caveats, decision, learnings, and next tests

Typical Use Cases

• Plan an A/B test for a landing page, pricing page, signup form, onboarding flow, checkout step, or product feature
• Check whether there is enough traffic to run a test before committing design or engineering work
• Turn a debate about copy, layout, CTA, pricing, or onboarding into a measurable experiment with fixed decision rules
• Find out that a proposed test is underpowered before wasting weeks of traffic
• Decide whether to run the full test, reduce variants, test a larger change, use a higher-traffic metric, or gather qualitative evidence instead
• Create a reusable experiment brief for product, growth, marketing, analytics, and engineering teams
• Prepare a results report after the test reaches its planned sample size

Example

Input

We want to test a shorter signup form. Baseline signup completion is 5.2%, we get 1,800 visitors per week, and we care about detecting at least a 20% lift. We can split traffic 50/50 in PostHog. Help me design the experiment.

Output

A structured A/B test plan with a hypothesis, control and variant definitions, primary and guardrail metrics, a sample-size calculation using the 5.2% baseline and 20% MDE, an estimated duration based on available traffic, a warning if the test is underpowered, a pre-launch QA checklist, success criteria, and a results analysis template.

Inputs It Handles

Users can provide:

• The page, feature, flow, or campaign to test
• The proposed change
• Current baseline conversion or retention rate
• Minimum detectable effect
• Traffic volume
• Testing tool or implementation method
• Known tracking constraints
• Any previous test history

If key numbers are missing, the skill identifies what must be measured first and explains whether the team should delay the test, use a larger effect size, test upstream where traffic is higher, or make a decision using qualitative evidence instead.

Best For

• Growth teams that need statistically valid conversion experiments
• Product teams validating feature, onboarding, or UX changes
• Marketing teams testing landing pages, pricing pages, campaign flows, and CTAs
• Founders who want to avoid false wins from noisy data
• Teams that want the experiment method locked before results create pressure to bend it