Designed a clean A/B test for extending our trial, with sample size, guardrails and decision rules upfront

You are the head of experimentation at a leading tech company (like Booking.com or Netflix). Help me design a rigorous business experiment to test a strategic hypothesis.\n\nHYPOTHESIS TO TEST: {{hypothesis_statement — e.g., 'Increasing trial period from 7 to 14 days will increase paid conversion by 20%'}}\nBUSINESS CONTEXT: {{context — product, market, current metrics}}\nDECISION DEPENDENT ON RESULT: {{what_will_you_do_based_on_the_outcome}}\nEXPERIMENT TYPE PREFERENCE: {{type — e.g., 'A/B test', 'Multivariate', 'Cohort', 'Pilot', 'Natural experiment'}}\nRISK TOLERANCE: {{risk_level — e.g., 'Low risk only', 'Moderate', 'Aggressive'}}\n\nOUTPUT — Complete Experiment Design Document:\n\n## 1. HYPOTHESIS REFINEMENT\nConvert the business hypothesis into a rigorous, testable format:\n\n**Null Hypothesis (H₀)**: [State the 'no effect' version]\n**Alternative Hypothesis (H₁)**: [State the predicted effect version]\n**Directional vs. Non-directional**: [One-tailed or two-tailed test?]\n\n**SMART Hypothesis Check**:\n| Criterion | Assessment | Pass? |\n|-----------|-----------|-------|\n| Specific | Is the effect clearly defined? | |\n| Measurable | Can we measure the outcome precisely? | |\n| Actionable | Will the result drive a decision? | |\n| Realistic | Is the expected effect plausible? | |\n| Time-bound | Can we get results in a reasonable timeframe? | |\n\n## 2. EXPERIMENTAL DESIGN\n### Design Type: [A/B / Multivariate / Before-After / Interrupted Time Series / Synthetic Control]\n\n**Treatment Group**:\n- Definition: Who/what gets the intervention?\n- Sample size: [N] — with power calculation justification\n- Treatment details: Exactly what changes?\n- Duration: How long will treatment last?\n\n**Control Group**:\n- Definition: Who/what doesn't get the intervention?\n- Sample size: [N]\n- Control condition: What do they experience instead?\n- Why this control: [Hold-out / Placebo / Status quo / Historical]\n\n**Randomization Strategy**:\n- Unit of randomization: [User / Session / Geographic / Time-based / Cluster]\n- Method: [Simple / Stratified / Cluster / Matched pair]\n- Stratification variables (if any):\n\n## 3. SUCCESS METRICS FRAMEWORK\n\n| Metric Type | Metric Name | Formula | Current Baseline | Target | Measurement Method |\n|-------------|-------------|---------|-----------------|--------|-------------------|\n\n**Primary Metric** (1 only — the decision metric):\n- Why this metric: [Directly tied to business outcome]\n- Minimum Detectable Effect (MDE): [X%] — the smallest effect worth detecting\n\n**Guardrail Metrics** (2-4 — must not degrade significantly):\n- e.g., Revenue per user, Customer satisfaction, Churn rate, Page load time\n\n**Secondary Metrics** (2-4 — exploratory, generate follow-up hypotheses):\n\n## 4. SAMPLE SIZE & POWER CALCULATION\n- Significance level (α): 0.05 (standard) or {{custom}}\n- Statistical power (1-β): 0.80 (standard) or {{custom}}\n- Minimum Detectable Effect: {{X%}} or calculate based on baseline\n- Required sample size per variant: [N]\n- Total sample needed: [N]\n- Estimated experiment duration to reach sample: [Days]\n- If sample is insufficient: Recommendations (sequential testing? lower power? larger MDE?)\n\n## 5. STATISTICAL ANALYSIS PLAN\n- Primary test: [t-test / chi-square / Mann-Whitney U / OLS regression / Bayesian]\n- Multiple comparison correction (if testing multiple metrics): [Bonferroni / Holm-Bonferroni / False Discovery Rate]\n- Segmentation analysis: Pre-planned subgroup analyses (avoid data dredging)\n- Covariates to control for: [List]\n\n## 6. EXPERIMENT PROTOCOL\n### Timeline:\n| Phase | Activity | Duration | Owner | Deliverable |\n|-------|----------|----------|-------|-------------|\n\n### Pre-launch Checklist:\n- [ ] QA on treatment delivery\n- [ ] Data pipeline validation\n- [ ] Sample size validation\n- [ ] Stakeholder alignment on early stopping rules\n\n### During Experiment:\n- Monitoring cadence: [Daily / Weekly]\n- Interim peeking rules: [No peeking / Group sequential / Always valid p-values]\n- Early stopping criteria: [Success boundary / Harm boundary / Futility]\n\n## 7. RESULTS INTERPRETATION FRAMEWORK\nBefore seeing results, define:\n- **Win**: Primary metric improvement statistically significant AND guardrails OK → Full rollout\n- **Loss**: Statistically significant harm on primary or guardrail metrics → Stop and investigate\n- **Inconclusive**: Not statistically significant but directionally positive → Iterate and retest with changes\n- **Flat**: No meaningful direction → Consider different hypothesis\n\n## 8. POST-EXPERIMENT PLAN\n- Rollout strategy (if win): Phased vs. immediate, monitoring period\n- Rollback plan (if loss): How to reverse treatment\n- Follow-up experiments generated by results\n- Learning documentation template