Skip to main content

Holdouts & Incrementality

Holdouts and incrementality testing create persistent control groups, geo-holdouts, and switchback tests to measure the true incremental impact of changes.

Overview​

Holdouts maintain control groups that never receive treatment, allowing you to measure the true incremental impact of changes by comparing treated users to untreated users.

Holdout Types​

Persistent Holdouts​

Maintain control groups over time:

{
  "holdout_type": "persistent",
  "control_percentage": 10,
  "assignment": "user_based",
  "persistence": "permanent"
}

Geo-Holdouts​

Geographic control groups:

{
  "holdout_type": "geo",
  "control_regions": ["region_a", "region_b"],
  "treatment_regions": ["region_c", "region_d"]
}

Switchback Tests​

Alternate between treatment and control:

{
  "holdout_type": "switchback",
  "pattern": "weekly",
  "control_weeks": [1, 3, 5],
  "treatment_weeks": [2, 4, 6]
}

Incrementality Measurement​

Measure true incremental impact:

{
  "incrementality_measurement": {
    "control_group": "holdout_10_percent",
    "treatment_group": "full_treatment",
    "metrics": [
      "incremental_conversions",
      "incremental_revenue",
      "incremental_margin"
    ],
    "attribution_window_days": 30
  }
}

Best Practices​

  1. Maintain holdouts: Keep control groups stable over time
  2. Measure incrementality: Focus on incremental impact, not absolute metrics
  3. Use geo-holdouts: When user-level holdouts aren't feasible
  4. Account for spillover: Consider cross-group effects
  5. Run long enough: Ensure sufficient data for measurement