Regression System¶

RoboCI regression comparison answers:

Did the current run get worse than a baseline run?

Files¶

Main implementation:

roboci/regression/comparator.py
roboci/regression/gates.py
roboci/regression/baseline.py

Dashboard display:

frontend/components/RegressionDiff.tsx
frontend/app/regressions/page.tsx

Gates vs Regression Comparison¶

RoboCI has two related concepts:

Gates¶

Gates are absolute thresholds.

Example:

gates:
  min_success_rate: 0.95
  max_collisions: 0
  min_safety_score: 80

If a run violates a gate, the run can fail even without a baseline.

Regression Comparison¶

Regression comparison is relative to a baseline.

Example:

roboci baseline create smoke --from runs/latest --description "Known-good smoke run"
roboci baseline promote runs/latest smoke
roboci compare --current runs/latest --baseline-name smoke

Named baselines are stored under baselines/<name>/. Each baseline is a copied run directory with a baseline.json metadata file:

{
  "name": "smoke",
  "source_run_id": "2026-06-14_10-00-00",
  "source_run_dir": "/path/to/workspace/runs/2026-06-14_10-00-00",
  "created_at": "2026-06-14T10:05:00+00:00",
  "description": "Known-good smoke run",
  "path": "baselines/smoke"
}

roboci baseline promote <run-dir> <name> is a positional baseline-promotion shortcut for the same named baseline flow. Baseline names are unique unless roboci baseline create <name> --from <run> --overwrite or roboci baseline promote <run> <name> --overwrite is used. --overwrite replaces the named baseline from a different source run; RoboCI rejects attempts to overwrite a baseline from its own directory so the source is not deleted. roboci baseline list and GET /baselines only include directories that contain baseline.json, so stray folders under baselines/ are ignored.

Path-based comparisons remain supported with roboci compare --current runs/latest --baseline baselines/smoke.

This writes metric deltas to:

runs/<current>/regression.json

Metric Direction¶

The comparator needs to know whether higher or lower is better.

Lower is better:

• collision_count
• offroad_count
• trajectory_ade
• trajectory_fde
• miss_rate
• control_l1_error
• prediction_latency_ms

Higher is better:

• success_rate
• safety_score
• route_completion_percent

regression.json¶

Example:

{
  "status": "fail",
  "failures": [
    "safety_score regressed by 8.10%"
  ],
  "deltas": {
    "safety_score": {
      "current": 91.9,
      "baseline": 100.0,
      "delta": -8.1,
      "percent": -8.1,
      "direction": "higher_is_better",
      "regressed": true
    }
  }
}

When a numeric baseline is 0 and the current value is nonzero, percent is null because a percent change from zero is undefined. Regression detection handles that case explicitly with a minimum absolute delta instead of reporting a fake 100% change.

Why Regression Diff May Be Empty¶

Normal run commands write regression.json with empty deltas:

{
  "status": "pass",
  "failures": [],
  "deltas": {}
}

That means no baseline comparison has been run yet.

To populate deltas:

roboci baseline create smoke --from runs/<baseline-run-id>
roboci compare --current runs/latest --baseline-name smoke

Dashboard Behavior¶

The API enriches run summaries with:

• regression_status
• regression_count
• regression_delta_count

The dashboard uses those fields for overview cards and run tables.

The Regression Explorer reads the full regression.json for runs with regression artifacts and groups real artifact data into:

• New regressions.
• Resolved regressions.
• Recurring regressions.
• Top degraded scenarios.
• Top improved scenarios.
• Critical safety regressions.

Closed-loop and open-loop signals are shown in separate sections. Metric cards show the previous value, current value, delta, severity, and affected run or sample scope. Scenario cards show the affected scenario, safety score delta when present, and status transitions when the scenario exists only on one side of the comparison.

Replay links are shown only when a current closed-loop scenario exists. Failure links are shown only when a matching failure artifact exists for the card's scenario or sample; the explorer does not fall back to unrelated failures.

Closed-loop comparisons also backfill failure lifecycle references in the compared artifacts. Resolved baseline failures receive the current run id in fixed_in_run_id, while current-run failures on previously successful scenarios receive the current run id in regressed_in_run_id. RoboCI preserves manual terminal decisions such as verified and wont_fix.

The detailed diff is loaded from:

GET /runs/{run_id}

which includes:

{
  "regression": {
    "status": "pass",
    "failures": [],
    "deltas": {}
  }
}

Adding New Regression Metrics¶

1. Ensure the metric is written to summary.json, metrics.json, or openloop/aggregate_metrics.json.
2. Add the metric to LOWER_IS_BETTER or HIGHER_IS_BETTER in roboci/regression/comparator.py.
3. Add tests for the new direction.
4. Confirm the dashboard can render the delta.