API Guide¶

The FastAPI backend is a local artifact API over the runs/ directory. Most evaluations are run from the CLI; the dashboard also uses a local Run Lab endpoint to launch selected closed-loop demos. The API reads and updates files produced by RoboCI.

Start it with:

roboci serve

If the requested port is busy, RoboCI starts on the next available port and prints the selected URL. Use roboci serve --strict-port to fail instead.

Default base URL:

http://127.0.0.1:8000

Security Model¶

roboci serve is designed for local artifact review and binds to 127.0.0.1 by default. Do not bind it to 0.0.0.0, a public interface, or a shared network unless you put it behind trusted network controls such as a firewall, VPN, SSH tunnel, or authenticated reverse proxy.

The API reads run artifacts and exposes file-open helpers for the workstation serving those artifacts. Mutating endpoints require either a localhost browser Origin or X-RoboCI-Local: 1 for non-browser local clients.

For shared trusted networks, set ROBOCI_API_TOKEN before starting the API. When set, every /runs, /failures, /metrics, /scenarios, and /scenario-marketplace route requires either Authorization: Bearer <token> or X-RoboCI-Token: <token>. / and /health remain unauthenticated readiness endpoints. For the dashboard, also set NEXT_PUBLIC_ROBOCI_API_TOKEN in the frontend process; roboci dev copies ROBOCI_API_TOKEN into that frontend variable automatically when it starts both servers. This is a single shared token, not user accounts, TLS termination, multi-tenant authorization, rate limiting, or public internet hardening.

Interactive docs:

http://127.0.0.1:8000/docs

Health¶

GET /¶

Returns API metadata and links for common routes.

GET /health¶

Returns:

{"status":"ok"}

Runs¶

GET /runs¶

Returns a list of run summaries from runs/*/summary.json.

RoboCI enriches each summary with:

• run_type: closed_loop or openloop
• regression_status
• regression_count
• regression_delta_count
• assertions for closed-loop runs that include assertion aggregate counts
• provenance_summary when the run has provenance

Example:

[
  {
    "run_id": "2026-06-11_05-32-19",
    "schema_version": "1.0",
    "run_type": "closed_loop",
    "status": "pass",
    "success_rate": 1.0,
    "safety_score": 100.0,
    "collision_count": 0,
    "assertions": {
      "total": 2,
      "passed": 2,
      "failed": 0,
      "blocking_failed": 0
    },
    "provenance_summary": {
      "git_commit_sha": "abc123...",
      "git_branch": "main",
      "dirty_working_tree": false,
      "scenario_domains": ["ground_robotics"],
      "scenario_task_types": ["route_following"],
      "scenario_metric_groups": ["ground_robotics"],
      "scenario_pack_metadata": [
        {
          "id": "urban-v1",
          "name": "Urban Core",
          "domain": "ground_robotics",
          "version": "v1",
          "task_type": "route_following",
          "metric_group": "ground_robotics"
        }
      ],
      "agent_path": "examples/agents/rule_based_agent.py",
      "suite_path": "examples/suites/smoke.yaml",
      "dataset_path": null,
      "dataset_hash": null,
      "simulator_name": "mock",
      "python_version": "3.11.15",
      "roboci_version": "0.1.0",
      "created_at": "2026-06-14T06:14:44.133630Z",
      "scenario_file_count": 3,
      "dataset_sample_count": 0,
      "dataset_annotation_count": 0,
      "collection_error_count": 0
    },
    "regression_status": "pass",
    "regression_count": 0,
    "regression_delta_count": 0
  }
]

GET /runs intentionally returns only provenance_summary, not full summary.provenance. This keeps large scenario suites and open-loop datasets from bloating the run list response. Use GET /runs/{run_id} or GET /runs/{run_id}/artifacts/read?path=provenance.json for full file hashes and collection notes.

DELETE /runs¶

Clears existing run artifacts from the configured runs directory.

Response:

{"removed":2}

The endpoint removes run directories plus latest and latest.json pointers. Dashboard support directories such as _ui_suites are preserved. Mutating requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

GET /runs/launch-options¶

Returns the local scenarios, scenario packs, agents, and simulator options used by the dashboard Run Lab.

Scenario entries include tags. Installed marketplace scenarios without a tags field in their YAML are loaded with canonical marketplace tags when possible. Marketplace scenario entries also include pack_id, pack_name, pack_domain, pack_version, task_type, and metric_group when those values are known.

POST /runs/launch¶

Queues a selected closed-loop run from the dashboard and returns 202 Accepted immediately. The request chooses scenario ids or pack ids, an agent path, a simulator, and optional regression-gate behavior. agent_path and non-mock simulator paths may be absolute paths from /runs/launch-options or repo-relative paths that resolve to the same allowlisted files. Mutating requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

Response:

{
  "run_id": "launch-2026-06-23_10-15-00",
  "run_dir": "runs/launch-2026-06-23_10-15-00",
  "summary": {
    "run_id": "launch-2026-06-23_10-15-00",
    "status": "running",
    "total_scenarios": 20
  },
  "status_url": "/runs/launch-2026-06-23_10-15-00"
}

Poll GET /runs/{run_id} until summary.status changes from running to pass or fail. If background execution fails before normal artifacts are written, summary.json is updated with status: "fail" and an error field.

GET /runs/{run_id}¶

Returns:

{
  "summary": {},
  "metrics": {},
  "failures": [],
  "regression": {}
}

Unlike GET /runs, this endpoint returns the full summary.provenance object when the run has provenance.

Source files:

• summary.json
• provenance.json through summary.provenance
• metrics.json
• failures.json
• regression.json

GET /runs/{run_id}/scenarios¶

Returns the scenarios section from metrics.json.

Closed-loop scenario results include tags, which the dashboard uses for run-detail filtering. Marketplace scenario results also include pack_id, pack_name, pack_domain, pack_version, task_type, metric_group, and domain_metrics when those values are known. When a scenario defines assertions, each result also includes assertions and assertion_summary; assertion failures are reported alongside generic failures rather than replacing failures.json.

GET /runs/{run_id}/metrics¶

Returns the full metrics.json.

GET /runs/{run_id}/openloop¶

Returns open-loop aggregate metrics and prediction path metadata.

GET /runs/{run_id}/openloop/samples/{sample_id}¶

Returns one open-loop sample with source sample data, ground truth, prediction, and per-sample metrics when available.

GET /runs/{run_id}/regression¶

Returns regression.json.

GET /baselines¶

Returns named golden baselines from baselines/. Only directories containing baseline.json are returned; stray folders under baselines/ are ignored.

[
  {
    "name": "smoke",
    "source_run_id": "2026-06-14_10-00-00",
    "source_run_dir": "/path/to/workspace/runs/2026-06-14_10-00-00",
    "created_at": "2026-06-14T10:05:00+00:00",
    "description": "Known-good smoke run",
    "path": "baselines/smoke"
  }
]

POST /runs/{run_id}/regression/compare¶

Compares a run against a requested baseline, writes regression.json, and returns the generated regression result. Closed-loop comparisons also update failures.json lifecycle references when scenario status transitions prove a fix or regression: baseline failures resolved by the current run receive fixed_in_run_id, and current failures on previously successful scenarios receive regressed_in_run_id. Mutating requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients. Send either a legacy run id or a named baseline:

{
  "baseline_name": "smoke",
  "max_regression_percent": 5
}

{
  "baseline_run_id": "2026-06-14_10-00-00",
  "max_regression_percent": 5
}

GET /runs/{run_id}/artifacts¶

Returns an index of known artifacts for a run. Each item includes the artifact name, run-relative path, local path, size, and whether the dashboard may preview it as text.

This is a read-only index. Reports are generated by runners and explicit roboci report commands, not lazily from this GET handler.

The endpoint indexes these local files when present:

• summary.json
• provenance.json
• metrics.json
• failures.json
• regression.json
• config.yaml
• episode files by their concrete filename, such as <scenario_id>.jsonl or <scenario_id>.steps.jsonl
• timeline.json
• geojson
• timeline_events.json
• legacy events.json when a run has no matching timeline_events.json
• report.html
• report.md
• open-loop files under openloop/

Only these root-level text artifacts are previewable through the API:

• summary.json
• metrics.json
• regression.json
• failures.json
• config.yaml
• provenance.json

Other indexed artifacts, including episodes/*.jsonl, replay files, reports, and open-loop files, remain local-only for preview in the dashboard. Replay .geojson coordinates are simulator-local meters by default and include local CRS metadata. If a scenario declares environment.replay_scene.coordinate_transform with a WGS84 origin, RoboCI exports [longitude, latitude] GeoJSON with CRS84 metadata and includes the transform that produced it. The API does not expose arbitrary filesystem reads.

GET /runs/{run_id}/artifacts/read¶

Reads a safe text artifact preview.

Query parameters:

• path: one of the allowlisted root-level preview files above.
• max_lines: optional line cap, default 200, maximum 1000.

The response includes content, kind, line_count, truncated, size_bytes, and local_path. JSON content is returned as text so the frontend can pretty-print complete JSON previews. Large files return the first max_lines lines with truncated: true.

Path traversal and absolute paths are rejected. Non-allowlisted run artifacts return 403 instead of file contents.

POST /runs/{run_id}/artifacts/open¶

Asks the local API process to open an indexed run artifact with the host OS default app. This is for local dashboard use only and requires either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

Query parameters:

• path: an artifact path returned by GET /runs/{run_id}/artifacts.

The endpoint does not read file contents. Path traversal and absolute paths are rejected, and files that are not part of the artifact index return 403.

Replay¶

GET /runs/{run_id}/replay/{scenario_id}¶

Returns:

{
  "timeline": {
    "scenario_id": "scenario_001",
    "frames": [
      {
        "t": 12.4,
        "frame_index": 124,
        "events": [
          {
            "type": "collision",
            "timestamp": 12.4,
            "frame_index": 124
          }
        ]
      }
    ],
    "events": [
      {
        "type": "collision",
        "timestamp": 12.4,
        "frame_index": 124
      }
    ]
  },
  "geojson": {},
  "events": [
    {
      "type": "scenario_start",
      "timestamp": 0.1,
      "frame_index": 0,
      "severity": "info",
      "title": "Scenario started",
      "description": "Scenario scenario_001 replay started."
    },
    {
      "type": "collision",
      "timestamp": 12.4,
      "frame_index": 124,
      "severity": "critical",
      "title": "Collision",
      "description": "Collision detected involving actor pedestrian_1.",
      "actor_id": "pedestrian_1",
      "related_actor_id": "pedestrian_1"
    }
  ]
}

The top-level events field is the normalized event timeline from replay/<scenario_id>.timeline_events.json. New runs write only this canonical event file. The API falls back to replay/<scenario_id>.events.json for older runs that do not have timeline_events.json.

The dashboard scenario replay route accepts deep-link query parameters:

/runs/{run_id}/scenarios/{scenario_id}?t={timestamp}&frame={frame_index}

When both are present, frame is used as the exact target and t remains useful for older replay artifacts or manual links.

Source files:

• replay/<scenario_id>.timeline.json
• replay/<scenario_id>.geojson
• replay/<scenario_id>.timeline_events.json

Failures¶

GET /failures¶

Aggregates all runs/*/failures.json files. Optional query parameters:

• status: filters by lifecycle status. Allowed values are open, acknowledged, assigned, in_progress, fixed, verified, regressed, and wont_fix.
• owner: filters by owner. Use an empty value, /failures?owner=, for unassigned failures.

Closed-loop failure objects include replay location fields when possible:

• timestamp: incident time in seconds.
• frame_index: zero-based frame index in the replay timeline.
• replay_path: relative path to the timeline artifact.
• replay_url: dashboard route that opens the scenario replay at the incident moment. It is stored as a relative path so artifacts do not assume a dashboard host or port.

GET /failures/{failure_id}¶

Returns one failure object.

POST /failures/{failure_id}/tags¶

Updates tags.

Unknown tags are dropped with a warning instead of making existing failures.json unreadable. Known tags are normalized and de-duplicated in their submitted order.

Body:

["collision", "needs_review"]

POST /failures/{failure_id}/notes¶

Updates notes.

Body:

{"notes":"Agent braked too late."}

POST /failures/{failure_id}/status¶

Updates status and appends a status_history entry when the value changes.

Body:

{"status":"fixed"}

PATCH /failures/{failure_id}¶

Updates one or more editable failure fields: tags, notes, status, owner, fixed_in_run_id, and regressed_in_run_id. Mutating failure requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

Status changes append entries to status_history in the same failures.json write. Re-saving the current status does not create a duplicate history entry. Send null for owner, fixed_in_run_id, or regressed_in_run_id to clear those optional fields.

Scenarios¶

GET /scenarios¶

Loads example scenarios from examples/scenarios. Each returned scenario includes optional tags when defined in YAML.

Set ROBOCI_EXAMPLES_DIR if examples live elsewhere.

GET /scenario-marketplace¶

Returns scenario pack metadata from the packaged catalog and workspace manifests at examples/scenario_marketplace/*/pack.yaml. Pack scenario metadata includes generated tags used by the Scenarios dashboard page for marketplace filtering. Pack metadata is domain-aware and includes backward-compatible existing fields plus domain, version, license, install_status, installed, versioned_install_status, versioned_installed, supported_tasks, metrics, metric_group, adapter_requirements, simulator_requirements, and domain_metadata. Legacy packs without a declared domain are returned as ground_robotics.

GET /scenario-marketplace/{pack_id}¶

Returns metadata for one scenario pack.

POST /scenario-marketplace/{pack_id}/install¶

Installs the named scenario pack into examples/scenario_marketplace/<pack-id>/ and writes examples/suites/<pack-id>.yaml. Add ?versioned=true to install into examples/scenario_marketplace/<pack-id>--<version>/ and write a matching version-suffixed suite. The response includes the install result and refreshed pack metadata, including install_status: "installed". Mutating requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

DELETE /scenario-marketplace/{pack_id}/install¶

Removes generated scenarios and the matching suite from the current workspace. Workspace-local pack.yaml manifests are preserved. Add ?versioned=true to remove the version-suffixed install. The response includes the remove result and refreshed pack metadata, including install_status: "available" when no generated scenarios remain. Mutating requests require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

Metrics¶

GET /metrics/trends¶

Returns ordered closed-loop trend data across the last N run summaries. Query parameter limit defaults to 20 and is capped at 200.

{
  "limit": 20,
  "total_runs": 3,
  "runs": [
    {
      "run_id": "2026-06-11_05-32-19",
      "run_dir_id": "2026-06-11_05-32-19",
      "created_at": "2026-06-11T05:32:19Z",
      "status": "pass",
      "success_rate": 1.0,
      "safety_score": 100.0,
      "collision_count": 0,
      "duration_seconds": 0.007,
      "run_index": 1
    }
  ]
}

Open-loop runs are excluded because they do not carry closed-loop success_rate, safety_score, and collision_count fields.

Path Resolution¶

The API uses workspace-aware path resolution:

1. If ROBOCI_RUNS_DIR is set, use that.
2. If runs/ exists in the current directory, use it.
3. Otherwise, walk up parent directories and use the first runs/ found.

This prevents empty dashboards when the API is launched from frontend/.