Architecture

RoboCI is organized around a simple local-first contract:

  1. Commands create evaluation runs.
  2. Runs write structured artifacts under runs/<run_id>/.
  3. Reports, regression checks, the FastAPI backend, and the dashboard read those artifacts.

The MVP uses file-based storage. This keeps CI usage simple and makes every run inspectable with ordinary tools.

System Diagram

flowchart LR CLI["Typer CLI\nroboci ..."] --> ClosedLoop["Closed-loop Runner"] CLI --> OpenLoop["Open-loop Runner"] CLI --> Compare["Regression Comparator"] ClosedLoop --> Sim["SimulatorAdapter\nMockSimulator"] ClosedLoop --> Agent["Agent\nPythonAgentLoader"] ClosedLoop --> Metrics["Closed-loop Metrics"] ClosedLoop --> Assertions["Scenario Assertions"] ClosedLoop --> Replay["Replay Exporters"] ClosedLoop --> Failures["Failure Detector"] OpenLoop --> Dataset["OpenLoopDataset"] OpenLoop --> Model["OpenLoopModel"] OpenLoop --> OLMetrics["Open-loop Metrics"] Metrics --> Runs["runs/run_id artifacts"] Assertions --> Runs Replay --> Runs Failures --> Runs OLMetrics --> Runs Compare --> Runs Runs --> Report["Static HTML Report"] Runs --> API["FastAPI Backend"] API --> Dashboard["Next.js Dashboard"]

Major Components

CLI

The CLI lives in roboci/cli/.

  • main.py defines the Typer command surface.
  • commands.py adapts command-line inputs into core Python calls.

The CLI is intentionally thin. It should parse arguments, call domain logic, print concise status, and set exit codes.

Closed-loop Runner

The closed-loop runner lives in roboci/core/runner.py.

It does this for each scenario:

  1. Load and validate the scenario.
  2. Create a simulator.
  3. Reset simulator state.
  4. Ask the agent for an action.
  5. Step the simulator.
  6. Append a frame to episode JSONL.
  7. Stop on success, failure, or timeout.
  8. Compute metrics.
  9. Export replay artifacts.
  10. Detect failures.
  11. Collect run provenance.
  12. Write summary, provenance, and report artifacts.

Open-loop Runner

The open-loop runner lives in roboci/openloop/runner.py.

It does this for each sample:

  1. Load dataset metadata and samples.
  2. Attach ground-truth annotations.
  3. Call model prediction.
  4. Measure prediction latency.
  5. Compute per-sample metrics.
  6. Aggregate metrics.
  7. Evaluate open-loop gates.
  8. Collect model and dataset provenance.
  9. Write open-loop artifacts.

Storage

The storage layer lives in roboci/storage/.

  • artifacts.py contains JSON and JSONL helpers.
  • paths.py resolves workspace-relative paths.
  • run_store.py creates and lists run directories. Run detail reads full summary.provenance, while run lists expose a compact provenance_summary to avoid shipping every source-file hash.

The most important design choice is that RunStore("runs") resolves runs/ from the current working directory or a parent workspace. This prevents the API from returning no runs when launched from frontend/.

ROBOCI_RUNS_DIR can point at a local path or mounted network share. roboci storage sync copies run artifacts to local/network paths, S3 via aws s3 sync, or GCS via gsutil -m rsync -r; the core API still reads a filesystem path so runs remain inspectable with ordinary tools.

API

The API lives in roboci/api/.

It is a read/write façade over local artifacts:

  • Reads summaries, provenance, metrics, regression files, replays, scenarios, and failures.
  • Updates failure tags, notes, lifecycle status, owner, fixed-in run, and regressed-in run fields.
  • Appends status history in failures.json when a failure moves through the debugging workflow.
  • Does not own core evaluation logic.

Frontend

The dashboard lives in frontend/.

It fetches from the FastAPI backend through frontend/lib/api.ts, then renders:

  • Dashboard KPIs.
  • Run lists and details.
  • Replay and trajectory views.
  • Regression Explorer, compare workbench, and run-level regression diffs.
  • Failure triage surfaces.

Run Artifact Flow

Closed-loop run:

sequenceDiagram participant User participant CLI participant Runner participant Simulator participant Store participant API participant UI User->>CLI: roboci run --suite ... --agent ... --sim mock CLI->>Runner: run_suite() Runner->>Simulator: reset(scenario) loop Each timestep Runner->>Simulator: step(action) Runner->>Store: append episode frame end Runner->>Store: write summary, metrics, failures, replay, report UI->>API: GET /runs API->>Store: read runs/run_id/summary.json API-->>UI: run summaries

The dashboard run detail page also requests GET /runs/{run_id}/artifacts to build a local artifact tree. The API returns an index of known run files, but text preview is intentionally narrower: GET /runs/{run_id}/artifacts/read?path=... only reads allowlisted root artifacts (summary.json, metrics.json, regression.json, failures.json, config.yaml, and provenance.json). Other indexed files remain local-only for preview; POST /runs/{run_id}/artifacts/open?path=... can ask the local API process to launch only indexed artifacts with the host default app, so the dashboard does not become arbitrary filesystem access.

Open-loop run:

sequenceDiagram participant User participant CLI participant Runner participant Dataset participant Model participant Store User->>CLI: roboci openloop run --dataset ... --model ... CLI->>Runner: run(dataset, task) Runner->>Dataset: samples() loop Each sample Runner->>Model: predict(sample) Runner->>Runner: compute metrics end Runner->>Store: write predictions, per-sample metrics, aggregate metrics, report

Design Principles

  • Local-first: Runs are ordinary files, not remote records.
  • Adapter-based: Simulators and agents are replaceable.
  • Small interfaces: SimulatorAdapter, Agent, and OpenLoopModel are intentionally minimal.
  • Artifact-driven UI: The dashboard reads the same files the CLI writes.
  • CI-friendly: Commands use exit codes and write HTML plus PR-friendly markdown reports.