Codebase Guide

This guide explains the main files and their responsibilities.

Python Package

roboci/cli

main.py

Defines the Typer application:

  • roboci init
  • roboci run
  • roboci compare
  • roboci report
  • roboci openloop run
  • roboci openloop compare
  • roboci serve

The functions here should stay small. They declare arguments and pass them to commands.py.

commands.py

Adapts CLI calls into domain logic:

  • Loads Python agents and models.
  • Creates runners.
  • Prints run locations.
  • Converts failed gates into non-zero CLI exit codes.
  • Starts Uvicorn for the API.

roboci/core

runner.py

Owns closed-loop execution.

Important class:

class ClosedLoopRunner:
    def run_suite(...)
    def run_scenario(...)

run_suite() writes run-level artifacts:

  • config.yaml
  • summary.json
  • provenance.json
  • metrics.json
  • failures.json
  • regression.json
  • report.html
  • report.md

roboci report can regenerate either static report for an existing run.

run_scenario() writes scenario-level artifacts:

  • episodes/<scenario_id>.jsonl
  • replay/<scenario_id>.timeline.json
  • replay/<scenario_id>.geojson
  • replay/<scenario_id>.timeline_events.json

After closed-loop metrics are computed, the runner evaluates optional scenario assertions from roboci/assertions. Assertion results are embedded in each scenario entry in metrics.json; aggregate assertion counts are written to both metrics.json and summary.json. Blocking assertion failures (critical and high) mark the scenario as failed.

engine.py

A small convenience wrapper for running a suite from Python code.

types.py

Shared Pydantic models and enums:

  • RunStatus
  • EpisodeStatus
  • RunPaths
  • ScenarioResult
  • RunSummary

errors.py

Project-specific exceptions:

  • RoboCIError
  • ScenarioError
  • SuiteError
  • AgentLoadError
  • RegressionFailed

roboci/simulators

base.py

Defines the simulator interface:

class SimulatorAdapter:
    def connect(self, config: dict) -> None: ...
    def reset(self, scenario: dict) -> dict: ...
    def step(self, action: dict) -> dict: ...
    def is_done(self) -> bool: ...
    def close(self) -> None: ...

All simulator integrations must implement this interface; SimulatorAdapter is an ABC and cannot be instantiated without all five methods.

mock.py

A deterministic simulator used for local and CI tests.

It models:

  • Ego kinematics.
  • Actor constant-velocity motion.
  • Drone z/3D velocity state.
  • Manipulator joint, gripper, and end-effector state.
  • Goal detection.
  • Collision and near-miss events.
  • Offroad detection.
  • Timeout detection.

carla.py

CARLA adapter surface. It connects to a running CARLA server, honors ROBOCI_CARLA_HOST and ROBOCI_CARLA_PORT, and has a Docker Compose smoke profile for examples/suites/carla.yaml. Use --sim mock for default CI and use the carla Compose profile on prepared CARLA runners.

roboci/adapters/simulators

Compatibility import layer for older code that imported simulator adapters from roboci.adapters.simulators.*. The implementation source of truth is roboci/simulators/*; files under roboci/adapters/simulators/ are thin re-export shims and should not grow adapter logic.

roboci/agents

base.py

Defines the closed-loop agent interface:

class Agent:
    def act(self, observation: dict) -> dict: ...

random_agent.py

Seeded random-control baseline. The example uses a fixed seed so repeated runs are reproducible; it is not intended to produce different stochastic behavior on every run.

python_agent.py

Loads user-provided Python files.

Supported closed-loop hooks:

  • create_agent()
  • Agent
  • act(observation)

Supported open-loop hooks:

  • create_model()
  • Model
  • predict(sample)
  • Agent hooks adapted into model predictions

roboci/scenarios

schema.py

Pydantic schema for scenario YAML.

Main models:

  • Pose
  • Goal
  • EgoConfig
  • ActorConfig
  • AssertionConfig
  • SuccessCriteria
  • FailureCriteria
  • Scenario

Scenario includes optional tags, validated as lowercase slug-style strings, and optional assertions for deterministic post-episode behavioral checks.

roboci/assertions

engine.py

Evaluates scenario assertions against episode frames and closed-loop metrics.

Supported assertion types:

  • no_collision
  • min_distance
  • max_speed
  • max_speed_when_actor_near
  • goal_reached

The engine returns JSON-native result dictionaries with passed, status, actual, expected, and message fields. critical and high failures are blocking. max_speed_when_actor_near returns not_applicable when no matching actor is observed, and not-applicable assertions are excluded from pass/fail/blocking counts.

loader.py

Reads YAML and validates it into a Scenario. It also backfills canonical marketplace tags for older installed scenario YAML files under examples/scenario_marketplace/<pack-id>/ when those files do not yet include tags.

validator.py

Adds additional semantic checks such as positive duration and timestep.

roboci/suites

schema.py

Pydantic schema for suite YAML and closed-loop gates.

loader.py

Loads suite YAML and resolves scenario paths relative to the suite file.

roboci/metrics

registry.py

Collects all closed-loop metrics into one dictionary.

safety.py

Computes:

  • collision_count
  • offroad_count
  • min_distance_to_actor
  • hard_brake_count
  • safety_score

comfort.py

Computes acceleration, deceleration, jerk, and steering-rate metrics.

task.py

Computes goal and route completion metrics.

openloop.py

Re-exports open-loop metrics from roboci/openloop/metrics.py.

roboci/openloop

model.py

Defines:

class OpenLoopModel:
    def predict(self, sample: dict) -> dict: ...

dataset.py

Loads dataset metadata, JSON samples, and optional annotations.

metrics.py

Computes:

  • trajectory_ade
  • trajectory_fde
  • miss_rate
  • control_l1_error
  • steering_error
  • throttle_error
  • brake_error
  • occupancy_iou from matching occupancy grids
  • detection_map from prediction and ground-truth detection boxes

runner.py

Runs a model over all samples, writes predictions and metrics, and evaluates open-loop gates.

Open-loop runs also collect provenance for the model binding and dataset inputs, including metadata, sample, annotation, and composite dataset hashes.

roboci/regression

comparator.py

Loads current and baseline metrics, computes deltas, and writes regression.json.

Metrics are interpreted by direction:

  • Lower is better: collisions, ADE, FDE, latency.
  • Higher is better: success rate, safety score, route completion.

gates.py

Evaluates closed-loop and open-loop gate thresholds.

baseline.py

Manages named golden baselines under baselines/<name>/. BaselineStore creates and promotes run-directory copies with baseline.json metadata, lists only directories that contain that metadata marker, resolves runs/latest to the concrete source run path, and guards --overwrite from deleting a baseline when the source is the same target directory.

roboci/replay

timeline.py

Converts episode frames to timeline JSON. Timeline frames and events include zero-based frame_index values so UI links can target an exact replay frame. Also derives normalized timeline events for the dashboard from the recorded episode plus assertion results. These include readable titles/descriptions for scenario start, goal reached, collision, near miss, offroad, hard brake, timeout, and assertion failure events.

geojson.py

Converts ego and actor trajectories to GeoJSON replay artifacts. Coordinates are simulator-local meters by default, with local CRS metadata so downstream tools do not treat them as GPS. Scenarios can opt into a local-tangent-plane WGS84 transform to export [longitude, latitude] replay coordinates for map-native AV workflows.

exporter.py

Writes timeline, GeoJSON, and normalized timeline events artifacts. Older runs may still contain the legacy events.json alias, and the replay API can read it as a fallback.

roboci/failures

schema.py

Pydantic model for a failure record. Closed-loop failure records may include timestamp, frame_index, replay_path, and replay_url for replay deep links. Failure records also carry lifecycle triage fields: status, optional owner, optional fixed_in_run_id, optional regressed_in_run_id, and status_history.

detector.py

Creates failure records from events and metrics. Event failures use the emitting episode frame as the replay target; metric-derived failures use the first matching frame when one can be identified.

store.py

Reads and updates failure lists stored in failures.json. Status updates append history entries before writing the updated list back to disk.

tags.py

Central list of known failure tags and allowed lifecycle statuses.

roboci/reporting

html.py

Renders the static Jinja2 report.

markdown.py

Renders the concise markdown report used for PR and CI summaries.

json_report.py

Loads report context from run artifacts.

templates/report.html.j2

HTML report template.

roboci/storage

artifacts.py

JSON and JSONL read/write helpers.

paths.py

Resolves local artifact paths from the current directory or a parent workspace.

Environment overrides:

  • ROBOCI_RUNS_DIR
  • ROBOCI_EXAMPLES_DIR

run_store.py

Creates run directories, updates runs/latest, lists runs, labels run type, attaches regression counts, exposes compact provenance summaries for run lists, and clears existing run artifacts for dashboard reset.

roboci/api

server.py

Creates the FastAPI app, configures CORS, and includes route modules.

routes/runs.py

Run listing, reset, launch, artifact indexing, allowlisted artifact text preview, regression, and run detail endpoints.

routes/replay.py

Replay artifact endpoint.

routes/scenarios.py

Scenario catalog endpoint.

routes/failures.py

Failure listing and update endpoints, including status/owner filters and atomic patch updates for tags, notes, status, owner, fixed-in run, and regressed-in run fields.

routes/metrics.py

Trend endpoint.

Frontend

The frontend is a Next.js App Router project in frontend/.

Important files:

  • app/layout.tsx: shell and navigation.
  • app/dashboard/page.tsx: overview KPIs, latest marketplace metadata, and latest-run domain/pack filters.
  • app/run-lab/page.tsx: self-serve closed-loop run launcher.
  • app/runs/page.tsx: run list.
  • app/runs/[run_id]/page.tsx: run detail, including scenario tags, ?tag=<tag>, ?domain=<domain>, and ?pack=<pack-id> filtering, plus an open-by-default local artifact tree.
  • app/runs/[run_id]/scenarios/[scenario_id]/page.tsx: replay detail, including scenario tag chips.
  • app/regressions/page.tsx: Regression Explorer, compare workbench, and regression artifact table.
  • app/failures/page.tsx: failure table with severity, status, owner, tag, and search filters.
  • app/scenarios/page.tsx: scenario marketplace and local catalog with marketplace/local tag, domain, and search filtering plus pack install/remove controls.
  • components/ResetDashboardButton.tsx: confirmable dashboard reset action.
  • components/ArtifactBrowser.tsx: local artifact tree, allowlisted text previews, and local open actions for indexed artifacts that are not exposed through the preview API.
  • components/RegressionExplorer.tsx: closed-loop/open-loop regression review sections and diff cards.
  • lib/regressionExplorer.ts: classification logic for regression review buckets and card links.
  • lib/runtimeConfig.ts: runtime API base helper for server-rendered pages and client-side workbench props.
  • components/*: reusable dashboard components.
  • lib/api.ts: backend fetch wrapper.
  • lib/types.ts: TypeScript data types.

Examples

The examples/ directory is executable documentation:

  • examples/scenarios/*.yaml: scenario definitions.
  • examples/suites/*.yaml: suite definitions.
  • examples/agents/*.py: closed-loop agents and open-loop models.
  • examples/datasets/sample_openloop: sample recorded dataset.

The installed package also carries a minimal source-of-truth copy under roboci/assets/ for commands that should work without a repository checkout:

  • roboci/assets/examples/: demo suites, scenarios, Python agents, and the sample open-loop dataset used by roboci demo.
  • roboci/assets/benchmarks/: packaged benchmark catalog used by roboci benchmarks list and roboci benchmarks run <id>.

When changing demo or benchmark source files, keep the repository examples/benchmarks and roboci/assets/ copies aligned before release.