CLI Guide

RoboCI uses Typer for the command-line interface.

Install

Most explicit path examples in this guide assume you are running from a cloned RoboCI repository root. Commands that do not take explicit paths, such as roboci demo, roboci demo --openloop, and roboci benchmarks ..., use packaged demo assets and work from any directory after installation.

git clone <repo-url>
cd roboci
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -c constraints.txt -e ".[dev]"

Initialize

roboci init
roboci init --template carla-av
roboci init --template ros2-mobile-robot
roboci init --template drone-navigation

Without --template, creates local workspace directories such as runs/ and example subdirectories if they do not exist. With --template, copies a ready starter workspace with an agent skeleton, simulator config, scenario, and suite. Existing files are kept rather than overwritten.

Doctor

roboci doctor

Checks local prerequisites and setup from the current working directory:

  • Python version and RoboCI installation
  • example files and the runs/ directory
  • Node.js and dashboard dependencies
  • API port availability

roboci doctor exits non-zero only for required CLI setup failures: Python, RoboCI installation metadata, and examples. Missing optional dashboard setup, a missing runs/ directory before the first run, a busy port, or a sandbox-blocked port probe prints a status line but remains advisory.

One-Command Demo

roboci demo
roboci demo --openloop
roboci demo --dashboard

roboci demo runs the local public-alpha happy path:

  1. Passing smoke suite.
  2. Intentional collision failure, handled internally.
  3. Named baseline promotion to demo-smoke.
  4. Current smoke run.
  5. Regression comparison against the named baseline.
  6. HTML and markdown report generation.

--openloop also runs the packaged sample open-loop dataset with the packaged reference model. --dashboard starts roboci dev after the demo when dashboard dependencies are installed; otherwise it prints Docker/local setup guidance. The core demo does not depend on Node and does not require a repo checkout.

Closed-loop Run

roboci run \
  --suite examples/suites/smoke.yaml \
  --agent examples/agents/rule_based_agent.py \
  --sim mock

Use --sim ros2 to run the same loop against a ROS2 bridge. The ROS2 bridge publishes reset and action JSON messages and waits for observation JSON messages; see docs/ros2.md.

What happens:

  1. Loads the suite YAML.
  2. Resolves scenario paths.
  3. Loads the Python agent.
  4. Creates a mock simulator for each scenario.
  5. Records episodes, metrics, replay artifacts, failures, and an HTML report.
  6. Writes artifacts to runs/<run_id>/.

Exit code:

  • 0 if gates pass and no blocking failures are present.
  • Non-zero if gates fail or failures violate run status.

Scenario Marketplace

roboci scenarios list
roboci scenarios install urban-v1
roboci scenarios install highway-v2
roboci scenarios remove urban-v1

roboci scenarios list prints the installable scenario packs. roboci scenarios install <pack-id> writes generated scenario YAML files to:

examples/scenario_marketplace/<pack-id>/

It also writes a runnable suite to:

examples/suites/<pack-id>.yaml

roboci scenarios remove <pack-id> removes generated scenarios and the matching suite from the selected workspace. For workspace-local packs declared by examples/scenario_marketplace/<pack-id>/pack.yaml, the manifest is preserved so the custom pack stays discoverable. Historical runs are preserved; matching scenario entries are marked with removed: true and recorded in removed_scenario_refs.json.

The current marketplace includes 11 categories, each with 20-30 scenarios across ground robotics, manipulators, and drones.

Compare Runs

roboci baseline create smoke --from runs/latest --description "Known-good smoke run"

roboci compare \
  --current runs/latest \
  --baseline-name smoke

Baseline commands:

roboci baseline create smoke --from runs/latest
roboci baseline promote runs/latest smoke
roboci baseline create smoke --from runs/latest --overwrite
roboci baseline list
roboci baseline show smoke

Named baselines are copied to baselines/<name>/. The metadata file baselines/<name>/baseline.json records the source run id, creation time, and optional description.

roboci baseline promote <run-dir> <name> is the positional equivalent of roboci baseline create <name> --from <run-dir>. roboci baseline list shows only directories with baseline.json; unrelated folders under baselines/ are ignored. Use --overwrite to replace an existing named baseline from a different source run. RoboCI rejects overwriting a baseline from its own baselines/<name>/ directory so the source is not deleted.

Path-based comparisons remain supported:

roboci compare \
  --current runs/latest \
  --baseline baselines/smoke

--baseline must point to an existing run directory, such as a previous runs/<run_id> from a known-good run or a directory under baselines/.

Writes:

runs/<current>/regression.json

The dashboard reads this file through the API. If the regression page shows no deltas, make sure roboci compare has been run for that current run.

Benchmarks

roboci benchmarks list
roboci benchmarks run smoke
roboci benchmarks run ground-v0
roboci benchmarks run drone-v0
roboci benchmarks run manipulator-v0

Benchmark definitions are packaged under roboci.assets/benchmarks and mirrored in the repository under benchmarks/<id>/benchmark.yaml. Each benchmark validates metadata, loads its reference agent, runs through the normal closed-loop runner, writes normal run artifacts, and adds benchmark.json to the run directory.

The bundled v0.1 benchmarks default to the deterministic mock simulator. Use --sim <name> only when an alternate simulator runtime is installed and configured locally.

Generate Report

Closed-loop and open-loop runs write both static reports automatically:

runs/<run_id>/report.html
runs/<run_id>/report.md

Regenerate reports for an existing run:

roboci report --run runs/latest
roboci report --run runs/latest --format markdown

Depending on --format, writes:

runs/<run_id>/report.html
runs/<run_id>/report.md

runs/latest is accepted as a run selector, but the report is written into the resolved run directory. Use --format markdown for a concise PR-friendly summary with status, pass/fail counts, safety score, regression changes, critical failures, open-loop metrics when available, and links to local report and replay artifacts.

Open-loop Run

roboci openloop run \
  --dataset examples/datasets/sample_openloop \
  --model examples/agents/rule_based_agent.py \
  --task trajectory_prediction

Writes:

runs/<run_id>/openloop/predictions.jsonl
runs/<run_id>/openloop/per_sample_metrics.jsonl
runs/<run_id>/openloop/aggregate_metrics.json
runs/<run_id>/metrics.json
runs/<run_id>/summary.json
runs/<run_id>/provenance.json
runs/<run_id>/regression.json
runs/<run_id>/report.html

Open-loop provenance includes the model path/hash plus dataset metadata, sample, annotation, and composite dataset hashes. This makes open-loop regression comparisons sensitive to dataset changes, not only model or code changes.

Open-loop Compare

roboci openloop compare \
  --current runs/latest \
  --baseline baselines/openloop-smoke

This uses the open-loop comparator, which reads open-loop aggregate metrics such as ADE, FDE, miss rate, control error, and latency without closed-loop scenario diffs.

Bundle A Run

roboci bundle runs/latest --output roboci-run.zip

The bundle command zips one resolved run directory. It includes run artifacts such as summary.json, metrics.json, reports, replay files, and open-loop outputs. It excludes workspace noise such as virtualenvs, node_modules, cache directories, and build output if those directories are accidentally present under the run directory.

Serve API

roboci serve

If port 8000 is busy, RoboCI picks the next available port and prints the selected URL. Use roboci serve --strict-port to fail when the requested port is unavailable.

Default API:

http://127.0.0.1:8000

roboci serve --host 0.0.0.0 exposes the artifact API beyond the local loopback interface. Only use non-loopback hosts on trusted networks or behind controls such as a firewall, VPN, SSH tunnel, or authenticated reverse proxy. Set ROBOCI_API_TOKEN to require Authorization: Bearer <token> or X-RoboCI-Token: <token> on routed API calls. Dashboard fetches use NEXT_PUBLIC_ROBOCI_API_TOKEN; roboci dev copies ROBOCI_API_TOKEN into that frontend variable when both servers start together.

Useful endpoints:

  • /
  • /health
  • /docs
  • /runs
  • /baselines
  • /runs/{run_id}/artifacts
  • /runs/{run_id}/artifacts/read
  • /runs/{run_id}/artifacts/open
  • /metrics/trends
  • /failures

Start API And Dashboard

Default Docker path:

roboci dev --docker

Equivalent Compose command:

docker compose --profile dashboard up --build api dashboard

Local Node development path:

roboci dev

Starts the FastAPI backend and Next.js dashboard together. Defaults:

API: http://127.0.0.1:8000
Dashboard: http://127.0.0.1:3000/dashboard

If either port is busy, RoboCI chooses the next available port and prints the URLs.

Choose ports explicitly with:

roboci dev --api-port 8010 --frontend-port 3010

For scripted local runs, the frontend port can also come from NEXT_PUBLIC_ROBOCI_PORT or ROBOCI_FRONTEND_PORT:

NEXT_PUBLIC_ROBOCI_PORT=3010 roboci dev

The local Node path expects dashboard dependencies to be installed first:

cd frontend
npm install
cd ..

Sync Artifacts

roboci storage sync runs/latest /mnt/team-share/roboci/latest
roboci storage sync runs/latest s3://my-bucket/roboci/latest
roboci storage sync runs/latest gs://my-bucket/roboci/latest

Local and mounted network-share destinations are copied directly. s3:// destinations use aws s3 sync; gs:// destinations use gsutil -m rsync -r, so those CLIs must be installed and authenticated in the environment running the command.

Exit Codes

RoboCI commands are CI-friendly.

  • 0: pass.
  • Non-zero: gates, regressions, or failures violated expected thresholds.

Common Workflows

Local smoke test

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim mock
roboci report --run runs/latest
roboci report --run runs/latest --format markdown

Known collision failure

roboci run --suite examples/suites/failing_collision.yaml --agent examples/agents/collision_test_agent.py --sim mock
roboci report --run runs/latest
roboci report --run runs/latest --format markdown

This flow is expected to fail: the agent does not brake or steer around the obstacle. Use it to inspect failure records, tags, replay artifacts, and the generated report.

CI regression check

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim mock --fail-on-regression
roboci baseline create smoke --from runs/latest --overwrite
roboci compare --current runs/latest --baseline-name smoke

Dashboard workflow

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim mock
roboci dev --docker