RoboCI¶

Local-first CI for physical AI: run closed-loop simulations, open-loop dataset checks, replay failures, and compare regressions from one developer workflow.

RoboCI is for robotics, autonomous vehicle, drone, and embodied AI teams that need repeatable evaluation before code reaches hardware. It writes every run to inspectable local artifacts, returns CI-friendly exit codes, and ships with a dashboard for replay, metrics, and failure triage.

Documentation

v0.0.1 Public Alpha¶

RoboCI v0.0.1 is a local-first public alpha. It is useful for demos, early evaluation, replay, artifact review, and regression checks, but it is not production safety validation and should not be used as certification evidence for robots, vehicles, drones, manipulators, or hardware-in-loop systems.

This release includes the local CLI, turnkey deterministic mock simulator runs, CARLA/Isaac/Gazebo/ROS2 adapter surfaces that require local runtime setup, open-loop sample dataset evaluation, file-backed run artifacts, HTML and markdown reports, regression comparison, a FastAPI artifact API, and a Next.js dashboard for run review, replay, regressions, and failure tagging.

Early contributors can help most by trying the demo, reporting rough edges, adding simulator and agent examples, improving dashboard workflows, and tightening artifact contracts before the first stable release.

git clone https://github.com/GagandeepReehal/roboci.git
cd roboci
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

roboci demo
roboci report --run runs/latest

Demo¶

The demo shows a passing smoke run, an intentional collision failure, named baseline promotion, regression comparison, and generated HTML/markdown reports using real local RoboCI output.

roboci demo
roboci demo --openloop
roboci demo --dashboard

Why RoboCI Exists¶

Physical AI regressions are costly because failures are often slow to reproduce, hard to compare, and scattered across logs, simulator output, metrics, and screenshots. RoboCI keeps that loop tight:

• Run closed-loop scenario suites and open-loop dataset evaluations from the CLI.
• Store replay, metrics, reports, failures, and regression output under runs/<run_id>/.
• Search and filter scenarios by metadata tags such as pedestrian, urban, crossing, rain, night, and highway.
• Compare runs against baselines with gates that work in CI.
• Inspect indexed run artifacts locally in a FastAPI + Next.js dashboard, with safe text previews for summary files and local open actions for larger artifacts.

Screenshots¶

Dashboard¶

Replay¶

Regression¶

Quickstart¶

Requirements:

• Python 3.11+
• Docker Compose for the default dashboard path
• Node.js 20+ and npm only for local dashboard development without Docker
• macOS, Linux, or any environment that can run the Python CLI

From a cloned repo root:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[dev]"

roboci demo
roboci report --run runs/latest
roboci doctor

The run writes artifacts to runs/<run_id>/. runs/latest resolves to the newest run. roboci doctor checks the local environment after the demo creates the workspace artifacts; dashboard and port checks are advisory for CLI-only use.

Published-package install uses the bundled demo and benchmark assets, so the first demo does not require a cloned repository checkout:

pip install roboci
roboci demo
roboci report --run runs/latest

If this version is not yet available from PyPI, use the editable repository install shown above.

Installation¶

Development install:

git clone <repo-url>
cd roboci
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Default dashboard path, no local Node required:

docker compose --profile dashboard up --build api dashboard

Optional health check:

roboci doctor

roboci doctor exits non-zero for required CLI setup failures. Missing dashboard dependencies, an absent runs/ directory before the first run, and busy or sandbox-blocked ports are reported as advisories.

One-command demo:

roboci demo

Optional sample dataset workflow:

roboci demo --openloop

Closed-Loop Demo¶

Run a passing closed-loop suite with the mock simulator:

roboci run \
  --suite examples/suites/smoke.yaml \
  --agent examples/agents/rule_based_agent.py \
  --sim mock

Expected output: a passing run, a run id, and artifacts under runs/<run_id>/.

Run an intentional failure:

roboci run \
  --suite examples/suites/failing_collision.yaml \
  --agent examples/agents/collision_test_agent.py \
  --sim mock

Expected output: a non-zero exit after RoboCI writes collision metrics, replay data, failure records, and a report-ready run directory.

The non-zero exit is expected for this demo because the suite intentionally triggers a collision. Use it when you want failure, replay, and tagging artifacts.

External closed-loop simulator adapters for CARLA, Isaac Sim, and Gazebo are documented in the simulator adapter guide. Their default tests use mocks, so the repository test suite does not require those simulator SDKs.

Open-Loop Demo¶

Evaluate a model against the sample open-loop dataset:

roboci openloop run \
  --dataset examples/datasets/sample_openloop \
  --model examples/agents/rule_based_agent.py \
  --task trajectory_prediction

Expected output: aggregate and per-sample open-loop metrics under runs/<run_id>/.

Benchmarks¶

List bundled public-alpha benchmarks:

roboci benchmarks list

The benchmark catalog is packaged with RoboCI. roboci benchmarks list and roboci benchmarks run <id> work from any working directory after installation.

Run the dependency-light smoke benchmark:

roboci benchmarks run smoke

Copy .github/workflows/roboci.yml into a project to run the smoke benchmark on pull requests, post report.md as a PR comment, and upload roboci-smoke-run.zip as an artifact.

The current catalog includes smoke, ground-v0, drone-v0, and manipulator-v0. They default to the deterministic mock simulator and write normal RoboCI artifacts under runs/<run_id>/. External simulators are adapter surfaces: select them explicitly only after local CARLA, Isaac Sim, Gazebo/ROS2, or another runtime is installed and configured.

Dashboard Demo¶

Start the API and prebuilt dashboard image with Docker Compose:

docker compose --profile dashboard up --build api dashboard

Default URLs:

• Dashboard: http://127.0.0.1:3000/dashboard
• API: http://127.0.0.1:8000
• API docs: http://127.0.0.1:8000/docs

The dashboard image is built from frontend/package-lock.json; it does not run npm install when the container starts.

For a local Node development loop, install dashboard dependencies once and run:

cd frontend
npm install
cd ..
roboci dev

If a port is busy, roboci dev picks the next available port and prints the selected URLs. Choose ports explicitly with roboci dev --api-port 8010 --frontend-port 3010.

Users without local Node can also run:

roboci dev --docker

roboci doctor checks whether the default API port is available. If the backend is already running, or if a sandbox blocks local port probing, the port check can report blocked; stop the running backend or use the URL printed by roboci dev.

Docker¶

The default Docker path is CLI/demo focused and does not include simulator SDKs:

docker compose build roboci
docker compose run --rm roboci roboci doctor
docker compose run --rm roboci roboci demo
docker compose run --rm roboci roboci benchmarks run smoke

Optional dashboard profile:

docker compose --profile dashboard up --build api dashboard

The dashboard profile builds a Next.js standalone image and uses NEXT_PUBLIC_ROBOCI_API=http://127.0.0.1:8000 for browser requests and ROBOCI_SERVER_API=http://api:8000 for server-rendered dashboard requests inside the Compose network.

CARLA smoke profile:

docker compose --profile carla up --build carla carla-roboci

This starts carlasim/carla:0.9.15, builds a RoboCI runner image, waits for the CARLA RPC port, and runs examples/suites/carla.yaml end to end. The default runner image does not bundle the CARLA Python API because published CARLA client packages are platform- and Python-version-specific. On prepared Linux/GPU CARLA runners, set ROBOCI_CARLA_PYTHON_PACKAGE to a matching CARLA Python API wheel or package before building; otherwise carla-roboci exits early with a clear setup error instead of failing during image build.

Replay And Failure Tagging¶

Generate a run first:

roboci run --suite examples/suites/failing_collision.yaml --agent examples/agents/collision_test_agent.py --sim mock

Then inspect it in the dashboard:

• Run list: http://127.0.0.1:3000/runs
• Run detail: http://127.0.0.1:3000/runs/<run_id>
• Scenario replay: http://127.0.0.1:3000/runs/<run_id>/scenarios/<scenario_id>
• Failure replay deep link: http://127.0.0.1:3000/runs/<run_id>/scenarios/<scenario_id>?t=<timestamp>&frame=<frame_index>

RoboCI stores detected failures as file-backed artifacts, so tags and triage state stay close to the run data they describe. Failure records include replay location fields when available, and failure links in the dashboard open the replay at the incident frame. The dashboard failure editor supports a lifecycle workflow with open, acknowledged, assigned, in_progress, fixed, verified, regressed, and wont_fix statuses. Optional owner, fixed_in_run_id, and regressed_in_run_id fields are stored in failures.json, and each status change appends a local status_history entry. Closed-loop regression comparisons also backfill lifecycle links: baseline failures that pass in the current run get fixed_in_run_id, while new failures on previously passing scenarios get regressed_in_run_id. Scenario replay pages show a synchronized replay scene and event timeline sourced from replay/<scenario_id>.timeline_events.json; clicking a timeline event seeks the replay scene to that event timestamp.

Scenario Metadata¶

Scenario YAML files can include optional lowercase, slug-style tags:

id: pedestrian_crossing_001
name: Pedestrian Crossing
tags:
  - pedestrian
  - crossing
  - urban

Tags are written into run scenario results under runs/<run_id>/metrics.json, exposed through the API, and shown in the dashboard. Use /scenarios?tag=<tag> to filter local examples and marketplace packs, /scenarios?domain=<domain> to filter marketplace packs by robotics domain, and /scenarios?q=<text> to search pack names, tasks, metrics, requirements, tags, and scenario names. Run detail pages support /runs/<run_id>?tag=<tag>, /runs/<run_id>?domain=<domain>, and /runs/<run_id>?pack=<pack-id> filters.

Marketplace packs are domain-aware. New packs can declare domain: ground_robotics, domain: manipulators, or domain: drones, plus supported tasks, domain metrics, adapter requirements, simulator requirements, tags, version, license, and domain-specific metadata. Workspace-local packs can be added with examples/scenario_marketplace/<pack-id>/pack.yaml, and roboci scenarios install <pack-id> --versioned installs a version-suffixed copy that can coexist with the default install. Existing packs without a domain field default to ground_robotics so installed packs and older run artifacts continue to load. Pack name, version, domain, task type, and metric group are propagated into run summaries, provenance, dashboard views, and generated reports.

Scenario YAML files can also include optional behavioral assertions:

assertions:
  - name: stop_before_pedestrian
    type: min_distance
    actor_type: pedestrian
    threshold: 1.5
    severity: critical

  - name: max_speed_near_pedestrian
    type: max_speed_when_actor_near
    actor_type: pedestrian
    distance: 10
    max_speed: 2.0
    severity: high

Supported assertion types are no_collision, min_distance, max_speed, max_speed_when_actor_near, and goal_reached. critical and high failures fail the scenario; lower severities are recorded as advisory results. Assertion results are written to metrics.json, summarized in summary.json, and shown in both static reports and the dashboard alongside generic failure counts.

Regression And Reports¶

Fresh runs write both static reports automatically:

runs/<run_id>/report.html
runs/<run_id>/report.md

Regenerate static reports for an existing run:

roboci report --run runs/latest
roboci report --run runs/latest --format markdown

The default report command regenerates report.html. The markdown format regenerates report.md, which is concise enough for PR comments and CI artifact review.

Static reports use local relative artifact links by default. To embed a dashboard deep link without assuming a port, set ROBOCI_DASHBOARD_URL before generating the report, for example to the dashboard URL printed by roboci dev.

Compare a run against a baseline:

roboci baseline create smoke --from runs/latest --description "Known-good smoke run"
roboci baseline promote runs/latest smoke
roboci baseline list
roboci baseline show smoke

roboci compare \
  --current runs/latest \
  --baseline-name smoke \
  --max-regression-percent 5

Named baselines are stored under baselines/<name>/ with baseline.json metadata containing the source run id, resolved source directory, creation time, and optional description. roboci baseline list only shows directories with baseline.json, so unrelated folders under baselines/ are ignored. roboci compare --baseline <run-dir> still works for path-based comparisons.

roboci compare returns a non-zero exit when regressions exceed the configured threshold. That behavior is intended for CI gates.

CLI Reference¶

Architecture Overview¶

RoboCI is built around a local artifact contract. The CLI runs evaluation work, the runners write structured files under runs/<run_id>/, and reports, regression checks, the API, and the dashboard all read the same artifacts. Every run writes provenance.json and embeds the same object in summary.json so regression comparisons can account for Git state, source file hashes, simulator/runtime versions, and open-loop dataset hashes. The dashboard indexes local artifacts but only previews allowlisted root text files (summary.json, metrics.json, regression.json, failures.json, config.yaml, and provenance.json) through the API; episode logs, replay artifacts, reports, and open-loop files remain local-only for preview and can be opened with the host default app.

Project Structure¶

Roadmap¶

See ROADMAP.md for stable v0.1 surfaces, unstable surfaces, current limitations, and the contribution roadmap.

Known Limitations¶

• RoboCI is local-first; ROBOCI_API_TOKEN is a shared trusted-network token, not a hosted multi-user service.
• RoboCI is not production safety validation. Treat outputs as deterministic regression evidence for development, not certification.
• The mock simulator is turnkey and deterministic. CARLA, Isaac Sim, Gazebo/ROS2, Python simulator extensions, and native simulators require local runtime setup.
• The dashboard reads file-backed artifacts and expects the FastAPI backend to be running.
• Dashboard access defaults to Docker Compose; local dashboard development still requires Node.js/npm.
• Native C++ and Rust examples require local compiler/runtime availability.
• ROS2 support is a reference bridge, not a turnkey production robotics deployment.
• CARLA integration depends on the CARLA Docker image or a local CARLA setup with a matching Python API.
• Demo media is captured from local example runs and should be refreshed when dashboard workflows change.

Contributing¶

Contributions are welcome. For most changes:

python -m pytest
cd frontend
npm run lint
npm run build

Guidelines:

• Keep generated runs/, named baseline data, and dashboard build output out of commits.
• Add focused tests when changing CLI, API, runner, regression, replay, or dashboard behavior.
• Update docs when command flags, artifact formats, or workflows change.
• Prefer small pull requests with a clear reproduction or demo command.

Start with the documentation map for deeper project documentation.

Build the documentation site locally with MkDocs:

pip install -e ".[docs]"
mkdocs serve
mkdocs build --strict

Citation¶

If RoboCI helps your research or development work, please cite the repo:

@software{roboci_2026,
  title = {RoboCI: Local-first CI for Physical AI Evaluation, Replay, and Regression Testing},
  author = {Reehal, Gagandeep},
  year = {2026},
  version = {0.0.1},
  note = {Public alpha release},
  url = {https://github.com/GagandeepReehal/roboci}
}

License¶

RoboCI is licensed under Apache-2.0. See LICENSE.