Extending RoboCI¶

RoboCI is designed around small interfaces. Most extensions should fit into one of these extension points.

Add a Simulator¶

Implement SimulatorAdapter or a tuple-returning closed-loop adapter:

from roboci.simulators.base import Action, Observation, SimState, SimulatorAdapter


class MySimulator(SimulatorAdapter):
    def reset(self, scenario: dict) -> dict:
        ...

    def step(self, action: dict) -> dict:
        ...

    def is_done(self) -> bool:
        ...

    def close(self) -> None:
        ...

For external closed-loop stacks, adapters can return:

def step(self, action: Action | dict) -> tuple[Observation, SimState, bool, dict]:
    ...

Then register the adapter in:

roboci/simulators/registry.py

RoboCI includes built-in adapters for carla, isaac, and gazebo, plus a ROS2 JSON bridge simulator as --sim ros2. See docs/closed_loop_simulators.md for the external simulator adapter contract and docs/ros2.md for the JSON topic bridge.

Python simulators can also be loaded directly with --sim path/to/simulator.py. A Python simulator file can expose create_simulator(), a Simulator class, or module-level reset() and step() functions:

class Simulator:
    def reset(self, scenario: dict) -> dict:
        ...

    def step(self, action: dict) -> dict:
        ...

    def is_done(self) -> bool:
        ...

    def close(self) -> None:
        ...

Run it with:

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim examples/simulators/simple_simulator.py

Simulator YAML manifests can also point at a registered simulator by name. For example, examples/simulators/mock_simulator.yaml advertises the built-in mock simulator in launch options while resolving to the same mock adapter at run time.

RoboCI also supports native process simulator bindings for C++ and Rust. Native simulators stay running for the episode and exchange newline-delimited JSON over stdin/stdout.

Prerequisites:

• C++ sources require g++ or clang++ on PATH.
• Rust sources require rustc on PATH.
• ROS2 simulator work requires a local ROS2 installation; see docs/ros2.md.

RoboCI sends:

{"command": "reset", "scenario": {"id": "example"}}
{"command": "step", "action": {"steer": 0.0, "throttle": 0.4, "brake": 0.0}}
{"command": "close"}

The simulator should return one JSON object per command. reset and step responses should include an observation object and may include done:

{
  "observation": {
    "t": 0.1,
    "ego": {"x": 0.1, "y": 0, "yaw": 0, "speed": 1},
    "actors": [],
    "action": {"steer": 0, "throttle": 0.4, "brake": 0},
    "events": []
  },
  "done": false
}

C++ sources (.cpp, .cc, .cxx) are compiled with g++ or clang++, and Rust sources (.rs) are compiled with rustc. Compiled binaries are cached under .roboci/bindings/. You can also pass an already-built executable directly:

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim examples/simulators/simple_simulator.cpp
roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim examples/simulators/simple_simulator.rs

Expected observation format:

{
  "t": 0.1,
  "ego": {"x": 0, "y": 0, "yaw": 0, "speed": 5},
  "actors": [],
  "action": {"steer": 0, "throttle": 0.4, "brake": 0},
  "events": []
}

Add an Agent¶

Create a Python file exposing one of:

def create_agent():
    return MyAgent()

or:

class Agent:
    def act(self, observation: dict) -> dict:
        return {"steer": 0.0, "throttle": 0.4, "brake": 0.0}

or:

def act(observation: dict) -> dict:
    return {"steer": 0.0, "throttle": 0.4, "brake": 0.0}

Run:

roboci run --suite examples/suites/smoke.yaml --agent path/to/agent.py --sim mock

RoboCI also supports native process bindings for C++ and Rust. A native binding reads one JSON object from stdin and writes one JSON object to stdout. For closed-loop runs, stdin is the observation and stdout is the action. For open-loop runs, stdin is the sample and stdout is the prediction.

C++ sources (.cpp, .cc, .cxx) are compiled with g++ or clang++ and Rust sources (.rs) are compiled with rustc. Compiled binaries are cached under .roboci/bindings/. You can also pass an already-built executable directly.

Minimal C++ agent:

#include <iostream>
#include <string>

int main() {
    std::string input((std::istreambuf_iterator<char>(std::cin)), std::istreambuf_iterator<char>());
    (void)input;
    std::cout << R"({"steer":0.0,"throttle":0.4,"brake":0.0})" << std::endl;
}

Minimal Rust agent:

use std::io::{self, Read};

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).unwrap();
    println!(r#"{{"steer":0.0,"throttle":0.4,"brake":0.0}}"#);
}

Run native agents the same way:

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.cpp --sim mock
roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.rs --sim mock

Open-loop native models use the same stdin/stdout contract:

roboci openloop run \
  --dataset examples/datasets/sample_openloop \
  --model path/to/model.rs \
  --task trajectory_prediction

Add a Closed-loop Metric¶

1. Implement the metric in roboci/metrics/.
2. Add it to compute_closed_loop_metrics() in roboci/metrics/registry.py.
3. Decide whether it should be aggregated in _aggregate_metrics() in roboci/core/runner.py.
4. Add tests.
5. Add it to docs/data-formats.md if it becomes part of stable output.

Add a Custom Domain Metric¶

Domain metrics are closed-loop metrics requested by scenario metadata:

environment:
  marketplace_pack_domain: ground_robotics
  domain_metrics:
    - collision_rate
    - route_completion

RoboCI validates domain names through DOMAIN_METRIC_REGISTRY in roboci/metrics/registry.py. To add a domain metric:

1. Add the metric name to the domain tuple in DOMAIN_METRIC_REGISTRY.
2. Add a branch in _compute_domain_metrics() that computes the value from episode, scenario, and the already-computed core metrics.
3. Return a JSON-native number. Domain metrics are rounded to six decimal places before being written.
4. Add scenario or metric tests that request the metric through environment.domain_metrics.
5. Document the metric name, units, and direction in docs/data-formats.md if users should rely on it.

Use stable, snake-case names. Unknown names in environment.domain_metrics are ignored today only because unsupported names have no compute branch, so tests should cover any new public metric.

Add an Open-loop Task¶

Open-loop tasks use the same model interface:

class OpenLoopModel:
    def predict(self, sample: dict) -> dict:
        ...

To add metrics:

1. Implement metric functions in roboci/openloop/metrics.py.
2. Update OpenLoopRunner.run() in roboci/openloop/runner.py.
3. Add gate support in roboci/regression/gates.py if needed.
4. Add tests.

Add a Failure Type¶

1. Add the tag to roboci/failures/tags.py.
2. Add event-to-failure logic in roboci/failures/detector.py.
3. Ensure the simulator or metrics emit enough information.
4. Add a test in tests/test_failure_detector.py.

Failure lifecycle statuses are also defined in roboci/failures/tags.py. If you add or rename a status, update API validation, frontend editor options, docs, and status-history tests together.

Add an API Endpoint¶

1. Create or update a route module in roboci/api/routes/.
2. Include the router in roboci/api/server.py if it is a new module.
3. Read artifacts through RunStore or resolve_workspace_path().
4. Add tests in tests/test_api_routes.py.
5. Document the endpoint in docs/api.md.

Add a Dashboard Page¶

1. Add a route under frontend/app/.
2. Add backend fetch helper in frontend/lib/api.ts if needed.
3. Add shared types in frontend/lib/types.ts.
4. Prefer reusable components under frontend/components/.
5. Run npm run build.

Add a Replay Visualization¶

Replay artifacts are:

• timeline JSON
• GeoJSON
• normalized timeline events JSON

Older runs may also have a legacy events JSON alias; new exporters should write only normalized timeline events JSON.

Use:

GET /runs/{run_id}/replay/{scenario_id}

Then extend:

frontend/components/ReplayViewer.tsx
frontend/components/EventTimeline.tsx
frontend/components/ScenarioReplayWorkspace.tsx
frontend/components/ScenarioReplayPanel.tsx
frontend/components/TrajectoryMap.tsx

Timeline events should use the normalized timestamp, severity, type, title, description, and optional actor/frame fields exported in replay/<scenario_id>.timeline_events.json. Event clicks on the scenario detail page seek the replay to the closest frame for that timestamp.

Add a New Artifact¶

Use helpers in:

roboci/storage/artifacts.py

Prefer JSON for structured objects and JSONL for sequences.

When adding an artifact:

1. Write it under runs/<run_id>/.
2. Add API indexing if the dashboard should show it in the local artifact tree.
3. Add text-preview API access only if the file is safe to expose through the allowlist. The artifact preview endpoint must not become arbitrary filesystem access.
4. Add docs in docs/data-formats.md.
5. Add a test that validates it is written and, if previewable, that unsafe paths are rejected.