Data Formats

RoboCI is artifact-driven. Understanding the file formats is the quickest way to understand the system.

Scenario YAML

Example:

id: pedestrian_crossing_001
name: Pedestrian Crossing
description: Ego vehicle must slow down for a pedestrian crossing from the right.
tags:
  - pedestrian
  - crossing
  - urban
simulator: mock
duration_seconds: 30
timestep_seconds: 0.1

ego:
  start:
    x: 0
    y: 0
    yaw: 0
    speed: 8
  goal:
    x: 80
    y: 0
    tolerance: 3

actors:
  - id: pedestrian_1
    type: pedestrian
    start:
      x: 35
      y: -5
      yaw: 90
      speed: 1.2
    behavior:
      type: constant_velocity

environment:
  weather: clear
  road_type: urban
  speed_limit: 40

success_criteria:
  max_collisions: 0
  must_reach_goal: true
  max_time_seconds: 30

failure_criteria:
  collision: true
  offroad: true
  timeout: true

assertions:
  - name: stop_before_pedestrian
    type: min_distance
    actor_type: pedestrian
    threshold: 1.5
    severity: critical

  - name: max_speed_near_pedestrian
    type: max_speed_when_actor_near
    actor_type: pedestrian
    distance: 10
    max_speed: 2.0
    severity: high

Schema owner:

roboci/scenarios/schema.py

tags is optional. When present, it must be a list of lowercase, slug-style strings such as pedestrian, urban, crossing, rain, night, or highway. Scenario tags are copied into closed-loop scenario results in metrics.json so API and dashboard filters can work from run artifacts.

Installed marketplace scenarios generated before the tags field existed may not have tags in their YAML files. RoboCI backfills canonical marketplace tags when loading those scenarios based on pack id and scenario id, so older installed packs still produce tagged run artifacts. Marketplace scenarios also carry environment metadata such as marketplace_pack, marketplace_pack_name, marketplace_pack_version, marketplace_pack_domain, marketplace_task_type, metric_group, domain_metrics, and domain_metadata; these fields are propagated into metrics.json, summary.json, provenance.json, dashboard metadata, and generated reports when available.

assertions is optional. Supported assertion types are no_collision, min_distance, max_speed, max_speed_when_actor_near, and goal_reached. critical and high assertion failures fail the scenario. medium and low assertion failures are recorded but do not fail the scenario by themselves.

Actor-scoped assertions use actor_type to filter frame actors. min_distance fails when no matching actor is observed. max_speed_when_actor_near returns status: not_applicable when no matching actor is observed, so scenarios where the actor never enters the observation stream do not look like safety violations. Not-applicable assertions are excluded from passed, failed, and blocking-failed counts; if a matching actor exists but is never within distance, the assertion passes because the near condition never occurred.

Assertion results are written to each scenario entry in metrics.json, with aggregate counts in both metrics.json and summary.json. Static reports and the dashboard show assertion results alongside existing metric and failure views.

simulator can also be a mapping for external closed-loop adapters:

simulator:
  name: carla
  host: localhost
  port: 2000
  sync: true
  fixed_delta_seconds: 0.05

sensors:
  cameras:
    - name: front
      type: rgb
      width: 1280
      height: 720
      fov: 90
  lidar:
    name: roof_lidar
    channels: 64
    range_m: 80

initial_state:
  pose:
    x: 0
    y: 0
    z: 0
    yaw: 0
  speed: 0

Supported external adapter names are carla, isaac, and gazebo. Existing string names such as mock, ros2, Python simulator paths, and native simulator paths still work. If initial_state.pose is present and ego is omitted, RoboCI derives ego.start from the pose so existing metrics, replay, and reports remain compatible. See docs/closed_loop_simulators.md for adapter-specific configuration.

Mock Closed-Loop Semantics

The built-in mock simulator is a deterministic smoke-test simulator, not a high-fidelity vehicle, robot, or physics simulator. It is intended to exercise RoboCI's closed-loop artifact pipeline locally: agent action, simulator step, episode JSONL, metrics, failures, replay, reports, API, and dashboard.

Mock scenarios use local Cartesian meters. Driving and ground-robot ego state is advanced with a simple timestep model: throttle adds acceleration, brake subtracts acceleration, speed is clamped at zero, and pose advances along yaw. When an ego route is present, the mock simulator steers toward the next route point instead of modeling tire dynamics. Actors move with constant velocity, optional waypoints, or the behavior pattern rules below.

Domain-aware mock runs add deterministic state for the marketplace domains:

  • Drone scenarios infer task_type: drone from environment.marketplace_pack_domain: drones, environment.metric_group: drones, or replay-scene domain metadata. Ego observations include z, altitude, vx, vy, and vz; drone actions can command vx, vy, vz, yaw_rate, linear_velocity, or target_speed. If no altitude is provided, environment.replay_scene.cruise_altitude_m is used.
  • Manipulator scenarios infer task_type: manipulator from manipulator domain/task metadata. Observations include a robot payload with joint_positions, joint_velocities, gripper, and a deterministic end_effector_pose. Joint actions can command joint_positions, joint_velocities, and gripper.

The mock simulator emits:

  • collision when ego is within 2 meters of an actor.
  • near_miss when ego is at least 2 meters and less than 4 meters from an actor.
  • offroad when ego leaves configured replay-scene bounds or is farther than environment.mock_offroad_margin from road paths or the fallback ego route.
  • goal_reached when ego is within ego.goal.tolerance meters of the goal.
  • timeout when elapsed time reaches duration_seconds.

Use mock for reproducible CI, dashboard demos, and adapter contract tests. Use CARLA, Isaac Sim, Gazebo, ROS2, Python simulators, or native simulators when the behavior under test depends on simulator-specific physics, sensors, maps, or robot dynamics.

Behavior Patterns

Actor behavior.pattern values are semantic labels used by marketplace scenarios, reports, and the deterministic mock simulator. External simulators may ignore them or map them to their own scenario logic. In the mock simulator, matching is case-insensitive.

Pattern values Mock behavior
Empty pattern, lane follow, constant speed, waypoint tracking with constant_velocity/constant_speed Actor holds its declared start speed and yaw unless waypoints are present.
blocked lane, stopped obstacle, false positive object Actor is static with speed set to zero.
rolling stop Actor decelerates each step until stopped.
lead vehicle braking, speed limit transition Actor starts decelerating after 1 second.
recovery after stop Actor brakes to a stop, then accelerates back toward its declared start speed.
yield negotiation, right of way yield Actor yields before entering the ego lane and stops at the yield line when near the ego path.
slow moving actor Actor speed is capped at 40% of its initial speed.
Any pattern containing crossing Actor keeps a single crossing direction and moves laterally across the ego lane instead of oscillating around it.
cut in, merge gap, right of way conflict Actor yaw is adjusted toward ego's lateral position.
tailgating pressure Actor tracks behind the ego pose and maintains pressure above the ego speed.
Emergency vehicles with lane follow, route deviation, or late detection Actor accelerates above its declared speed so emergency scenarios are behaviorally distinct.
unexpected actor, staggered obstacle, moving obstacle avoidance Actor maintains at least its declared speed so the object remains dynamic.
Any other value Actor keeps its configured behavior type, speed, yaw, and waypoints.

Waypoints take precedence over pattern motion. When behavior.waypoints is present, the actor moves toward each waypoint at its current speed and advances to the next waypoint when reached.

Suite YAML

Example:

id: urban_driving_smoke
name: Urban Driving Smoke Suite

scenarios:
  - ../scenarios/simple_lane_follow.yaml
  - ../scenarios/pedestrian_crossing.yaml
  - ../scenarios/obstacle_avoidance.yaml

gates:
  min_success_rate: 0.95
  max_collisions: 0
  max_regression_percent: 5
  min_safety_score: 80

Scenario paths are resolved relative to the suite file.

Schema owner:

roboci/suites/schema.py

Episode JSONL

Each line is one timestep.

{
  "t": 1.2,
  "ego": {
    "x": 12.4,
    "y": 0.3,
    "yaw": 0.01,
    "speed": 7.8
  },
  "actors": [
    {
      "id": "pedestrian_1",
      "type": "pedestrian",
      "x": 35.1,
      "y": -2.2,
      "speed": 1.2
    }
  ],
  "action": {
    "steer": 0.02,
    "throttle": 0.4,
    "brake": 0.0
  },
  "events": []
}

Written to:

runs/<run_id>/episodes/<scenario_id>.jsonl

External adapters also write a per-step log with normalized observation, sim_state, and simulator info:

runs/<run_id>/episodes/<scenario_id>.steps.jsonl

Run Directory

Closed-loop run:

runs/<run_id>/
  config.yaml
  summary.json
  provenance.json
  metrics.json
  regression.json
  failures.json
  report.html
  report.md
  episodes/
    <scenario_id>.jsonl
    <scenario_id>.steps.jsonl
  replay/
    <scenario_id>.timeline.json
    <scenario_id>.geojson
    <scenario_id>.timeline_events.json

Replay .geojson files use GeoJSON geometry containers. By default their coordinates are simulator-local meters, not WGS84 [longitude, latitude]. Each default file includes crs.name: urn:roboci:crs:local-simulator-meters plus a coordinate_system note so standard mapping tools do not silently interpret local trajectories as GPS positions.

Scenarios can opt into WGS84 export by adding a local tangent-plane transform under environment.replay_scene.coordinate_transform:

environment:
  replay_scene:
    coordinate_transform:
      type: local_tangent_plane_wgs84
      origin:
        latitude: 37.4219999
        longitude: -122.0840575
      heading_degrees: 0

When present, replay GeoJSON coordinates are [longitude, latitude], crs.name is urn:ogc:def:crs:OGC:1.3:CRS84, and the file includes the transform metadata used to convert local simulator meters.

Open-loop run:

runs/<run_id>/
  config.yaml
  summary.json
  provenance.json
  metrics.json
  regression.json
  report.html
  report.md
  openloop/
    predictions.jsonl
    per_sample_metrics.jsonl
    aggregate_metrics.json

summary.json

Closed-loop summary:

{
  "schema_version": "1.0",
  "run_id": "2026-06-11_05-32-19",
  "status": "pass",
  "suite_id": "urban_driving_smoke",
  "total_scenarios": 3,
  "passed_scenarios": 3,
  "failed_scenarios": 0,
  "success_rate": 1.0,
  "collision_count": 0,
  "offroad_count": 0,
  "safety_score": 100.0,
  "duration_seconds": 0.007,
  "commit_sha": "abc123",
  "branch": "main",
  "agent_version": null,
  "assertions": {
    "total": 2,
    "passed": 2,
    "failed": 0,
    "blocking_failed": 0
  },
  "provenance": {
    "git_commit_sha": "abc123...",
    "git_branch": "main",
    "dirty_working_tree": false,
    "agent_path": "examples/agents/rule_based_agent.py",
    "agent_file_hash": "sha256...",
    "suite_path": "examples/suites/smoke.yaml",
    "suite_file_hash": "sha256...",
    "scenario_file_hashes": {
      "examples/scenarios/simple_lane_follow.yaml": "sha256..."
    },
    "hash_algorithm": "sha256",
    "simulator_name": "mock",
    "python_version": "3.11.15",
    "roboci_version": "0.1.0",
    "created_at": "2026-06-14T06:14:44.133630Z",
    "hostname": "ci-worker-1"
  }
}

safety_score is a bounded score from 0 to 100 computed from the aggregate closed-loop metrics:

timeout_penalty = 10 when timeout is true, otherwise 0

safety_score = clamp(
  100
  - 50 * collision_count
  - 20 * offroad_count
  - 10 * red_light_violations
  - 5 * hard_brake_count
  - timeout_penalty,
  0,
  100
)

Collisions have the largest penalty, followed by offroad events, red-light violations, hard braking, and timeout. The score is clamped so repeated severe events cannot produce a negative value, and clean runs remain at 100.

The mock simulator emits red_light_violation events when the ego path crosses a red stop line declared under environment.traffic_lights or environment.traffic_signals:

environment:
  traffic_lights:
    - id: main_signal
      state: red
      stop_line:
        x: 0
        y: 0
        width: 6

stop_line may also be a segment with start and end points.

CARLA emits red_light_violation from the live traffic-light API when the ego vehicle is at a red signal. Gazebo can ingest simulator events from configured std_msgs/String JSON topics; publish {"type":"red_light_violation","traffic_light_id":"..."} to make the same safety metric path apply outside the mock.

Schema Version Policy

Run artifacts include schema_version. RoboCI treats additive fields as backward compatible when readers can ignore unknown keys. A future incompatible artifact layout must increment the schema version, keep old artifacts readable where practical, and document whether roboci compare can compare across versions. If a migration tool is needed, it should write new artifacts beside the original run rather than mutating historical evidence in place.

Open-loop summary:

{
  "schema_version": "1.0",
  "run_id": "openloop-2026-06-11_05-33-36",
  "status": "pass",
  "task": "trajectory_prediction",
  "sample_count": 2,
  "openloop": {
    "trajectory_ade": 0.0,
    "trajectory_fde": 0.0,
    "miss_rate": 0.0,
    "prediction_latency_ms": 0.001
  },
  "provenance": {
    "agent_path": "examples/agents/rule_based_agent.py",
    "agent_file_hash": "sha256...",
    "dataset_path": "examples/datasets/sample_openloop",
    "dataset_hash": "sha256...",
    "dataset_metadata_hash": "sha256...",
    "dataset_sample_hashes": {
      "examples/datasets/sample_openloop/samples/000001.json": "sha256..."
    },
    "dataset_annotation_hashes": {
      "examples/datasets/sample_openloop/annotations/trajectories.json": "sha256..."
    },
    "hash_algorithm": "sha256"
  }
}

provenance.json

Every completed run writes provenance.json, and summary.json embeds the same object under provenance. The file is intended to make regression comparisons auditable.

Common fields:

  • git_commit_sha, git_branch, dirty_working_tree
  • agent_path, agent_file_hash
  • hash_algorithm
  • python_version, roboci_version
  • created_at, hostname
  • collection_errors when a value could not be collected

Closed-loop provenance also includes:

  • suite_path, suite_file_hash
  • scenario_file_hashes
  • simulator_name

Open-loop provenance also includes:

  • dataset_path
  • dataset_hash
  • dataset_metadata_hash
  • dataset_sample_hashes
  • dataset_annotation_hashes

Provenance collection is best-effort. Missing Git, unreadable files, or hostname lookup failures are recorded in collection_errors instead of failing the run.

GET /runs does not return this full object. It replaces summary.provenance with provenance_summary, which keeps the run list small by reporting identifying fields, dataset hash, and hash counts instead of every scenario, sample, and annotation hash.

metrics.json

Closed-loop:

{
  "aggregate": {},
  "assertions": {
    "total": 2,
    "passed": 1,
    "failed": 1,
    "blocking_failed": 1
  },
  "scenarios": {
    "pedestrian_crossing_001": {
      "scenario_id": "pedestrian_crossing_001",
      "name": "Pedestrian Crossing",
      "tags": ["pedestrian", "crossing", "urban"],
      "status": "failure",
      "metrics": {},
      "assertion_summary": {
        "total": 2,
        "passed": 1,
        "failed": 1,
        "blocking_failed": 1
      },
      "assertions": [
        {
          "name": "stop_before_pedestrian",
          "type": "min_distance",
          "severity": "critical",
          "status": "fail",
          "passed": false,
          "actual": 1.2,
          "expected": ">= 1.5",
          "message": "stop_before_pedestrian fail: actual 1.2, expected >= 1.5"
        }
      ],
      "events": [],
      "episode_path": "...",
      "replay_timeline_path": "...",
      "replay_geojson_path": "..."
    }
  }
}

Open-loop:

{
  "openloop": {
    "trajectory_ade": 0.0,
    "trajectory_fde": 0.0,
    "miss_rate": 0.0
  }
}

Open-loop perception metrics are computed from structured annotations when present. occupancy_iou compares binary occupancy grids from occupancy_grid, occupancy_mask, occupancy_map, or occupancy; scalar occupancy_iou fields in model output are ignored. detection_map computes class-aware average precision from detections, objects, or boxes lists using bbox/box/bounding_box coordinates and optional score or confidence; scalar detection_map fields are ignored. Datasets without those annotations keep these metrics at 0.0.

regression.json

Written by roboci compare or by gate evaluation.

{
  "status": "fail",
  "failures": [
    "safety_score regressed by 8.10%"
  ],
  "deltas": {
    "safety_score": {
      "current": 91.9,
      "baseline": 100.0,
      "delta": -8.1,
      "percent": -8.1,
      "direction": "higher_is_better",
      "regressed": true
    }
  }
}

When a numeric baseline is 0 and the current value is nonzero, percent is null because a percent change from zero is undefined. Regression detection handles zero-baseline changes explicitly rather than reporting a fake 100% delta.

The dashboard regression page reads this file through GET /runs/{run_id}.

baseline.json

Written by roboci baseline create <name> --from <run-dir> under baselines/<name>/. This file is the marker for a first-class named baseline; directories under baselines/ without baseline.json are ignored by roboci baseline list and GET /baselines.

{
  "name": "smoke",
  "source_run_id": "2026-06-14_10-00-00",
  "source_run_dir": "/path/to/workspace/runs/2026-06-14_10-00-00",
  "created_at": "2026-06-14T10:05:00+00:00",
  "description": "Known-good smoke run",
  "path": "baselines/smoke"
}

source_run_dir is the resolved source directory, so creating from runs/latest records the concrete run directory rather than the latest selector.

Failure Object

Failure records include enough replay location data for the dashboard to open the incident moment directly. Closed-loop failures should include run_id, scenario_id, timestamp, frame_index, and either replay_url or replay_path when replay artifacts are available.

{
  "failure_id": "fail_001",
  "run_id": "2026-06-11_05-32-19",
  "scenario_id": "pedestrian_crossing_001",
  "sample_id": null,
  "timestamp": 12.4,
  "frame_index": 124,
  "replay_path": "replay/pedestrian_crossing_001.timeline.json",
  "replay_url": "/runs/2026-06-11_05-32-19/scenarios/pedestrian_crossing_001?t=12.4&frame=124",
  "severity": "critical",
  "tags": ["collision", "pedestrian_failure"],
  "notes": "Collision detected during scenario replay.",
  "owner": null,
  "status": "open",
  "fixed_in_run_id": null,
  "regressed_in_run_id": null,
  "status_history": [
    {
      "changed_at": "2026-06-11T05:02:41Z",
      "from_status": "open",
      "to_status": "acknowledged"
    }
  ],
  "created_at": "2026-06-11T04:32:10Z"
}

Allowed lifecycle statuses are open, acknowledged, assigned, in_progress, fixed, verified, regressed, and wont_fix. owner, fixed_in_run_id, and regressed_in_run_id are optional and may be null. status_history is stored in the same failures.json file and records status transitions as {changed_at, from_status, to_status} entries.

Closed-loop regression comparison may populate lifecycle fields automatically. If a baseline failure's scenario succeeds in the current run, RoboCI writes the current run id to fixed_in_run_id and can transition the failure to fixed. If a current failure appears on a scenario that succeeded in the baseline, RoboCI writes the current run id to regressed_in_run_id and can transition the new failure to regressed. Manual verified and wont_fix statuses are preserved.

Replay Timeline

replay/<scenario_id>.timeline.json stores ordered episode frames. Each exported frame includes a zero-based frame_index; each raw frame event in events also includes the frame_index where it occurred and a numeric timestamp.

{
  "scenario_id": "pedestrian_crossing_001",
  "duration_seconds": 20.0,
  "frames": [
    {
      "t": 12.4,
      "frame_index": 124,
      "events": [
        {
          "type": "collision",
          "timestamp": 12.4,
          "frame_index": 124
        }
      ]
    }
  ],
  "events": [
    {
      "type": "collision",
      "timestamp": 12.4,
      "frame_index": 124
    }
  ]
}

replay/<scenario_id>.timeline_events.json stores the readable event timeline used by the dashboard scenario detail page. RoboCI derives it after simulation from the recorded episode, metrics, and assertion results, so it does not change core simulator behavior. Each event includes:

  • timestamp
  • severity
  • type
  • title
  • description
  • actor_id and related_actor_id when the event can be associated with an actor
  • frame_index when the event maps to a replay frame

The exported event types include scenario_start, goal_reached, collision, near_miss, offroad, hard_brake, timeout, and assertion_failure. scenario_start is anchored to the first replay frame when frames exist, so clicking the event in the dashboard seeks to the same timestamp shown in the timeline. Older runs may contain replay/<scenario_id>.events.json; new runs write only timeline_events.json.

[
  {
    "type": "scenario_start",
    "timestamp": 0.1,
    "frame_index": 0,
    "severity": "info",
    "title": "Scenario started",
    "description": "Scenario pedestrian_crossing_001 replay started."
  },
  {
    "type": "collision",
    "timestamp": 12.4,
    "frame_index": 124,
    "severity": "critical",
    "title": "Collision",
    "description": "Collision detected involving actor pedestrian_1.",
    "actor_id": "pedestrian_1",
    "related_actor_id": "pedestrian_1"
  }
]

Open-loop Dataset

datasets/
  val_set/
    metadata.yaml
    samples/
      000001.json
      000002.json
    sensor/
      camera_front/
    annotations/
      trajectories.json
      objects.json
      lanes.json

Sample:

{
  "sample_id": "000001",
  "input": {
    "ego": {"x": 0, "y": 0, "speed": 5},
    "candidate_trajectory": [
      {"x": 1, "y": 0},
      {"x": 2, "y": 0}
    ]
  },
  "ground_truth": {
    "control": {"steer": 0.0, "throttle": 0.4, "brake": 0.0}
  }
}

Annotations can add ground-truth trajectories:

{
  "000001": [
    {"x": 1, "y": 0},
    {"x": 2, "y": 0}
  ]
}