Scenario Marketplace¶

The Scenario Marketplace is RoboCI's built-in catalog of installable closed-loop scenario packs. A pack is a curated category such as urban-v1, manipulator-pick-v1, or drone-navigation-v1; installing it writes concrete scenario YAML files and a suite YAML into the local workspace. Packs are domain-aware across ground_robotics, manipulators, and drones.

User Workflow¶

List available packs:

roboci scenarios list

Install a pack:

roboci scenarios install urban-v1
roboci scenarios install highway-v2

Install a second copy that is isolated by pack version:

roboci scenarios install urban-v1 --versioned

Remove an installed pack from a workspace:

roboci scenarios remove urban-v1
roboci scenarios remove urban-v1 --versioned

Removing a pack does not delete historical runs. If existing run artifacts reference a scenario from the removed install variant, RoboCI marks that scenario entry in metrics.json with removed: true and writes removed_scenario_refs.json into the run directory. Versioned installs are matched by marketplace_pack_install_id or provenance path where available; ambiguous legacy runs are not mutated when removing a versioned install.

Run an installed suite:

roboci run \
  --suite examples/suites/urban-v1.yaml \
  --agent examples/agents/rule_based_agent.py \
  --sim mock

Community Seed Pack¶

RoboCI also ships a hand-authored community seed suite outside the generated marketplace catalog:

roboci run --suite examples/suites/community_seed.yaml --agent examples/agents/rule_based_agent.py --sim mock

The seed pack lives under examples/scenarios/community_seed/ and includes six non-trivial scenarios covering occluded pedestrian crossing, yielding cross traffic, emergency vehicle overtake, lead-vehicle braking, cyclist cut-in, and construction-zone clearance. Every scenario has assertions, replay-scene metadata, and documented expected outputs in examples/scenarios/community_seed/README.md plus examples/suites/community_seed_expected_summary.json. With the reference rule-based agent, the verified expected result is status: pass, success_rate: 1.0, zero collisions, and 18 passing assertions.

Installed files:

examples/scenario_marketplace/<pack-id>/<scenario-id>.yaml
examples/suites/<pack-id>.yaml

Versioned installs use a version-suffixed install id so they can coexist with the default install:

examples/scenario_marketplace/<pack-id>--<version>/<scenario-id>.yaml
examples/suites/<pack-id>--<version>.yaml

The dashboard reads the catalog from /scenario-marketplace and shows it on the Scenarios page. /scenarios?domain=<domain> filters marketplace packs by robotics domain, /scenarios?tag=<tag> filters marketplace packs by generated scenario tags plus local example scenarios by YAML tags, and /scenarios?q=<text> searches pack names, ids, tasks, metrics, adapter requirements, simulator requirements, tags, descriptions, and scenario names. Pack cards show install status and include install/remove controls that call the same install/remove flow as the CLI.

Source Of Truth¶

Marketplace packs are defined in:

roboci/scenarios/marketplace_catalog.yaml

Local workspaces can also add catalog entries without editing package code by placing a manifest at:

examples/scenario_marketplace/<pack-id>/pack.yaml

The manifest can contain the pack mapping directly or under a top-level pack: key. The id must match the directory name. Workspace manifests are scanned by marketplace_catalog(), marketplace_pack(), and install_pack() along with the packaged catalog.

The important pieces are:

Symbol	Purpose
`marketplace_catalog.yaml`	Pack-level catalog entries. Add or edit categories here.
`SCENARIO_PATTERNS`	Reusable pattern names used to generate scenario names, tags, and behaviors.
`load_pack_definitions()`	Loads the YAML catalog for validation and generation.
`load_workspace_pack_definitions()`	Scans `examples/scenario_marketplace/*/pack.yaml` for workspace-local pack manifests.
`marketplace_catalog()`	Returns the API/dashboard catalog.
`install_pack()`	Writes generated scenario YAML files and the matching suite.
`remove_pack()`	Removes generated scenarios and the matching suite from a workspace while preserving workspace-local `pack.yaml` manifests and tombstoning matching historical run refs.
`build_scenario()`	Builds the actual scenario YAML payload for each scenario.
`marketplace_tags_for_scenario()`	Computes canonical tags for an installed marketplace scenario id.
`validate_pack_definitions()`	Validates duplicate ids, allowed domains, metric names, required metadata, and safe install paths.
`_scenario_metadata()`	Builds generated IDs, names, descriptions, difficulty, and tags.

Edit An Existing Pack¶

To edit a pack, update the matching entry in roboci/scenarios/marketplace_catalog.yaml.

Example:

- id: urban-v1
  name: Urban Core
  category: Urban Driving
  description: Dense city driving with pedestrians, parked vehicles, cyclists, and low-speed conflicts.
  domain: ground_robotics
  road_type: urban
  weather: clear
  speed_limit: 40
  actor_type: pedestrian
  scenario_count: 24
  difficulty: mixed
  supported_tasks: [route_following, obstacle_avoidance, collision_avoidance]
  metrics: [collision_rate, route_completion, off_path_error, time_to_collision, comfort_score]
  adapter_requirements: [2D pose observation, speed control, steering control]
  simulator_requirements: [ego_pose, trajectory, obstacles, collision_events, lane_reference]
  tags: [urban_driving, ground_robotics, marketplace]
  version: v1
  license: MIT

Field notes:

Field	Meaning
`id`	Stable install id. Use kebab-case plus version, for example `urban-v1`.
`name`	Human-readable pack name shown in CLI/API/dashboard.
`category`	Marketplace category shown on pack cards and scenario names.
`description`	One-sentence summary of the pack's coverage.
`domain`	Robotics domain. Supported values are `ground_robotics`, `manipulators`, and `drones`. Omitted legacy packs default to `ground_robotics`.
`road_type`	Written into each generated scenario under `environment.road_type`.
`weather`	Written into each generated scenario under `environment.weather`.
`speed_limit`	Written into each scenario and used to derive ego start speed.
`actor_type`	Actor type used for generated scenario actors.
`scenario_count`	Number of scenarios generated for the pack. Keep this between 20 and 30 unless tests and product expectations change.
`difficulty`	Pack-level difficulty label shown in the marketplace.
`supported_tasks`	Domain task types shown in the marketplace and reports.
`metrics`	Domain metric names. Built-in names must be valid for the selected domain unless a metric is declared as `{"name": "...", "custom": true}`.
`adapter_requirements`	Agent or adapter capabilities required to run the pack.
`simulator_requirements`	Simulator observations/events required for faithful evaluation.
`tags`	Pack-level search tags. Generated scenario tags remain scenario-specific.
`version`	Pack version shown in API, dashboard, reports, and provenance. Defaults to the id suffix such as `v1`.
`license`	Pack license shown in the marketplace. Defaults to `MIT`.
`domain_metadata`	Domain-specific data contract. Defaults are provided for legacy ground robotics packs.

Avoid changing an existing id unless you intend to break the old install command and suite path.

Create A New Pack¶

Add a new entry to roboci/scenarios/marketplace_catalog.yaml.
Pick a stable id such as night-v1, rural-v1, or warehouse-v1.
Choose domain: ground_robotics, domain: manipulators, or domain: drones.
Set scenario_count to 20-30.
Choose an actor_type that the simulator and downstream tooling understand.
Run the catalog and install checks below.

For a workspace-local pack, create examples/scenario_marketplace/<pack-id>/pack.yaml instead of editing marketplace_catalog.yaml. Use the same fields as packaged packs:

pack:
  id: warehouse-v1
  name: Warehouse Navigation
  category: Warehouse
  description: Indoor aisle navigation with static and dynamic obstacles.
  domain: ground_robotics
  road_type: warehouse
  weather: indoor
  speed_limit: 12
  actor_type: obstacle
  scenario_count: 20
  difficulty: medium
  supported_tasks: [route_following, obstacle_avoidance]
  metrics: [collision_rate, route_completion, off_path_error, time_to_collision, comfort_score]

Minimal example:

- id: rural-v1
  name: Rural Roads
  category: Rural
  description: Two-lane roads, limited shoulders, farm access, animals, and long sight lines.
  domain: ground_robotics
  road_type: rural
  weather: clear
  speed_limit: 80
  actor_type: vehicle
  scenario_count: 24
  difficulty: medium
  supported_tasks: [route_following, obstacle_avoidance]
  metrics: [collision_rate, route_completion, off_path_error, time_to_collision, comfort_score]
  adapter_requirements: [2D pose observation, speed control, steering control]
  simulator_requirements: [ego_pose, trajectory, obstacles, collision_events, lane_reference]
  tags: [rural, ground_robotics, marketplace]
  version: v1
  license: MIT

After adding it, this command should appear:

roboci scenarios list

Expected line:

rural-v1: Rural Roads (Rural, 24 scenarios) - roboci scenarios install rural-v1

Change Generated Scenario Behavior¶

Use build_scenario() when pack metadata is not enough. This function controls:

generated scenario IDs, names, and descriptions through _scenario_metadata()
ego start pose, speed, and goal
actor start pose, speed, type, and behavior
environment metadata
success and failure criteria

Keep generated scenarios valid against:

roboci/scenarios/schema.py

If you add new schema fields, update both the marketplace generator and the data format docs.

Generated scenarios include tags and marketplace environment metadata such as marketplace_pack, marketplace_pack_name, marketplace_pack_version, marketplace_pack_domain, marketplace_task_type, metric_group, domain_metrics, and domain_metadata. The loader also backfills canonical marketplace tags and pack metadata for older installed scenario YAML files when possible, using the pack id and scenario id. This keeps existing examples/scenario_marketplace/<pack-id>/*.yaml files filterable and ensures run artifacts include pack name, domain, task type, metric group, and tags.

Change Scenario Patterns¶

SCENARIO_PATTERNS controls the repeated scenario variants inside each pack. Adding a pattern automatically expands future generated names and tags. If a pack has more scenarios than patterns, patterns repeat from the beginning.

Use short behavior-focused phrases:

"lane follow"
"lead vehicle braking"
"occluded crossing"

Avoid pack-specific terms in SCENARIO_PATTERNS; put category-specific language in the pack's category, description, road_type, or build_scenario() branching.

Domain Metric Registry¶

Built-in metrics are validated by domain:

Domain	Built-in metrics
`ground_robotics`	`collision_rate`, `route_completion`, `off_path_error`, `time_to_collision`, `comfort_score`
`manipulators`	`grasp_success_rate`, `pose_error`, `contact_stability`, `collision_rate`, `task_completion`
`drones`	`trajectory_error_3d`, `obstacle_clearance`, `geofence_violation_rate`, `landing_precision`, `energy_efficiency`

Use a custom metric object only when the metric is intentionally outside the built-in registry:

"metrics": [
    {"name": "workspace_heatmap_score", "custom": True}
]

Domain metadata is stored in generated scenario environment.domain_metadata, then propagated into run metrics.json, summary.json, provenance.json, the dashboard, and static reports. Pack name, pack version, domain, task type, and metric group are shown in dashboard metadata, run detail provenance, scenario result rows, report.md, and report.html. Manipulator packs can declare fields such as joint_state, end_effector_pose, object_pose, gripper_state, contact_events, collision_events, and task_success_condition. Drone packs can declare fields such as pose_6dof, velocity_3d, altitude, wind_condition, obstacle_clearance, geofence_violation, landing_zone_error, and battery_state.

Validate Changes¶

Run the Python tests:

python -m pytest

If your shell does not have pytest, activate your virtual environment or use the Python interpreter where RoboCI is installed:

python -m pytest

Check the CLI:

roboci scenarios list
roboci scenarios install <pack-id> --workspace /tmp/roboci-marketplace-check
roboci scenarios install <pack-id> --workspace /tmp/roboci-marketplace-check --versioned
roboci scenarios remove <pack-id> --workspace /tmp/roboci-marketplace-check

Install validation writes each generated scenario YAML and immediately reloads it through load_scenario(). A malformed generated scenario fails the install before the suite is written.

Check that the installed suite loads:

python -c "from pathlib import Path; from roboci.suites.loader import load_suite_scenarios; print(len(load_suite_scenarios(Path('/tmp/roboci-marketplace-check/examples/suites/<pack-id>.yaml'))))"

Build the dashboard after changing API or frontend catalog fields:

cd frontend
npm run build

API Contract¶

GET /scenario-marketplace returns a list of packs. Each pack includes:

{
  "id": "urban-v1",
  "name": "Urban Core",
  "category": "Urban Driving",
  "description": "Dense city driving with pedestrians, parked vehicles, cyclists, and low-speed conflicts.",
  "domain": "ground_robotics",
  "domain_label": "Ground Robotics",
  "version": "v1",
  "license": "MIT",
  "install_command": "roboci scenarios install urban-v1",
  "versioned_install_command": "roboci scenarios install urban-v1 --versioned",
  "remove_command": "roboci scenarios remove urban-v1",
  "versioned_remove_command": "roboci scenarios remove urban-v1 --versioned",
  "install_status": "installed",
  "installed": true,
  "versioned_install_status": "available",
  "versioned_installed": false,
  "scenario_count": 24,
  "difficulty": "mixed",
  "supported_tasks": ["route_following", "obstacle_avoidance", "collision_avoidance"],
  "metrics": ["collision_rate", "route_completion", "off_path_error", "time_to_collision", "comfort_score"],
  "metric_group": "ground_robotics",
  "adapter_requirements": ["2D pose observation", "speed control", "steering control"],
  "simulator_requirements": ["ego_pose", "trajectory", "obstacles", "collision_events", "lane_reference"],
  "tags": ["urban_driving", "ground_robotics", "marketplace"],
  "domain_metadata": {
    "ego_pose": true,
    "trajectory": true,
    "obstacles": true,
    "path_lane_reference": true,
    "collision_events": true,
    "comfort_safety_metrics": true
  },
  "road_type": "urban",
  "speed_limit": 40,
  "scenarios": [
    {
      "id": "urban_v1_001",
      "name": "Urban Driving Lane Follow 01",
      "difficulty": "easy",
      "tags": ["urban_driving", "lane_follow", "mixed"]
    }
  ]
}

GET /scenario-marketplace/{pack_id} returns one pack or 404 for an unknown id.

POST /scenario-marketplace/{pack_id}/install installs a pack into the current workspace and returns the refreshed pack metadata. DELETE /scenario-marketplace/{pack_id}/install removes generated scenarios and the matching suite, tombstones matching historical run refs for the removed install variant, then returns refreshed pack metadata. Workspace-local pack.yaml manifests are preserved so custom packs remain discoverable after removal. The remove response includes result.tombstone_updates.runs_updated and the removed scenario ids. Both endpoints are mutating local workspace operations and require either a localhost Origin header or X-RoboCI-Local: 1 for non-browser local clients.

When you add fields to the API response, update:

frontend/app/scenarios/page.tsx
frontend/lib/api.ts if a new client helper is needed
tests/test_api_routes.py
this guide