Testing

RoboCI uses pytest for Python tests and npm run lint plus npm run build for frontend validation.

Run Python Tests

python -m pytest

Run Frontend Checks

cd frontend
npm run lint
npm run build

Test Files

tests/test_scenario_loader.py

Covers:

  • Scenario YAML loading.
  • Scenario tag validation and marketplace tag backfill.
  • Suite YAML loading.
  • Relative scenario resolution from suite files.

tests/test_runner.py

Covers:

  • Mock simulator goal reaching.
  • Closed-loop runner artifact writing.

tests/test_simulator_adapters.py

Covers:

  • Simulator registry names and config mappings.
  • CARLA, Isaac, and Gazebo action conversion helpers.
  • Observation and sensor frame serialization.
  • Clear missing-dependency errors for optional simulator SDKs.
  • Mocked Isaac and Gazebo reset/step paths without installing external simulators.

tests/test_metrics.py

Covers:

  • Closed-loop metric computation.
  • Collision count.
  • Hard brake count.
  • Safety score.

tests/test_regression.py

Covers:

  • Regression comparator detecting safety drops.
  • regression.json writing.

tests/test_launch_cli.py

Covers:

  • roboci demo orchestration.
  • Benchmark metadata listing/loading.
  • roboci benchmarks run smoke.
  • Invalid benchmark handling.
  • roboci bundle zip creation.

tests/golden/

Covers deterministic hand-computable metric and regression examples used by the metric documentation.

tests/test_openloop_metrics.py

Covers:

  • Open-loop dataset loading.
  • Trajectory ADE/FDE.
  • Aggregate metrics.
  • Open-loop run provenance for dataset metadata, sample, annotation, and composite dataset hashes.

tests/test_provenance.py

Covers:

  • Closed-loop source file hashing.
  • Open-loop dataset hashing.
  • Git-unavailable fallback.
  • Clean Git working tree detection.

tests/test_replay_export.py

Covers:

  • Timeline export.
  • GeoJSON export.
  • Normalized timeline events export.
  • Legacy events artifact compatibility.

tests/test_failure_detector.py

Covers:

  • Failure generation from collision events.
  • Pedestrian failure tagging.

tests/test_api_routes.py

Covers:

  • API health route.
  • Scenario route.
  • Workspace-aware run path resolution.
  • Open-loop run type labeling.
  • Regression count enrichment.

What to Test When Changing Code

Changing scenario or suite schemas

Run:

python -m pytest tests/test_scenario_loader.py

If schema changes affect run artifacts, launch options, or dashboard payloads, also run:

python -m pytest tests/test_api_routes.py tests/test_runner.py

Changing simulator or runner behavior

Run:

python -m pytest tests/test_runner.py tests/test_simulator_adapters.py tests/test_metrics.py tests/test_replay_export.py

CARLA, Isaac Sim, and Gazebo integration checks should only run in environments with those simulator stacks installed. Gate them explicitly with:

ROBOCI_RUN_CARLA_TESTS=1 python -m pytest tests/test_simulator_adapters.py
ROBOCI_RUN_ISAAC_TESTS=1 python -m pytest tests/test_simulator_adapters.py
ROBOCI_RUN_GAZEBO_TESTS=1 python -m pytest tests/test_simulator_adapters.py

The CARLA Compose smoke path is also opt-in and requires Docker, a runner that can launch the CARLA image, and suitable graphics/runtime support:

docker compose --profile carla up --build carla carla-roboci

Set ROBOCI_CARLA_PYTHON_PACKAGE to the matching CARLA Python API wheel/package for the prepared runner. The default runner image builds without bundling the CARLA client and exits before the scenario run if the API is missing.

The same live checks are wired into .github/workflows/simulator-integration.yml as manual jobs for self-hosted runners labeled with the matching simulator stack. The default GitHub-hosted CI continues to run mocked adapter tests only.

Changing API behavior

Run:

python -m pytest tests/test_api_routes.py

Changing frontend

Run:

cd frontend
npm run lint
npm run build

Changing regression logic

Run:

python -m pytest tests/test_regression.py tests/test_api_routes.py

Changing demo, benchmarks, or bundle behavior

Run:

python -m pytest tests/test_launch_cli.py tests/golden

Manual Smoke Tests

Closed-loop:

roboci run --suite examples/suites/smoke.yaml --agent examples/agents/rule_based_agent.py --sim mock

Known collision failure:

roboci run --suite examples/suites/failing_collision.yaml --agent examples/agents/collision_test_agent.py --sim mock

This is expected to return a non-zero exit code after writing collision failure artifacts.

Open-loop:

roboci openloop run --dataset examples/datasets/sample_openloop --model examples/agents/rule_based_agent.py --task trajectory_prediction

Report:

roboci report --run runs/latest
roboci report --run runs/latest --format markdown

Demo and benchmark:

roboci demo
roboci benchmarks list
roboci benchmarks run smoke
roboci bundle runs/latest --output roboci-run.zip

API:

roboci serve

Then:

http://127.0.0.1:8000/health
http://127.0.0.1:8000/runs