RoboMetrics¶

RoboMetrics is a lightweight Python library for evaluating robotics and autonomy outputs with typed, local metric functions. It covers trajectories, multi-modal prediction, driving safety, comfort, physics feasibility, task outcomes, manipulation signals, calibration, coverage, and experiment comparison without requiring a simulator, dashboard, ROS install, or cloud service.

Local-first Typed Python API CLI-ready JSON Simulator-agnostic

Start in Python

Install the package, run Evaluator, and inspect EvaluationResult output.

Start from files

Validate CSV or JSON policy outputs, evaluate them, and generate a report.

Pick metrics

Browse the metric families, units, directionality, and edge-case behavior.

Prefer a notebook? Open the Colab demo and run the same local-first evaluation flow without cloning the repository.

Installation¶

pip install robometrics

Install optional I/O support when you want CSV helpers that use pandas:

pip install "robometrics[io]"

Install only the extras you need:

pip install "robometrics[mcap]"      # MCAP JSON-message adapter
pip install "robometrics[loggers]"   # W&B and MLflow logging helpers

30-Second Evaluation¶

import numpy as np
from robometrics import Evaluator

predictions = [
    np.array([[0.0, 0.0], [1.1, 0.0], [2.0, 0.0]]),
    np.array([[0.0, 0.0], [1.3, 0.0], [2.5, 0.0]]),
]
ground_truths = [
    np.array([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]]),
    np.array([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]]),
]

result = Evaluator().evaluate_dataset(
    predictions=predictions,
    ground_truths=ground_truths,
    metrics=["ade", "fde"],
    thresholds={"ade": 0.5, "fde": 1.0},
    bootstrap_ci=1000,
)

print(result.to_markdown())

Choose Your Path¶

Evaluate policy outputs

Use the policy-output guide when your data comes from Isaac, MuJoCo, LeRobot, ROS 2, logs, or a custom rollout exporter.

Compare checkpoints

Use Comparing Policies for baseline-vs-candidate comparisons and EvaluationResult.compare().

Automate CI checks

Use CI Integration to wire robometrics compare into GitHub Actions.

Extend the registry

Use Writing a Pack when your project needs custom metrics while keeping the same evaluator and CLI surface.

What To Read Next¶

Need	Page
Confirm project fit and boundaries	What RoboMetrics Is And Isn't
Learn accepted input shapes, units, and return types	Input And Output Contract
Try the fastest local workflow	Quickstart
Run multiple metrics together	Evaluation Guide
Load directories or matched datasets	Loader Examples
Handle result JSON versions	Result Schema Migration
Use the shell interface	CLI Commands
Browse metric units and direction	Metric Catalog
Check exported symbols	API Reference

RoboMetrics is the metrics layer. It does not replace a simulator, planner, replay system, dashboard, or experiment tracker; it gives those systems a small, explicit evaluation contract.