# Policy Evaluation The evaluation system measures how well a trained policy performs in closed loop in simulation, on scenes and tasks automatically generated by the datagen system. The evaluation system runs on your machine, from a workstation with a single GPU to a multi-GPU machine that evaluates hundreds of episodes at a time spread over several GPUs. It uses a queue to distribute episodes over GPU-parallel simulation environments in IsaacSim/IsaacLab. It works with any policy that can be served over a WebSocket, such as a hierarchical VLA. Each evaluation episode can start from a randomly-generated unseen scene and task definition or from the scene and task definition of an episode already recorded in a dataset (with the same robot pose, block poses, block sizes, table height and goal). The policy has to reach the task's goal from that initial state. Because the eval scene can be rebuilt and initialized from a dataset episode, success metrics are indicative of policy performance rather than of differences between the training and eval setups. ## What you can do with it - Benchmark ManiFlow (low-level) policies, alone or under a Molmo2 (high-level) policy, on the episodes of a dataset. - Run ablations that replace the high-level or the low-level policy with an oracle, to help isolate where errors might be coming from. - Replay the expert actions recorded in a dataset in an open-loop fashion, as a sanity check of the simulation settings, the dataset, and the evaluation system overall. - Plug in a new model by implementing a small policy-server interface. - Capture the episodes on which a policy failed and send them back to data generation, which plans new expert demonstrations and recovery behaviors from those states ([TAMPER](https://github.com/allenai/ToyBlocks/blob/main/docs/tamper.md)). ## Design goals for our evaluation system - **Scalability and GPU parallelization.** An eval runs many episodes at once at two levels. On one GPU, many parallel environments (up to 50 per GPU by default) live in one Isaac scene, and the policy server answers rollout requests from all of them with one GPU-batched inference call. Across GPUs, several eval workers (`--num-eval-gpus`) pull episodes from a single episode queue, so adding GPUs adds throughput without changing which episodes are evaluated. (The Molmo2 + ManiFlow server accepts a single eval worker, so it scales through environments only.) - **Reproducible statistically rather than bit-by-bit.** Every episode starts from a randomly-generated unseen scene and task or from the scene and task in a recorded dataset episode, handed out by an episode queue. Seeds are fixed. A rerun therefore gives the same success rates within some noise. Individual trajectories, however, are *not* guaranteed to match bit-for-bit between runs. Sharing one physics scene across environments is what makes this evaluation system fast, but it is also the main reason results are not bit-for-bit reproducible (see reasons below). - **One pipeline from one GPU to many.** The same evaluation pipeline runs a smoke test on a one-GPU workstation and a full evaluation over several GPUs; only the profile flags change. - **Faithful to data generation.** When the eval scene (robot, scene preset, table height, block sizes, cameras) is taken from an existing episode in a dataset, provenance is recorded, and a scene verification step compares the eval scene against the recording before any numbers are reported. - **Model-agnostic.** Any policy that implements the `BasePolicy` interface can be served over WebSocket. Observation transforms convert the simulator's observations into the format each model expects, so the eval client is the same for every model. - **Decomposable.** Policies that are split into a high-level part (which decides what to do next, e.g. "pick the red block") and a low-level part (which moves the arm) can have either part replaced by an oracle. This isolates whether a failure comes from the high-level decision or from its execution. - **Easy definition of metrics.** A new metric is one tensor with one value per environment, added in the task's `evaluate()`. Per-episode summaries, aggregates across episodes, per-step CSVs and Weights & Biases (W&B) tables are produced automatically, with no additional logging code. See [Metrics](evals/metrics.md) for the built-in metrics and how to add your own. - **Automatic definition of partial progress metrics** Each scene and task definition contains a PDDL logical goal that is defined over a set of logical predicates over the objects in the scene. We leverage this to measure how many predicates of the logical goal specification have been achieved at a particular point in an episode. Counting how many of these goal predicates are successful allows us to automatically have a partial progress metric estimated for every episode, without the need to write custom code. Moreover, the partial progress metrics are measured the same way for all tasks. - **Diagnosable.** Runs can produce videos, per-episode metrics, detailed diagnostic figures, W&B tables, checks of the exact inputs the policy received, and LeRobot recordings of the rollouts. This makes it easier to detect bugs during the development process. :::{note} **Why results are not bit-for-bit reproducible.** Two runs of the same eval give the same success rates within noise, but not necessarily identical trajectories: - **All parallel environments share one PhysX scene.** Isaac Sim 5.1 (through Isaac Lab 2.3.2) steps every environment in a single GPU physics solve. One environment's result can depend on which other environments are in the batch and what they are doing, which changes with the number of environments, the queue order, episodes ending at different times, and idle environments whose queue has run out. - **The GPU physics solver is not deterministic.** Contact-rich PhysX simulation is not reproducible across GPUs, drivers or Isaac versions. `--deterministic-cuda` turns on PhysX enhanced determinism, a single solver partition and deterministic cuDNN. That narrows the gap on one machine at a speed cost, but it does not guarantee identical trajectories across hardware. - **There is no per-environment physics step.** Any extra physics step (for example a settle after a controller gain switch) advances every environment, and PhysX keeps contact history when objects are teleported at reset, so earlier episodes in a run can affect later ones. - **Some render state and the reseed are shared.** Lighting, and camera intrinsics under tiled rendering, are shared by all environments in the scene, and the per-episode reseed applies to the whole process, so the policy's sampling noise depends on scheduling. In practice: compare runs by aggregate metrics over many episodes, use replay or a single environment to debug one trajectory, and do not expect runs with a different number of environments or GPUs to give identical per-episode outcomes. ::: ## Before you run the eval pipeline: creating an environment and task `evaluate.py` runs an Isaac Lab gym task (`--task `). Before a policy can be evaluated on a scenario, a task for it has to exist and be registered, and it has to define three things: - the scene: the robot, the table and the blocks; - how each episode starts; - an `evaluate()` method that returns a per-step `success` flag and any metric stats. Evaluation scores episodes on that `success` signal and those stats, not on an RL reward. **Blocks tasks are generated, not written by hand.** Both generators below render the domain template in `src/toyblocks/tamp/blocks/template/` into a package `src/toyblocks/tasks/autogen//` (gitignored) that registers the gym id `-v0`. Code generation also writes the success criterion and the metrics into the task's `evaluate()`: - `success` means the PDDL goal holds: every conjunct is true, the goal blocks have settled and the gripper is open; - one `goal_conj_` stat per conjunct. So there is no reward or metric code to write: "Partial progress from the PDDL goal" below explains how these stats become a partial progress score. **From a dataset:** [`scripts/generate_task_from_dataset.py`](https://github.com/allenai/ToyBlocks/blob/main/scripts/generate_task_from_dataset.py). This is the task `launch_eval.py` evaluates, and the pipeline generates it for you from `--dataset-path` before starting the workers. The slug is the dataset directory name with `-` replaced by `_`, unless you pass `--task-name`. From the LeRobot dataset it reads: - the block names, from the `observation.state.object_pose.` columns; - every distinct `pddl_goal`. A dataset with several goals gives one task that selects each environment's goal per episode. - the recorded block sizes (`cube_scale`); - the recorded `scene_id` and `robot_id`. A dataset that records no scene or robot is rejected. It only generates code: each episode's starting state comes from the [episode queue](evals/launch_eval.md#the-episode-queue), so no reset states are sampled. To run it by hand, for example to debug code generation: ```bash python scripts/generate_task_from_dataset.py --dataset-path data/datasets/my_dataset \ --domain-path src/toyblocks/tamp/blocks --output-dir src/toyblocks/tasks/autogen --headless ``` **From a PDDL problem:** [`scripts/generate_task_from_pddl.py`](https://github.com/allenai/ToyBlocks/blob/main/scripts/generate_task_from_pddl.py), the [environment generation](datagen/env_generation.md) stage of data generation. Besides the package it samples starting scenes from the problem's `:init` into `reset_states.pkl`, which the task starts from when no queue is attached. Use it to evaluate on a problem that has no dataset, by running `evaluate.py --task -v0` directly. `launch_eval.py` does not use it (this will be fixed by December 2026). **Hand-written tasks** in `src/toyblocks/tasks/` (for example `ToyBlocks-Lift-v0`) define their own `evaluate()`; [Metrics](evals/metrics.md) shows how to add task-specific metrics. **Regenerate packages after updating the code.** Each package records the codegen version that built it (`codegen_version` in `profiles.json`), and evaluation refuses a package built by a different version. Otherwise a stale package's goal checks and success gate would silently change success rates. ## How to run the eval pipeline There are two entry points: - `scripts/launch_eval.py` is the recommended one. It runs the whole queue-driven evaluation pipeline on your machine. - `scripts/evaluate.py` is the eval worker the evaluation pipeline runs. You can also run this script directly, for example to debug one process against a policy server you started by hand. The evaluation pipeline itself lives in [`scripts/eval/run_eval_pipeline.sh`](https://github.com/allenai/ToyBlocks/blob/main/scripts/eval/run_eval_pipeline.sh) and `launch_eval.py --dry-run` prints the exact invocation of it. Here's what processes are launched when running `scripts/launch_eval.py`: ```text launch_eval.py --policy {molmo2_maniflow,maniflow} (or --replay) └─ run_eval_pipeline.sh ├─ GPU 0 policy server: serve_maniflow_policy --hl {model,oracle} (none under --replay) ├─ codegen builds the gym task from the dataset (generate_task_from_dataset.py) ├─ CPU serve_episode_queue.py hands out one recorded setup per episode ├─ GPU 1..N evaluate.py --queue_url ... (one worker per GPU, many envs each) ├─ CPU wandb_consolidate.py merges all workers into one W&B run (--wandb) └─ CPU merge_eval_runs.py + skill_success_from_eval_run.py merge the workers' results into one run directory under runs/ ``` On a machine with fewer GPUs than that layout, the workers share the GPUs round-robin, so a one-GPU workstation runs the server and one worker on GPU 0. Detailed references: - [`launch_eval.py` Reference](evals/launch_eval.md): every `launch_eval.py` flag, the default and smoke-test profiles, the per-policy defaults, the episode queue, dataset requirements and the output layout. - [`evaluate.py` Reference](evals/evaluate.md): `evaluate.py` flag reference. - [Metrics](evals/metrics.md): built-in metrics, output formats, and how to add metrics. ## Quick start with `launch_eval.py` The eval pipeline initializes episodes from the recorded episodes of the LeRobot dataset given by `--dataset-path` (required): the queue reads one setup per episode from it, and the gym task is generated from it. `--smoke-test` switches to a small validation profile (10 episodes, 1 eval GPU, 2 environments, 3-second episodes); drop it for a full run over the whole dataset. Policy evals need the `maniflow-tb` and `molmo2` submodules (see [Installation and setup](installation_and_setup_instructions.md)). ```bash # Learned Molmo2 high-level policy + ManiFlow low-level policy (the default policy) python scripts/launch_eval.py --smoke-test \ --model-path data/checkpoints/my_maniflow_run/checkpoints/steps_400000_model.safetensors \ --hl-model-path data/checkpoints/my_molmo2_run \ --dataset-path data/datasets/my_dataset # ManiFlow low-level policy, with the high-level oracle (the default for maniflow) python scripts/launch_eval.py --smoke-test --policy maniflow \ --model-path data/checkpoints/my_maniflow_run/checkpoints/steps_400000_model.safetensors \ --dataset-path data/datasets/my_dataset # Replay the dataset's recorded actions (no policy, no checkpoint) python scripts/launch_eval.py --replay \ --dataset-path data/datasets/my_dataset --episode-indices "0" ``` Useful additions: `--episode-indices "0,3,10-19"` to pick episodes, `--num-eval-gpus N` to spread a `maniflow` eval over N eval GPUs, `--wandb` to log the run to W&B, and `--dry-run` to print the pipeline command instead of running it. Full episode rollouts and videos are recorded as a LeRobot dataset by default (`--no-save-lerobot` to skip it). See the [`launch_eval.py` Reference](evals/launch_eval.md) for every flag and its defaults. ## High-level conditioning modes ManiFlow low-level policies are conditioned on a high-level command (a task string and a target point in the front camera image). Where that command comes from is chosen at launch: - **High-level oracle** (`--policy maniflow`, `--use-oracle` on by default): a symbolic planner running next to the simulation produces the commands from the episode's PDDL problem. This measures the low-level policy on its own. - **Learned high-level policy** (`--policy molmo2_maniflow`): a Molmo2 model served alongside ManiFlow produces the commands from camera images. By default it is queried at every action-chunk boundary; with a play-mode low-level checkpoint, `--async-hl` queries it only at the start of an episode and whenever the low-level policy ends a skill. - **Low-level oracle** (`--policy molmo2_maniflow --ll-oracle`): the Molmo2 commands are executed by the motion-planned skills used in data generation (cuRobo v2) instead of by ManiFlow. This measures the high-level policy on its own. The recorded expert actions can also be replayed open-loop with `--replay`, which needs no policy at all; see [Dataset Replay](evals/replay.md). ## Running `evaluate.py` directly `evaluate.py` is the eval worker the pipeline runs, and you can also run it by hand: against a policy server you started yourself (the easiest way to debug either side with breakpoints), with random or zero actions, or to replay a dataset. [`evaluate.py` Reference](evals/evaluate.md) covers its action sources, a two-terminal server + worker recipe, the observation transforms, every flag and its outputs. ## Episode scoring and outputs An episode ends when the goal is reached, at the timeout, or when a play-mode low-level policy stops (`--episode_end`); `--expert-timeout-scale` ties the timeout to the length of each episode's expert demonstration. A block that was not supposed to move (by more than 2 cm or 15 degrees) counts as a collision and fails the episode (`--collision`). Each eval worker writes `eval_results.json` (per-episode values and aggregates) and `eval_episodes.json` (one record per episode) to its output directory under `runs/`, plus a LeRobot dataset of the rollouts (`--save-lerobot`, on by default in pipeline runs) and per-step CSVs with `--save_step_metrics`. At the end the pipeline merges the workers into one run directory with per-skill success rates (`skill_success.json`), and with `--wandb` into one W&B run; see [`launch_eval.py` outputs](evals/launch_eval.md#outputs). See [Metrics](evals/metrics.md) for every metric, the output formats and how to add a task-specific metric, and the [`evaluate.py` Reference](evals/evaluate.md) for the scoring flags and for capturing failed episodes for TAMPER. ### Easily measuring partial progress from the PDDL goal Success alone says nothing about how far a failed episode got. Because every episode comes with a PDDL goal, the eval system measures partial progress for every task automatically, with no task-specific code, and measures it the same way for all tasks. **Where the goal comes from.** Each episode carries its PDDL goal (the queue setup's `pddl_goal`), for example: ```text (and (on_table block_a workspace_table) (on block_b block_a) (on block_c block_b)) ``` When the gym task is generated, [`codegen.py`](https://github.com/allenai/ToyBlocks/blob/main/src/toyblocks/tamp/blocks/template/codegen.py) compiles each conjunct of each goal into a batched check on the live simulation state (`_goal_conj_`, which calls `batched_goal_conjunct` in [`pddl_goal_tensor_conversions.py`](https://github.com/allenai/ToyBlocks/blob/main/src/toyblocks/tamp/blocks/pddl_goal_tensor_conversions.py)). At reset, `set_episode_goal` selects each environment's own goal, and fails rather than grade an episode against a goal the task was not compiled for. The supported predicates are `on`, `on_table`, `on_left`, `on_right`, `on_front`, `on_back`, `next_to`, `ori_of`, `clear` and `pick_obstructs`; `ori_of` accepts a yaw error up to `--goal-ori-yaw-tol-deg` (40 degrees by default). **Every step**, the task reports `goal_conj_`, whether conjunct `c` holds (for example `(on block_b block_a)`), and `goal_success`: all conjuncts hold and the settle gate passes (the goal blocks are at rest and the gripper is open). The settle gate only affects `goal_success`, not the conjunct counts. **Each episode** counts the conjuncts in the goal (GT), those true at the start (IT), those true at the end (CT), and those true at some point, and reports `partial_progress = (CT - IT) / (GT - IT)`, which lies in `[-1, 1]`. In the goal above, `block_a` starts on the table, so IT = 1 and GT = 3. Finishing the stack gives 1.0, placing only `block_b` gives 0.5, and ending with none of the three conjuncts holding gives -0.5. When the goal already holds at the start (GT = IT), `partial_progress` is NaN and left out of the mean. **Where it appears:** - per episode in `eval_episodes.json`: `partial_progress`, the `num_goal_conjuncts*` counts, and a `goal_progress` entry per conjunct with its PDDL text, its `initial`, `first_true_step` and `final` values and its change-point `trace`; - as the `partial_progress` aggregate in `eval_results.json`; - as `mean_partial_progress` in the W&B aggregates table, with a goal-progress plot per episode in the episodes table's `goal_progress` column. See [Metrics](evals/metrics.md) for every goal-progress key. ## Setting up a new policy server Any model can be evaluated with this pipeline by wrapping it in a small policy-server interface. See [Setting up a new policy server](imitation_learning/new_policy_server.md). ```{toctree} :hidden: evals/replay evals/metrics evals/launch_eval evals/evaluate ```