Training a two-level policy, then finding out which level failed

Training

During data generation the planner reads every block's pose straight from the simulator. A trained policy doesn't get that. It has to work out where the blocks are from camera images, the same way it would on a real robot, so both models map raw camera images and the robot's joint state to actions.

We split the policy in two. Molmo 2, a vision-language model, is the high level: it chooses the skill to run and points at the block to run it on. ManiFlow is the low level and carries out that one instruction. ManiFlow is trained to pause once it has finished a skill and ask Molmo 2 for the next one.

Prompt format

Molmo 2 gets the goal as a tower listed from top to bottom, followed by fixed instructions. It answers with the progress so far, the next step, and the action with its target point:

The prompt can start with the previous step, written the same way as an answer. Point coordinates run from 0 to 1000 across the image width and height. Each target sits 5 px (in a 640×360 frame) below the block's center, on the face that points at the camera. ManiFlow then receives the action word ("pick") and a heatmap drawn around that point.

Data: toyblocks-data

Pausing and play mode

ManiFlow doesn't just move the arm. It also learns to say "I'm done." ManiFlow's ninth action dimension is the play mode: 1 keeps going, 0 indicates to the model that the attempt is complete and the robot is retracting, and −1 says the skill is done. ManiFlow learns it from the training labels: when a pick, place, flip or rotate finishes the attempt and retract stages partway through a chunk, it emits a −1 and the rest of that chunk repeats the final pose and is marked as done. At eval time, 1 and 0 are fed back as part of the next state. A −1 ends the skill, and the arm holds where it stopped until the next query. The skill boundaries behind these labels come straight from data generation, where every frame is labelled with its sub-goal.

Molmo 2 can be queried on two schedules:

• Synchronous (the default): Molmo 2 is queried at every action-chunk boundary, together with ManiFlow, so it can change its command in the middle of a skill.
• Asynchronous: Molmo 2 is queried at the start of the episode and again only after ManiFlow reports −1. That needs far fewer high-level calls, and the command stays fixed for the whole skill. This mode needs a play-mode checkpoint.

Molmo 2 queriedno queryManiFlow running (1)holding after −1−1 skill done

Docs: high-level conditioning modes · HL options · pause labelling in training · play-mode metrics

Evaluation Modes

A mode is a choice of high level and low level. All of them run from the same launcher.

There is no mode with both oracles: that is just data generation. The low-level oracle always queries Molmo 2 when a skill ends, so it has no every-chunk variant. ManiFlow can also run with no high level at all, given only the episode's goal sentence.

Docs: high-level conditioning modes · per-policy defaults · dataset replay

ResultsHow Molmo 2 does with the motion-planned skills or ManiFlow as its low level, on easy, medium and hard tasks with seen and held-out blocks.Molmo 2 results →

Evaluation at Scale

Every eval episode starts from an episode in a dataset. An episode queue reads the robot pose, block poses and sizes, table height and goal from the recording, and the policy has to reach the goal from there. Since the scene comes straight from the dataset, a difference in success rate between two models reflects the models and not a difference in setup.

The queue hands episodes out to GPU-parallel environments, so the same command works on a single workstation GPU or across a multi-GPU machine.

Docs: design goals · the episode queue

TAMPER: training on your failures

TAMPER (TAMP Expert Relabeling) is a DAgger-style loop with a planning expert. An eval run finds the states where the policy goes wrong; the expert then plans new demonstrations starting from those states. The next round of training data then covers situations the policy actually ends up in, not only the ones data generation happens to sample.

1. Evaluate with capture on. Turn on failure capture in an eval run. The final scene of every failed or timed-out episode is saved as a new environment package.
2. Look at what was captured. A summary file lists each captured state with its episode, outcome and goal.
3. Relabel. vecgen plans from the captured scene toward the recorded goal and executes it with the usual verified skills.
4. Retrain and evaluate again. Annotate and merge the new episodes, add them to the training mix, and repeat.

In a small test on one RTX 5090, a random-action eval on a 16-episode stacking set captured 16 states, and vecgen recovered 12–13 of them in about 72 seconds.

Docs: TAMPER guide

Outputs

Each run scores every episode against its goal and writes per-step metrics and run summaries. Full rollouts, videos included, are recorded as a LeRobot dataset by default, in the same format as the training data. Any policy that can be served over a WebSocket can be evaluated by implementing a small policy-server interface.

Docs: eval metrics · scoring and outputs · adding a policy server

Running it

The quick start covers a small smoke-test run (10 episodes, one eval GPU, two environments) and a full run over a dataset, for each mode above.

Peak GPU memory we measured per process:

ManiFlow-only evals fit on a 16 GB card. Molmo 2 evals need about 24 GB, or two GPUs: the launcher puts the policy server on GPU 0 and the eval worker on GPU 1.

Docs: quick start · evaluation overview