You write down what should be true at the end of a task, for example "the red block is on the green block". The pipeline samples starting scenes, plans each episode with a TAMP (task and motion planning) planner, runs the plan on many simulated environments in parallel, checks every step, renders the result and writes a LeRobot dataset.
A ToyBlocks task is a PDDL goal: which block should be on which, and which face should point where. The simulation environment and its starting scenes are generated from that goal, so adding a task means writing a goal, not simulation code.
Task generators produce whole families of these goals: flipping a block to a new face, turning it, stacking one block on another, three-block towers in scenes of 4–10 letter, digit and color blocks (the family behind the released training sets), and scenes for play data and letter search. Block size, table height and friction can vary from one environment to the next.
Docs: data generation overview · task generators · running the pipeline
Demonstrations come from a task and motion planner. The task level works over the blocks PDDL domain and decides the order of skills: what has to be cleared, moved aside, flipped or rotated before the goal stack can be built. The motion level checks each step against the actual scene. Is there room to flip this block to the right, or is another block in its landing spot? Is the side the gripper needs to reach free? When a step is blocked, the plan gets extra steps to clear the way first, and the arm motions themselves come from cuRobo.
vecgen, the engine that runs data generation, is built on updated versions of our TAMP code, reworked so that planning and execution run batched across hundreds of environments.
Execution runs on vecgen, our vectorized engine. One GPU runs hundreds of environments at once, and the skills (pick, place, flip and rotate) run batched across all of them. On a single GPU this gets up to 24× real time.
After each skill, its effects are checked against the simulator. If the check fails, the skill is retried; if it keeps failing, the planner re-plans from wherever the scene is now. An episode only counts if the goal actually holds at the end, and failed episodes are kept out of the training data.
Docs: correct by construction · what an episode contains · design goals · adding a skill to vecgen
Physics and pixels are separate stages. Execution records what happened; rendering replays each episode and shoots it again with randomized lighting, HDRI backgrounds, table materials, exposure, and camera poses and intrinsics. The robot's motion doesn't change, so one simulated episode can be rendered several different ways.
Docs: domain randomization · render passes · HDRI backgrounds
Play data has no fixed task. A goal policy looks at the current scene and picks the next goal, the TAMP planner solves it, and the robot carries it out. The scene isn't reset between goals, so each new goal starts from whatever the robot left behind.
You can balance the skill mix (by default about a third pick-and-place, a third flips and a third rotations), cut some grasps off partway so the robot learns to recover, and script search strategies such as flipping blocks until a hidden letter turns up. Play episodes then go through the same rendering and labelling as everything else.
Docs: play and search scenes
The planner knows what each part of a trajectory is for, so the labels come for free. Nobody annotates anything by hand.
Every frame records which plan step the robot is carrying out (for example, moving block A onto block 4) and which motion phase it is in. The planner's symbolic state is recorded alongside. So each trajectory splits cleanly into one segment per sub-goal.
Annotation projects the recorded block poses through the recorded front camera. That gives the pixel position of the block being acted on and of where it should go, on every frame, along with the action as text ("pick", "rotate cw" and so on).
The text and the point are exactly what Molmo 2 learns to predict and what ManiFlow is conditioned on. The segment boundaries give ManiFlow its retract and "done" labels, which is how it learns to say when a skill is finished (play mode).
Docs: annotation · recorded fields · pause labelling in training
The merge stage collects every problem into a single LeRobot v3 dataset. Annotation then adds the per-frame action text and target pixel positions described above, and the dataset is ready for training.
Docs: merge · annotation · recorded fields · Released data: allenai/toyblocks-data
The local-run guide walks through a complete run: four stacking problems, 16 episodes each, rendered and merged into one dataset, with the command to start it.
Docs: a complete run · GPUs and concurrency