Kids learn a lot about space and planning from a pile of toy blocks: rebuilding a tower after it falls, turning a block over to find the letter they want. We wanted a robot learning setup with the same flavor. The domain is small enough that we know everything about every scene (each block, its letters, its pose, the goal), but the robot still has to push, flip, rotate and stack precisely, and some tasks take a dozen skills in a row.
Because nothing is hidden from us, a failed rollout is something we can actually debug. Was it the training data, the model, or the way we evaluated it? ToyBlocks is built so you can change one of those and hold the other two fixed.
ToyBlocks has four main parts. Each has its own page here, and every page links to the matching section of the documentation.
PDDL goals in, LeRobot datasets out. A TAMP planner plans every episode, motion-planned skills run hundreds of environments per GPU, check every step, and the episodes are rendered with domain randomization and annotated. Includes play data.
Train Molmo 2 and ManiFlow on generated data. Then replace the high level or the low level with an oracle to see which one is failing, replay datasets open loop, and send failure states back to data generation with TAMPER.
Molmo 2 + ManiFlow, trained only on ToyBlocks data, spelled three-letter words on a real Franka FR3 in 15 of 30 attempts.
Each skill is exposed as an MCP tool, so a frontier model can drive the robot from camera images. We use it to study how these models search, plan and recover.
Results from our first experiments are on the Experiments page.