A first set of experiments run with ToyBlocks: how well learned policies generalize, how a high-level model copes with new blocks, how frontier models plan when they act through skill tools, and how well world models can stand in for a simulator.
Vision-language-action models (VLAs) use pretraining on large robot datasets. The purpose is that a VLA adapts to a new workspace with a small number of demonstrations. With ToyBlocks, we evaluate the spatial generalization of these action models in a simple setting. Each change in task space is meaningful and easy to reproduce. We change only the block positions or the stack height, and we keep all other conditions the same.
We use four tasks: Stack, Unstack, Flip and Rotate. Flip has four directions: forward, backward, left and right. Rotate has two directions: clockwise and counterclockwise. The instruction gives the direction.
We fine-tune three architectures: ManiFlow, π0.5 and MolmoAct 2. Each model gets 1,000 demonstrations for each task. The ToyBlocks TAMP planner makes these demonstrations at random positions in the training region. When the test region is the same as the training region, the mean success rate is 72% to 90%.
Then we move the test region. We also use three more splits. We train on the inner region and test on the outer region. We train on the outer region and test on the inner region. We train on stacks of 2 and 3 blocks and test on stacks of 4 and 5 blocks.
Surprisingly, these small changes cause large decreases in VLA performance. The models perform well in the training region. However, in the test region, their success rate is low, or they fail. This occurs although the test conditions are very similar to the training conditions. It also occurs although some of these models have pretraining on large datasets.
We tested five changes to the training. We added each change by itself.
Keep some of the initial data. Co-training gives the largest single gain on the right half of the table (+45 points). Mid-training uses the same type of data only one time, before fine-tuning, and gives +14 points. Fine-tuning on one task also causes the model to forget the other tasks. Co-training with the data from these tasks prevents most of this loss. Thus, open robot data is important. A fine-tuned VLA gets its full generalization only when it continues to train on some of its initial data.
Predict delta actions. Delta actions give better results than absolute joint targets for π0.5 and for MolmoAct 2. On new stack heights, delta actions give +26 points. This gain does not come from a match with the pretraining action format. MolmoAct 2 used absolute joint actions in pretraining, and π0.5 used velocity actions.
Add video prediction when the scenes look new. By itself, WAM gives +3 points on the right half of the table and +23 points on new stack heights. WAM gives the best results together with co-training. We trained MolmoAct 2 on stacks of 2 and 3 blocks and tested it on stacks of 4 and 5 blocks. With WAM and co-training, its success rate increases from 29% to 80%. We think that the video head removes a visual limit, because the training images do not show stacks of 4 or 5 blocks.
Fine-tuned VLAs perform well in the training region, but small shifts in task space cause large decreases in success. We explored and evaluated many techniques in detail, and these techniques improve generalization in our simple settings. Nevertheless, VLA generalization is far from solved. ToyBlocks gives researchers a flexible and reproducible way to study new generalization techniques and more difficult settings.
Details. We use 1,000 training demonstrations for each task. Co-training and mid-training use 20,000 more demonstrations (5,000 for each task). We fine-tune for 30,000 steps. We evaluate each 10,000 steps and report the best checkpoint. Each run has 500 evaluation episodes. We normalize the joint actions to [−1, 1] with statistics from the full workspace, so that the robot can reach all regions.
For the setup of the evaluation episodes, refer to Evaluation at Scale.
Molmo 2 runs as the high level, with either the motion-planned skills (the low-level oracle) or ManiFlow as the low level. Seen uses the training blocks in new arrangements, and unseen uses held-out blocks. The three difficulty levels differ in how the letters start out, as the starting scenes below the chart show.
The modes behind this: Evaluation Modes
We trained Molmo 2 and ManiFlow only on ToyBlocks data, with no real-world demonstrations, and ran them on a Franka FR3 with a Robotiq gripper. We tested two things: single skills with ManiFlow (pick, place, flip and rotate), and full three-letter spelling tasks with Molmo 2 + ManiFlow. Real-world runs used thirty scene configurations with 4–10 blocks that appear in the training data.
For both, the real-robot results are consistent with simulation. Both also need more trials before they can support any substantial claim. The spelling runs used the easiest setting, with every letter visible and upright.
How the transfer works: Sim-to-real Toolkit
With the Agentic Framework, a frontier model (Claude Opus 5.5 in these runs) sees the camera images and calls ToyBlocks skills as MCP tools, pointing at the block it wants. It gets no success signal, so it has to decide for itself when the task is done, or when it can't be done.
The agent has to stack a three-letter word, finding the letters among up to eight blocks. In half of the scenes a letter is missing, and then the right answer is to say the task is impossible. On patterned blocks, each block carries one letter in several forms (upper and lower case, plus pictures that start with it), so the agent can work out what a hidden face must show. On unpatterned blocks the faces are unrelated. With patterns the agent finishes having turned over fewer faces, and on infeasible scenes it declares the task impossible much sooner.
When a skill behaves unexpectedly, the agent forms a hypothesis, runs a small test and revises its understanding, then writes its own descriptions of the skills. Those learned descriptions work better than the ones an expert wrote.
Docs: agentic experiments
Action-conditioned world models predict how a scene changes in response to a robot's actions, and using them in place of model-based simulators is an active area of research. Because ToyBlocks knows the true state of every block, we can test how accurately they predict physical interactions.
Setup. We trained or fit three types of models on four tasks (pick-and-place, flip, rotate, and a mixed task): a video prediction model (Wan 2.2), an agentic model that created its own physics simulator (Astra), and a latent world model (LeWM). Each task has 900 trajectories. Half succeed and half fail because the trajectory was changed slightly, so always predicting success is right only half the time. Each model gets an initial image and the actions. On full trajectories this is the first image of the episode; on short interaction segments (a single pick, place, flip, or rotate) it is the real image immediately before that interaction. Small probes trained on simulator labels read block pose, gripper pose and success from each prediction.
Video models predict interactions best. Wan 2.2 tracks the blocks to about a centimeter and predicts 99% of outcomes correctly.
Tracking the robot is not the same as simulating contact. Astra follows the gripper closely (1.1 cm) but predicts only 67% of outcomes.
Latent models capture short interactions but drift over long ones. LeWM predicts 65% of outcomes on short interaction segments, but its block error grows to 24 cm over full trajectories.
Better world models produce better training data. Adding 100 Wan 2.2 rollouts from a part of the table with no real demonstrations raises ManiFlow's success there from 19% to 78%.