Sim-to-real Toolkit

Trained in simulation, tested on a real arm

We trained Molmo 2 (high level) and ManiFlow (low level) only on ToyBlocks data, with no real-world demonstrations, and ran them on a Franka FR3 with a Robotiq gripper.

The same task on the real robot (left) and in simulation (right), with the same high-level model. We re-timed the simulation clip so grasps and releases line up. Pink rings mark the high-level model's target point for the current skill.

Results

ResultsSingle skills and full spelling tasks succeeded about as often on the real robot as in simulation. See the numbers and the chart.Real-robot results →

How the policy runs

Molmo 2 looks at the front camera and decides what to do next: which skill, and a point in the image that marks the block. ManiFlow takes that command and controls the arm. Both were trained on renders with heavy domain randomization (lighting, backgrounds, table materials, camera poses and intrinsics), which is a big part of why the real lab doesn't look out of place to them.

Docs: rendering · training ManiFlow

The real robot stacking green on N on X.

What's in the repo

The simulation side is in the repo today: generated environments start from a calibrated copy of our real workcell.

Coming soon: a toolkit for sim-to-real transfer, including the code that runs policies on the real robot and a guide to performing system identification.

The real workcell from the front camera The sim2real scene from the front camera
The real workcell (left) and the sim2real scene in simulation (right), both seen from the front camera.
Randomized render of the tabletopRandomized render of the tabletop Randomized render of the tabletopRandomized render of the tabletop Randomized render of the tabletopRandomized render of the tabletop Randomized render of the tabletopRandomized render of the tabletop

Frames from two randomized render passes of the training data.

← Training and evaluationAgentic Framework →