Poke &Wiggle

Reality Check Leaderboard

PAW-GEN-10

View Episodes

Share of episodes rated Ok or Excellent.

Mid-training DataM0 has no mid-training data and post-trains the model directly on the environment demonstrations. M100 first mid-trains it on 100 hours of embodiment data (coming soon).Data ScalingScores after training on ~10, ~100 or ~300 demonstrations per environment (D10, D100, D300). The sets are nested.Spatial GeneralizationNominal uses the training placements for evaluation. Interpolation uses unseen placements between them. Extrapolation uses unseen placements outside their convex hull.Per EnvironmentThe model's score on every environment of the benchmark, e01 onwards. Hover an environment's column for its name.
#Model# Evals
1Rho*32%360032%Coming
soon		7%41%49%35%29%Coming soon28%53%54%21%3%6%44%61%46%9%
2MolmoAct 2*28%360028%Coming
soon		4%37%44%31%25%Coming soon10%44%52%15%21%4%42%52%18%24%
3Pi 0.5*21%360021%Coming
soon		5%26%33%23%19%Coming soon4%39%66%2%9%3%13%59%13%2%
4DiT-Flow19%360019%Coming
soon		2%24%29%21%16%Coming soon2%17%48%2%0%1%32%63%16%4%
5GR00T N1.7*12%360012%Coming
soon		4%14%19%13%12%Coming soon3%14%25%1%1%1%14%50%13%0%

* All VLAs were started from their official checkpoints, but fine-tuned by us on the target tasks and data regimes.

Export

  1. Added Rho, Microsoft Research's 5B VLA, fine-tuned from its official checkpoint and evaluated on every track. Corrected annotations for 27 evaluation episodes of the other models. No success rate changed. Six of those corrections only moved step boundaries or split a step, so they are not listed.

    Corrected episodes:

    • Step 3 re-rated excellent → ok: DiT-Flow, e09, D10 - Interpolation (view)
    • Step 1 re-rated failed → excellent: DiT-Flow, e09, D10 - Nominal (view)
    • Step 2 re-rated ok → failed: DiT-Flow, e09, D10 - Nominal (view)
    • Step 3 re-rated excellent → ok: GR00T N1.7, e04, D10 - Nominal (view)
    • Step 2 re-rated excellent → ok: Pi 0.5, e04, D100 - Nominal (view)
    • Step 2 re-rated excellent → ok: DiT-Flow, e03, D300 - Interpolation (view)
    • Step 1 re-rated failed → excellent: GR00T N1.7, e09, D300 - Interpolation (view)
    • Step 1 re-rated failed → excellent: GR00T N1.7, e09, D300 - Interpolation (view)
    • Step 1 re-rated failed → excellent: GR00T N1.7, e09, D300 - Interpolation (view)
    • Step 1 re-rated failed → excellent, step 4 re-rated excellent → failed: GR00T N1.7, e09, D300 - Interpolation (view)
    • Step 5 re-rated ok → failed: MolmoAct 2, e09, D300 - Interpolation (view)
    • Step 5 re-rated ok → failed: MolmoAct 2, e09, D300 - Interpolation (view)
    • Step 8 re-rated excellent → ok: MolmoAct 2, e06, D300 - Interpolation (view)
    • Episode re-rated excellent → ok: Pi 0.5, e09, D300 - Interpolation (view)
    • Step 5 re-rated excellent → failed: Pi 0.5, e09, D300 - Interpolation (view)
    • Episode re-rated excellent → ok: Pi 0.5, e09, D300 - Interpolation (view)
    • Step 4 re-rated excellent → failed: Pi 0.5, e09, D300 - Interpolation (view)
    • Episode re-rated excellent → ok: Pi 0.5, e09, D300 - Interpolation (view)
    • Step 1 re-rated excellent → ok, step 5 re-rated excellent → failed: Pi 0.5, e09, D300 - Interpolation (view)
    • Step 7 re-rated excellent → ok: MolmoAct 2, e04, D300 - Nominal (view)
    • Step 4 re-rated ok → failed: MolmoAct 2, e09, D300 - Nominal (view)
    • Step 7 re-rated failed → ok, step 9 re-rated ok → excellent: MolmoAct 2, e01, D300 - Nominal (view)
    • Step 8 relabelled "Pick black screw from table" → "Place screw in respective bin": MolmoAct 2, e01, D300 - Nominal (view)
    • Step 4 relabelled "Step 1" → "Step 2": Pi 0.5, e07, D300 - Nominal (view)
    • Step 8 re-rated ok → failed: Pi 0.5, e05, D300 - Nominal (view)

  2. Corrected annotations for nine evaluation episodes. No success rate or overall ranking changed. Overall execution quality changed by less than 1% for GR00T N1.7 and Pi 0.5, and progress by less than 0.1% for every model.

    Corrected episodes:

    • Step 1 re-rated failed → excellent: DiT-Flow, e08, D10 - Interpolation
    • Step 1 re-rated failed → ok: GR00T N1.7, e08, D10 - Interpolation
    • Step 1 re-rated failed → ok: MolmoAct 2, e08, D100 - Interpolation
    • Step 4 re-rated failed → ok, step 9 re-rated failed → excellent: Pi 0.5, e09, D100 - Interpolation
    • Step 1 re-rated ok → excellent: GR00T N1.7, e06, D100 - Nominal
    • Step 3 re-rated excellent → failed: Pi 0.5, e06, D100 - Nominal
    • Step 5 relabelled "Pick black screw from table" → "Place screw in respective bin": Pi 0.5, e01, D100 - Nominal
    • Episode re-rated excellent → ok: GR00T N1.7, e02, D300 - Interpolation
    • Episode re-rated ok → excellent: Pi 0.5, e02, D300 - Nominal

  3. First release of PAW-GEN-10.

Data Scaling

Model scaling by training episodes per environment: SuccessRho: D10 7% (95% CI 5%–10%), D100 41% (95% CI 37%–45%), D300 49% (95% CI 45%–53%). MolmoAct 2: D10 4% (95% CI 3%–6%), D100 37% (95% CI 33%–40%), D300 44% (95% CI 40%–48%). Pi 0.5: D10 5% (95% CI 3%–6%), D100 26% (95% CI 22%–29%), D300 33% (95% CI 29%–37%). DiT-Flow: D10 2% (95% CI 1%–4%), D100 24% (95% CI 21%–28%), D300 29% (95% CI 26%–33%). GR00T N1.7: D10 4% (95% CI 2%–5%), D100 14% (95% CI 11%–17%), D300 19% (95% CI 16%–22%).Success rate (%)03060Rho: 7% (95% CI 5%–10%) on D10Rho: 41% (95% CI 37%–45%) on D100Rho: 49% (95% CI 45%–53%) on D300MolmoAct 2: 4% (95% CI 3%–6%) on D10MolmoAct 2: 37% (95% CI 33%–40%) on D100MolmoAct 2: 44% (95% CI 40%–48%) on D300Pi 0.5: 5% (95% CI 3%–6%) on D10Pi 0.5: 26% (95% CI 22%–29%) on D100Pi 0.5: 33% (95% CI 29%–37%) on D300DiT-Flow: 2% (95% CI 1%–4%) on D10DiT-Flow: 24% (95% CI 21%–28%) on D100DiT-Flow: 29% (95% CI 26%–33%) on D300GR00T N1.7: 4% (95% CI 2%–5%) on D10GR00T N1.7: 14% (95% CI 11%–17%) on D100GR00T N1.7: 19% (95% CI 16%–22%) on D300D10D100D300

D10, D100, and D300 indicate how many demonstration episodes per environment were used to train the model. Whiskers are 95% confidence intervals over the evaluation episodes.

Spatial Generalization

Spatial generalization by model: SuccessRho: Nominal 35% (95% CI 32%–39%), Interpolation 29% (95% CI 27%–33%), Extrapolation coming soon. MolmoAct 2: Nominal 31% (95% CI 28%–34%), Interpolation 25% (95% CI 23%–28%), Extrapolation coming soon. Pi 0.5: Nominal 23% (95% CI 20%–26%), Interpolation 19% (95% CI 17%–22%), Extrapolation coming soon. DiT-Flow: Nominal 21% (95% CI 19%–24%), Interpolation 16% (95% CI 13%–18%), Extrapolation coming soon. GR00T N1.7: Nominal 13% (95% CI 11%–15%), Interpolation 12% (95% CI 10%–14%), Extrapolation coming soon.Success rate (%)03060Coming soonRho: 35% (95% CI 32%–39%) on NominalRho: 29% (95% CI 27%–33%) on InterpolationMolmoAct 2: 31% (95% CI 28%–34%) on NominalMolmoAct 2: 25% (95% CI 23%–28%) on InterpolationPi 0.5: 23% (95% CI 20%–26%) on NominalPi 0.5: 19% (95% CI 17%–22%) on InterpolationDiT-Flow: 21% (95% CI 19%–24%) on NominalDiT-Flow: 16% (95% CI 13%–18%) on InterpolationGR00T N1.7: 13% (95% CI 11%–15%) on NominalGR00T N1.7: 12% (95% CI 10%–14%) on InterpolationNominalInterpolationExtrapolation

Nominal uses training placements; interpolation uses unseen placements between them; extrapolation uses unseen placements outside their convex hull. Whiskers are 95% confidence intervals over the evaluation episodes.

Per Environment (Success)

Model performance by environmentRho overall: Sort screws into bins 28%, Datum-corner alignment 53%, Pour screws into the funnel 54%, Connect DC jack 21%, Route cable through closed hoops 3%, Put tools in standing toolbox 6%, Place loaded boxes in container (bimanual) 44%, Place spray bottles upright, nozzle forward (bimanual handover) 61%, Open the toolbox with a screwdriver 46%, Clamp electrical component 9%. MolmoAct 2 overall: Sort screws into bins 10%, Datum-corner alignment 44%, Pour screws into the funnel 52%, Connect DC jack 15%, Route cable through closed hoops 21%, Put tools in standing toolbox 4%, Place loaded boxes in container (bimanual) 42%, Place spray bottles upright, nozzle forward (bimanual handover) 52%, Open the toolbox with a screwdriver 18%, Clamp electrical component 24%. Pi 0.5 overall: Sort screws into bins 4%, Datum-corner alignment 39%, Pour screws into the funnel 66%, Connect DC jack 2%, Route cable through closed hoops 9%, Put tools in standing toolbox 3%, Place loaded boxes in container (bimanual) 13%, Place spray bottles upright, nozzle forward (bimanual handover) 59%, Open the toolbox with a screwdriver 13%, Clamp electrical component 2%. DiT-Flow overall: Sort screws into bins 2%, Datum-corner alignment 17%, Pour screws into the funnel 48%, Connect DC jack 2%, Route cable through closed hoops 0%, Put tools in standing toolbox 1%, Place loaded boxes in container (bimanual) 32%, Place spray bottles upright, nozzle forward (bimanual handover) 63%, Open the toolbox with a screwdriver 16%, Clamp electrical component 4%. GR00T N1.7 overall: Sort screws into bins 3%, Datum-corner alignment 14%, Pour screws into the funnel 25%, Connect DC jack 1%, Route cable through closed hoops 1%, Put tools in standing toolbox 1%, Place loaded boxes in container (bimanual) 14%, Place spray bottles upright, nozzle forward (bimanual handover) 50%, Open the toolbox with a screwdriver 13%, Clamp electrical component 0%.Sort screws into binse01Datum-corner alignmente02Pour screws into the funnele03Connect DC jacke04Route cable through closed hoopse05Put tools in standing toolboxe06Place loaded boxes in container (bimanual)e07Place spray bottles upright, nozzle forward (bimanual handover)e08Open the toolbox with a screwdrivere09Clamp electrical componente10Basic skillsHigh precisionBimanualTool use

e01–e10 represent the environments in PAW-GEN-10, each value in the selected metric averaged across every data tier and placement condition.