ModelMolmoAct 2
Allen Institute for AI
#1
Rank · PAW-GEN-10
Model information
PAW-GEN-10
Share of episodes rated Ok or Excellent.
Share of the task's steps completed successfully.
Excellent-vs-Ok share of successful episodes.
Mean active duration of successful episodes. Lower is better.
Task steps completed per active minute, pooled over the successful episodes.
Spectral arc length (SPARC) of the arm's motion, a unitless measure ≤ -1: the closer to -1, the smoother. It is measured over each whole episode and averaged over episodes.
How abruptly the arm's acceleration changed: the end effector's jerk, filtered at 10 Hz and averaged over each episode, then over episodes. It grows with speed, so a faster motion reads as rougher. Lower is better.
Share of failed episodes that ended with both arms ok, from the arms' own status when the recording stopped.
The largest force either arm exerted on its surroundings during an episode, averaged over episodes. It is based on the FR3's joint torque sensors. Lower is better.
Summary
- Overall Success
- 28%
- Overall Progress
- 45%
- Overall Execution quality
- 79%
- Overall Avg execution time
- 40s
- Overall Execution speed
- 7.9/min
- Overall Smoothness
- -6.1
- Overall Avg jerk
- 4.9 m/s³
- Overall Safe failure rate
- 69%
- Overall Max contact force
- 66 N
- Evaluations
- 3600
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 0%#495% CI 0%–11% | 20%#195% CI 10%–37% | 27%#195% CI 14%–44% | 0%#195% CI 0%–11% | 0%#195% CI 0%–11% | 13%#195% CI 5%–30% | Coming soon | ||
| Datum-corner alignment | 10%#195% CI 3%–26% | 43%#195% CI 27%–61% | 73%#195% CI 56%–86% | 7%#195% CI 2%–21% | 60%#195% CI 42%–75% | 73%#195% CI 56%–86% | Coming soon | ||
| Pour screws into the funnel | 30%#195% CI 17%–48% | 77%#295% CI 59%–88% | 57%#395% CI 39%–73% | 13%#295% CI 5%–30% | 67%#295% CI 49%–81% | 67%#295% CI 49%–81% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 0%#195% CI 0%–11% | 30%#195% CI 17%–48% | 60%#195% CI 42%–75% | 0%#195% CI 0%–11% | 0%#395% CI 0%–11% | 0%#195% CI 0%–11% | Coming soon | ||
| Route cable through closed hoops | 0%#195% CI 0%–11% | 17%#195% CI 7%–34% | 50%#195% CI 33%–67% | 0%#195% CI 0%–11% | 3%#295% CI 1%–17% | 53%#195% CI 36%–70% | Coming soon | ||
| Put tools in standing toolbox | 0%#295% CI 0%–11% | 3%#195% CI 1%–17% | 10%#195% CI 3%–26% | 0%#195% CI 0%–11% | 3%#195% CI 1%–17% | 10%#195% CI 3%–26% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 3%#295% CI 1%–17% | 70%#195% CI 52%–83% | 80%#195% CI 63%–90% | 0%#295% CI 0%–11% | 33%#195% CI 19%–51% | 63%#195% CI 46%–78% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 13%#395% CI 5%–30% | 70%#395% CI 52%–83% | 63%#495% CI 46%–78% | 0%#395% CI 0%–11% | 73%#395% CI 56%–86% | 90%#195% CI 74%–97% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 0%#195% CI 0%–11% | 33%#195% CI 19%–51% | 17%#495% CI 7%–34% | 0%#395% CI 0%–11% | 33%#195% CI 19%–51% | 27%#295% CI 14%–44% | Coming soon | ||
| Clamp electrical component | 0%#195% CI 0%–11% | 43%#195% CI 27%–61% | 27%#195% CI 14%–44% | 0%#195% CI 0%–11% | 50%#195% CI 33%–67% | 23%#195% CI 12%–41% | Coming soon | ||
| Pooled | 6%#195% CI 4%–9% | 41%#195% CI 35%–46% | 46%#195% CI 41%–52% | 2%#395% CI 1%–4% | 32%#195% CI 27%–38% | 42%#195% CI 37%–48% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 5%#495% CI 2%–9% | 44%#295% CI 31%–57% | 63%#295% CI 50%–75% | 3%#295% CI 0%–6% | 25%#395% CI 17%–34% | 44%#395% CI 35%–53% | Coming soon | ||
| Datum-corner alignment | 10%#195% CI 0%–21% | 43%#195% CI 25%–61% | 73%#195% CI 57%–89% | 7%#195% CI 0%–16% | 60%#195% CI 42%–78% | 73%#195% CI 57%–89% | Coming soon | ||
| Pour screws into the funnel | 33%#195% CI 17%–50% | 80%#295% CI 66%–94% | 63%#495% CI 48%–79% | 17%#295% CI 4%–29% | 69%#295% CI 53%–85% | 70%#395% CI 54%–86% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 7%#395% CI 3%–10% | 57%#195% CI 44%–70% | 80%#195% CI 69%–91% | 4%#195% CI 2%–7% | 23%#295% CI 19%–26% | 23%#395% CI 20%–26% | Coming soon | ||
| Route cable through closed hoops | 13%#195% CI 12%–15% | 50%#195% CI 38%–62% | 65%#195% CI 51%–79% | 15%#195% CI 12%–19% | 31%#395% CI 21%–41% | 68%#195% CI 55%–82% | Coming soon | ||
| Put tools in standing toolbox | 10%#295% CI 4%–16% | 31%#195% CI 22%–40% | 33%#295% CI 22%–45% | 0%#395% CI 0%–0% | 28%#195% CI 17%–39% | 33%#195% CI 21%–46% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 40%#195% CI 29%–51% | 92%#195% CI 87%–97% | 93%#195% CI 85%–100% | 38%#295% CI 29%–46% | 82%#195% CI 76%–87% | 92%#195% CI 86%–98% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 35%#295% CI 21%–49% | 91%#295% CI 85%–98% | 74%#495% CI 61%–87% | 40%#295% CI 30%–50% | 83%#395% CI 72%–94% | 91%#295% CI 81%–100% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 11%#495% CI 8%–13% | 51%#295% CI 37%–65% | 37%#495% CI 26%–48% | 13%#495% CI 10%–16% | 47%#195% CI 34%–61% | 51%#395% CI 39%–64% | Coming soon | ||
| Clamp electrical component | 12%#295% CI 5%–20% | 69%#195% CI 57%–81% | 60%#195% CI 49%–71% | 18%#295% CI 10%–25% | 80%#195% CI 72%–88% | 53%#195% CI 42%–64% | Coming soon | ||
| Pooled | 18%#295% CI 15%–21% | 61%#195% CI 56%–65% | 64%#195% CI 60%–68% | 15%#295% CI 13%–18% | 53%#195% CI 48%–57% | 60%#195% CI 56%–64% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | Results unavailable | 67%#195% CI 30%–90% | 38%#295% CI 14%–69% | Coming soon | Coming soon | 25%#195% CI 5%–70% | Coming soon | ||
| Datum-corner alignment | 67%#195% CI 21%–94% | 100%#195% CI 77%–100% | 82%#195% CI 61%–93% | 0%#295% CI 0%–66% | 72%#195% CI 49%–88% | 68%#195% CI 47%–84% | Coming soon | ||
| Pour screws into the funnel | 67%#395% CI 35%–88% | 87%#295% CI 68%–95% | 82%#495% CI 59%–94% | 50%#295% CI 15%–85% | 90%#395% CI 70%–97% | 75%#495% CI 53%–89% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 100%#195% CI 70%–100% | 94%#195% CI 74%–99% | Coming soon | Results unavailable | Coming soon | Coming soon | ||
| Route cable through closed hoops | Coming soon | 0%#295% CI 0%–43% | 87%#195% CI 62%–96% | Coming soon | 100%#195% CI 21%–100% | 44%#195% CI 23%–67% | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 0%#295% CI 0%–79% | 0%#495% CI 0%–56% | Coming soon | 100%#195% CI 21%–100% | 67%#195% CI 21%–94% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 0%#495% CI 0%–79% | 86%#295% CI 65%–95% | 83%#395% CI 64%–93% | Results unavailable | 50%#295% CI 24%–76% | 74%#395% CI 51%–88% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 100%#195% CI 51%–100% | 86%#295% CI 65%–95% | 95%#295% CI 75%–99% | Results unavailable | 86%#395% CI 67%–95% | 96%#295% CI 82%–99% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | Coming soon | 90%#395% CI 60%–98% | 80%#395% CI 38%–96% | Results unavailable | 70%#195% CI 40%–89% | 75%#295% CI 41%–93% | Coming soon | ||
| Clamp electrical component | Coming soon | 100%#195% CI 77%–100% | 50%#395% CI 22%–78% | Coming soon | 93%#195% CI 70%–99% | 71%#195% CI 36%–92% | Coming soon | ||
| Pooled | 71%#195% CI 47%–87% | 85%#295% CI 78%–90% | 80%#395% CI 72%–86% | 33%#395% CI 10%–70% | 80%#295% CI 71%–87% | 72%#395% CI 64%–79% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | Results unavailable | 58s#195% CI 48s–68s | 67s#295% CI 53s–80s | Coming soon | Coming soon | 64s#195% CI 53s–76s | Coming soon | ||
| Datum-corner alignment | 14s#195% CI 13s–14s | 14s#195% CI 13s–15s | 15s#195% CI 14s–17s | 15s#195% CI 13s–16s | 13s#195% CI 12s–14s | 14s#295% CI 13s–15s | Coming soon | ||
| Pour screws into the funnel | 14s#195% CI 13s–15s | 14s#195% CI 13s–14s | 14s#295% CI 13s–14s | 14s#195% CI 13s–14s | 13s#195% CI 12s–13s | 13s#195% CI 13s–14s | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 62s#295% CI 60s–65s | 61s#195% CI 59s–63s | Coming soon | Results unavailable | Coming soon | Coming soon | ||
| Route cable through closed hoops | Coming soon | 66s#195% CI 59s–72s | 66s#195% CI 64s–69s | Coming soon | 64s#1 | 62s#195% CI 60s–64s | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 66s#2 | 74s#495% CI 70s–77s | Coming soon | 62s#1 | 66s#195% CI 62s–70s | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 39s#2 | 39s#195% CI 37s–40s | 40s#395% CI 39s–41s | Results unavailable | 37s#295% CI 36s–39s | 40s#395% CI 38s–41s | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 63s#195% CI 59s–67s | 64s#395% CI 63s–65s | 63s#195% CI 61s–65s | Results unavailable | 65s#395% CI 64s–67s | 64s#195% CI 63s–66s | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | Coming soon | 40s#395% CI 39s–41s | 41s#295% CI 39s–43s | Results unavailable | 40s#295% CI 39s–42s | 41s#295% CI 40s–43s | Coming soon | ||
| Clamp electrical component | Coming soon | 42s#395% CI 38s–46s | 38s#195% CI 33s–44s | Coming soon | 38s#195% CI 34s–42s | 43s#195% CI 40s–47s | Coming soon | ||
| Pooled | 27s#195% CI 17s–37s | 40s#395% CI 37s–44s | 44s#495% CI 40s–47s | 14s#195% CI 13s–15s | 35s#295% CI 31s–39s | 41s#395% CI 37s–45s | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | Results unavailable | 8.3/min#195% CI 6.8/min–9.7/min | 7.0/min#295% CI 5.6/min–8.4/min | Coming soon | Coming soon | 5.1/min#295% CI 3.0/min–7.3/min | Coming soon | ||
| Datum-corner alignment | 4.4/min#195% CI 4.1/min–4.7/min | 4.4/min#195% CI 4.0/min–4.7/min | 3.9/min#195% CI 3.6/min–4.2/min | 4.1/min#195% CI 3.8/min–4.4/min | 4.6/min#195% CI 4.1/min–5.0/min | 4.3/min#295% CI 4.0/min–4.6/min | Coming soon | ||
| Pour screws into the funnel | 12.9/min#195% CI 12.4/min–13.3/min | 13.3/min#195% CI 12.8/min–13.8/min | 13.1/min#295% CI 12.6/min–13.6/min | 13.2/min#195% CI 12.5/min–13.9/min | 14.0/min#195% CI 13.6/min–14.5/min | 13.5/min#195% CI 12.9/min–14.0/min | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 7.7/min#295% CI 7.4/min–8.0/min | 7.8/min#195% CI 7.5/min–8.1/min | Coming soon | Results unavailable | Coming soon | Coming soon | ||
| Route cable through closed hoops | Coming soon | 7.3/min#195% CI 6.6/min–8.0/min | 7.1/min#195% CI 6.8/min–7.5/min | Coming soon | 7.5/min#1 | 7.7/min#195% CI 7.5/min–7.9/min | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 9.1/min#2 | 8.2/min#495% CI 7.8/min–8.5/min | Coming soon | 9.7/min#1 | 9.1/min#195% CI 8.6/min–9.6/min | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 6.2/min#2 | 6.2/min#195% CI 6.0/min–6.4/min | 6.0/min#395% CI 5.9/min–6.1/min | Results unavailable | 6.4/min#295% CI 6.1/min–6.7/min | 6.1/min#395% CI 5.8/min–6.3/min | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 8.6/min#195% CI 8.0/min–9.1/min | 8.4/min#395% CI 8.2/min–8.5/min | 8.5/min#195% CI 8.2/min–8.8/min | Results unavailable | 8.3/min#395% CI 8.1/min–8.5/min | 8.4/min#195% CI 8.2/min–8.5/min | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | Coming soon | 13.3/min#395% CI 12.8/min–13.8/min | 13.1/min#295% CI 12.4/min–13.8/min | Results unavailable | 12.9/min#195% CI 12.1/min–13.7/min | 13.0/min#195% CI 12.5/min–13.5/min | Coming soon | ||
| Clamp electrical component | Coming soon | 4.3/min#395% CI 3.9/min–4.7/min | 4.7/min#195% CI 4.0/min–5.4/min | Coming soon | 4.7/min#195% CI 4.3/min–5.2/min | 4.2/min#195% CI 3.8/min–4.5/min | Coming soon | ||
| Pooled | 9.2/min#195% CI 8.1/min–10.3/min | 8.0/min#495% CI 7.5/min–8.4/min | 7.5/min#495% CI 7.2/min–7.8/min | 10.0/min#295% CI 6.2/min–13.9/min | 8.2/min#495% CI 7.7/min–8.8/min | 7.8/min#495% CI 7.4/min–8.3/min | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | -5.3#295% CI -5.9–-4.7 | -6.4#295% CI -7.3–-5.5 | -8.1#495% CI -8.8–-7.3 | -5.6#395% CI -6.4–-4.9 | -6.4#395% CI -7.4–-5.5 | -8.3#495% CI -8.9–-7.7 | Coming soon | ||
| Datum-corner alignment | -5.1#195% CI -5.8–-4.4 | -5.1#195% CI -5.7–-4.4 | -5.2#195% CI -5.6–-4.8 | -5.3#395% CI -5.9–-4.6 | -5.8#195% CI -6.3–-5.3 | -5.8#195% CI -6.2–-5.4 | Coming soon | ||
| Pour screws into the funnel | -4.0#195% CI -4.4–-3.7 | -4.3#195% CI -4.7–-4.0 | -4.4#195% CI -4.8–-3.9 | -3.5#195% CI -3.9–-3.1 | -4.2#195% CI -4.7–-3.8 | -4.5#195% CI -4.8–-4.1 | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | -4.6#495% CI -5.1–-4.2 | -7.2#495% CI -8.4–-6.1 | -9.1#495% CI -10.0–-8.2 | -4.3#395% CI -4.6–-3.9 | -4.7#295% CI -5.2–-4.3 | -4.8#295% CI -5.4–-4.2 | Coming soon | ||
| Route cable through closed hoops | -6.0#295% CI -6.6–-5.4 | -8.3#495% CI -9.0–-7.7 | -8.7#495% CI -9.3–-8.0 | -6.2#395% CI -7.0–-5.5 | -8.4#495% CI -9.4–-7.4 | -8.8#395% CI -9.5–-8.0 | Coming soon | ||
| Put tools in standing toolbox | -4.3#295% CI -5.1–-3.5 | -5.7#295% CI -6.7–-4.8 | -5.8#395% CI -6.6–-5.1 | -3.4#195% CI -3.7–-3.1 | -6.2#395% CI -7.1–-5.3 | -5.6#395% CI -6.5–-4.8 | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | -3.5#195% CI -3.8–-3.1 | -4.0#295% CI -4.2–-3.8 | -4.1#395% CI -4.5–-3.7 | -3.4#295% CI -3.7–-3.1 | -4.3#295% CI -4.6–-3.9 | -4.0#395% CI -4.2–-3.7 | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | -6.1#395% CI -7.2–-5.1 | -9.5#495% CI -10.1–-8.8 | -7.5#195% CI -8.3–-6.8 | -6.5#395% CI -7.5–-5.6 | -8.7#295% CI -9.4–-8.0 | -9.1#395% CI -10.0–-8.3 | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | -2.7#195% CI -3.1–-2.4 | -7.4#295% CI -8.6–-6.3 | -7.4#195% CI -8.6–-6.3 | -3.1#195% CI -3.5–-2.7 | -7.1#295% CI -8.2–-6.0 | -7.6#195% CI -8.7–-6.4 | Coming soon | ||
| Clamp electrical component | -4.8#195% CI -5.3–-4.3 | -11.5#495% CI -13.5–-9.5 | -9.6#495% CI -11.6–-7.5 | -5.4#495% CI -6.0–-4.8 | -9.9#495% CI -11.9–-7.8 | -7.5#495% CI -9.3–-5.8 | Coming soon | ||
| Pooled | -4.7#195% CI -4.9–-4.4 | -6.9#495% CI -7.3–-6.5 | -7.0#495% CI -7.4–-6.6 | -4.7#295% CI -4.9–-4.4 | -6.6#395% CI -6.9–-6.2 | -6.6#395% CI -6.9–-6.3 | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 5.5 m/s³#195% CI 4.7 m/s³–6.2 m/s³ | 5.4 m/s³#195% CI 4.1 m/s³–6.8 m/s³ | 4.4 m/s³#195% CI 3.5 m/s³–5.3 m/s³ | 4.5 m/s³#195% CI 4.1 m/s³–4.9 m/s³ | 6 m/s³#295% CI 4.7 m/s³–7.3 m/s³ | 4.5 m/s³#195% CI 3.6 m/s³–5.4 m/s³ | Coming soon | ||
| Datum-corner alignment | 3.6 m/s³#195% CI 2.5 m/s³–4.8 m/s³ | 5 m/s³#295% CI 3.1 m/s³–6.9 m/s³ | 3.2 m/s³#195% CI 2.7 m/s³–3.6 m/s³ | 3.5 m/s³#195% CI 2.8 m/s³–4.2 m/s³ | 3.7 m/s³#195% CI 2.9 m/s³–4.6 m/s³ | 2.9 m/s³#195% CI 2.3 m/s³–3.5 m/s³ | Coming soon | ||
| Pour screws into the funnel | 6 m/s³#195% CI 5.3 m/s³–6.7 m/s³ | 8.4 m/s³#195% CI 6.1 m/s³–11 m/s³ | 8.7 m/s³#195% CI 7 m/s³–10 m/s³ | 7.4 m/s³#195% CI 5.6 m/s³–9.2 m/s³ | 9.6 m/s³#195% CI 6.8 m/s³–12 m/s³ | 7.8 m/s³#195% CI 6.3 m/s³–9.3 m/s³ | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 4.1 m/s³#195% CI 3.3 m/s³–4.8 m/s³ | 3.5 m/s³#295% CI 3 m/s³–4.1 m/s³ | 3.9 m/s³#295% CI 3.2 m/s³–4.7 m/s³ | 3.1 m/s³#195% CI 2.5 m/s³–3.8 m/s³ | 3.4 m/s³#195% CI 3 m/s³–3.8 m/s³ | 4.4 m/s³#295% CI 3.4 m/s³–5.3 m/s³ | Coming soon | ||
| Route cable through closed hoops | 2.7 m/s³#195% CI 2.1 m/s³–3.4 m/s³ | 2.3 m/s³#195% CI 2.1 m/s³–2.5 m/s³ | 2.8 m/s³#195% CI 2.2 m/s³–3.4 m/s³ | 2.8 m/s³#195% CI 2.2 m/s³–3.4 m/s³ | 2.7 m/s³#195% CI 2.2 m/s³–3.1 m/s³ | 2.6 m/s³#195% CI 2.2 m/s³–3 m/s³ | Coming soon | ||
| Put tools in standing toolbox | 4.4 m/s³#195% CI 3.5 m/s³–5.3 m/s³ | 4.2 m/s³#195% CI 3.6 m/s³–4.7 m/s³ | 5.4 m/s³#195% CI 4 m/s³–6.7 m/s³ | 5 m/s³#195% CI 3.9 m/s³–6.1 m/s³ | 4.7 m/s³#195% CI 3.9 m/s³–5.5 m/s³ | 5 m/s³#195% CI 4.1 m/s³–5.9 m/s³ | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 6.7 m/s³#195% CI 5.2 m/s³–8.3 m/s³ | 5.1 m/s³#195% CI 4.1 m/s³–6 m/s³ | 6.3 m/s³#295% CI 4.7 m/s³–8 m/s³ | 5.6 m/s³#195% CI 5.3 m/s³–5.9 m/s³ | 6.8 m/s³#295% CI 5 m/s³–8.6 m/s³ | 5.4 m/s³#195% CI 4.3 m/s³–6.5 m/s³ | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 6.5 m/s³#195% CI 5.2 m/s³–7.8 m/s³ | 6 m/s³#195% CI 4.9 m/s³–7.2 m/s³ | 5.3 m/s³#195% CI 4.4 m/s³–6.2 m/s³ | 6.4 m/s³#195% CI 5.2 m/s³–7.7 m/s³ | 4.9 m/s³#195% CI 4.2 m/s³–5.7 m/s³ | 5.8 m/s³#195% CI 4.5 m/s³–7.1 m/s³ | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 3.8 m/s³#195% CI 3.4 m/s³–4.3 m/s³ | 6.7 m/s³#195% CI 4.8 m/s³–8.5 m/s³ | 5.1 m/s³#195% CI 4 m/s³–6.2 m/s³ | 4.5 m/s³#195% CI 3.4 m/s³–5.6 m/s³ | 5.4 m/s³#295% CI 4.4 m/s³–6.4 m/s³ | 5.9 m/s³#295% CI 4.4 m/s³–7.5 m/s³ | Coming soon | ||
| Clamp electrical component | 5 m/s³#195% CI 4 m/s³–6 m/s³ | 4.3 m/s³#195% CI 3.3 m/s³–5.3 m/s³ | 3.5 m/s³#195% CI 2.9 m/s³–4.1 m/s³ | 4.6 m/s³#195% CI 4 m/s³–5.3 m/s³ | 5.2 m/s³#195% CI 3.9 m/s³–6.6 m/s³ | 3.8 m/s³#195% CI 3 m/s³–4.6 m/s³ | Coming soon | ||
| Pooled | 4.8 m/s³#195% CI 4.5 m/s³–5.2 m/s³ | 5.1 m/s³#195% CI 4.6 m/s³–5.5 m/s³ | 4.9 m/s³#195% CI 4.5 m/s³–5.2 m/s³ | 4.7 m/s³#195% CI 4.4 m/s³–5.1 m/s³ | 5.2 m/s³#195% CI 4.8 m/s³–5.7 m/s³ | 4.8 m/s³#195% CI 4.4 m/s³–5.2 m/s³ | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 7%#495% CI 2%–21% | 38%#495% CI 21%–57% | 77%#295% CI 57%–90% | 0%#495% CI 0%–11% | 50%#495% CI 33%–67% | 73%#495% CI 54%–86% | Coming soon | ||
| Datum-corner alignment | 100%#195% CI 88%–100% | 88%#495% CI 66%–97% | 100%#195% CI 68%–100% | 96%#395% CI 82%–99% | 100%#195% CI 76%–100% | 100%#195% CI 68%–100% | Coming soon | ||
| Pour screws into the funnel | 95%#295% CI 77%–99% | 86%#395% CI 49%–97% | 69%#395% CI 42%–87% | 96%#395% CI 81%–99% | 90%#295% CI 60%–98% | 80%#295% CI 49%–94% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 13%#495% CI 5%–30% | 62%#495% CI 41%–79% | 83%#395% CI 55%–95% | 20%#395% CI 10%–37% | 43%#495% CI 27%–61% | 87%#495% CI 70%–95% | Coming soon | ||
| Route cable through closed hoops | 90%#495% CI 74%–97% | 68%#495% CI 48%–83% | 47%#495% CI 25%–70% | 67%#495% CI 49%–81% | 83%#395% CI 65%–92% | 50%#495% CI 27%–73% | Coming soon | ||
| Put tools in standing toolbox | 73%#495% CI 56%–86% | 59%#495% CI 41%–74% | 44%#495% CI 28%–63% | 67%#495% CI 49%–81% | 55%#495% CI 38%–72% | 41%#495% CI 25%–59% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 52%#495% CI 34%–69% | 89%#495% CI 57%–98% | 67%#495% CI 30%–90% | 40%#495% CI 25%–58% | 80%#495% CI 58%–92% | 64%#495% CI 35%–85% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 88%#495% CI 71%–96% | 89%#295% CI 57%–98% | 73%#295% CI 43%–90% | 60%#495% CI 42%–75% | 88%#395% CI 53%–98% | 33%#495% CI 6%–79% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 90%#395% CI 74%–97% | 75%#495% CI 53%–89% | 80%#495% CI 61%–91% | 90%#395% CI 74%–97% | 90%#495% CI 70%–97% | 77%#495% CI 57%–90% | Coming soon | ||
| Clamp electrical component | 100%#195% CI 89%–100% | 100%#195% CI 82%–100% | 100%#195% CI 85%–100% | 100%#195% CI 89%–100% | 87%#495% CI 62%–96% | 96%#495% CI 79%–99% | Coming soon | ||
| Pooled | 70%#495% CI 64%–75% | 70%#495% CI 63%–76% | 73%#495% CI 65%–79% | 63%#495% CI 57%–68% | 70%#495% CI 64%–76% | 72%#495% CI 65%–79% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 135 N#495% CI 121 N–149 N | 118 N#495% CI 104 N–132 N | 69 N#395% CI 58 N–81 N | 149 N#495% CI 143 N–154 N | 126 N#495% CI 114 N–137 N | 55 N#195% CI 43 N–66 N | Coming soon | ||
| Datum-corner alignment | 27 N#395% CI 20 N–35 N | 25 N#395% CI 19 N–32 N | 21 N#295% CI 16 N–26 N | 36 N#495% CI 28 N–44 N | 22 N#195% CI 17 N–27 N | 20 N#195% CI 16 N–24 N | Coming soon | ||
| Pour screws into the funnel | 26 N#395% CI 14 N–38 N | 25 N#395% CI 17 N–33 N | 29 N#495% CI 16 N–42 N | 19 N#395% CI 11 N–27 N | 30 N#495% CI 21 N–39 N | 39 N#495% CI 26 N–52 N | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 151 N#495% CI 143 N–159 N | 99 N#495% CI 85 N–114 N | 80 N#295% CI 70 N–90 N | 150 N#495% CI 140 N–159 N | 114 N#495% CI 100 N–127 N | 90 N#495% CI 76 N–103 N | Coming soon | ||
| Route cable through closed hoops | 73 N#395% CI 60 N–85 N | 51 N#395% CI 42 N–59 N | 65 N#395% CI 52 N–77 N | 70 N#395% CI 59 N–81 N | 58 N#495% CI 44 N–72 N | 54 N#495% CI 43 N–64 N | Coming soon | ||
| Put tools in standing toolbox | 94 N#495% CI 80 N–108 N | 100 N#495% CI 85 N–115 N | 114 N#495% CI 102 N–126 N | 95 N#495% CI 80 N–111 N | 116 N#495% CI 104 N–129 N | 120 N#495% CI 110 N–129 N | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 86 N#495% CI 72 N–100 N | 48 N#395% CI 39 N–57 N | 41 N#395% CI 32 N–49 N | 93 N#495% CI 76 N–111 N | 70 N#395% CI 57 N–84 N | 57 N#395% CI 42 N–71 N | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 55 N#495% CI 44 N–66 N | 43 N#495% CI 35 N–50 N | 54 N#495% CI 41 N–67 N | 84 N#495% CI 70 N–98 N | 44 N#495% CI 36 N–53 N | 45 N#495% CI 38 N–53 N | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 55 N#495% CI 39 N–70 N | 69 N#495% CI 55 N–84 N | 68 N#495% CI 52 N–85 N | 53 N#495% CI 39 N–68 N | 78 N#495% CI 64 N–91 N | 87 N#495% CI 71 N–104 N | Coming soon | ||
| Clamp electrical component | 32 N#395% CI 22 N–43 N | 32 N#495% CI 27 N–36 N | 27 N#395% CI 24 N–31 N | 31 N#495% CI 24 N–38 N | 31 N#495% CI 26 N–36 N | 26 N#495% CI 22 N–30 N | Coming soon | ||
| Pooled | 73 N#495% CI 67 N–79 N | 61 N#495% CI 56 N–66 N | 57 N#495% CI 52 N–62 N | 78 N#495% CI 72 N–84 N | 69 N#495% CI 63 N–74 N | 59 N#495% CI 54 N–64 N | Coming soon | ||
Data Scaling
Spatial Generalization
Per Environment (SuccessProgressExecution qualityAvg execution timeExecution speedSmoothnessAvg jerkSafe failure rateMax contact force)
Training parameters
- Starting checkpoint
- allenai/MolmoAct2
- Frozen part
- None
- Total batch size
- 64
- Learning rate
- Action expert 5e-5LLM 1e-5ViT 5e-6Connector 5e-6
- Schedule
- Warmup, then cosine to 0.1× peak
- Action chunk
- 30
- Action space
- Delta commanded TCP pose
Rollouts
Mid-training dataM0
Task-specific dataD300
SpatialNominal
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating; a placement several episodes shared is drawn once, in their ratings' colours mixed. Use the button beside each environment to open its evaluation dataset in the dataset viewer. One episode is available to watch; the remaining episodes can be requested in case of a dispute or for another legitimate reason.
Sort screws into bins - Success
- 27%
- Progress
- 63%
Pour screws into the funnel - Success
- 57%
- Progress
- 63%
Connect DC jack - Success
- 60%
- Progress
- 80%
Route cable through closed hoops - Success
- 50%
- Progress
- 65%
Open the toolbox with a screwdriver - Success
- 17%
- Progress
- 37%
- Held-out benchmark environment
Datum-corner alignment - Success
- 73%
- Progress
- 73%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 10%
- Progress
- 33%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 80%
- Progress
- 93%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 63%
- Progress
- 74%
- Held-out benchmark environment
Clamp electrical component - Success
- 27%
- Progress
- 60%
Sort screws into bins - Success
- 13%
- Progress
- 44%
Pour screws into the funnel - Success
- 67%
- Progress
- 70%
Connect DC jack - Success
- 0%
- Progress
- 23%
Route cable through closed hoops - Success
- 53%
- Progress
- 68%
Open the toolbox with a screwdriver - Success
- 27%
- Progress
- 51%
- Held-out benchmark environment
Datum-corner alignment - Success
- 73%
- Progress
- 73%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 10%
- Progress
- 33%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 63%
- Progress
- 92%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 90%
- Progress
- 91%
- Held-out benchmark environment
Clamp electrical component - Success
- 23%
- Progress
- 53%
Sort screws into bins - Success
- 20%
- Progress
- 44%
Pour screws into the funnel - Success
- 77%
- Progress
- 80%
Connect DC jack - Success
- 30%
- Progress
- 57%
Route cable through closed hoops - Success
- 17%
- Progress
- 50%
Open the toolbox with a screwdriver - Success
- 33%
- Progress
- 51%
- Held-out benchmark environment
Datum-corner alignment - Success
- 43%
- Progress
- 43%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 3%
- Progress
- 31%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 70%
- Progress
- 92%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 70%
- Progress
- 91%
- Held-out benchmark environment
Clamp electrical component - Success
- 43%
- Progress
- 69%
Sort screws into bins - Success
- 0%
- Progress
- 25%
Pour screws into the funnel - Success
- 67%
- Progress
- 69%
Connect DC jack - Success
- 0%
- Progress
- 23%
Route cable through closed hoops - Success
- 3%
- Progress
- 31%
Open the toolbox with a screwdriver - Success
- 33%
- Progress
- 47%
- Held-out benchmark environment
Datum-corner alignment - Success
- 60%
- Progress
- 60%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 3%
- Progress
- 28%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 33%
- Progress
- 82%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 73%
- Progress
- 83%
- Held-out benchmark environment
Clamp electrical component - Success
- 50%
- Progress
- 80%
Sort screws into bins - Success
- 0%
- Progress
- 5%
Pour screws into the funnel - Success
- 30%
- Progress
- 33%
Connect DC jack - Success
- 0%
- Progress
- 7%
Route cable through closed hoops - Success
- 0%
- Progress
- 13%
Open the toolbox with a screwdriver - Success
- 0%
- Progress
- 11%
- Held-out benchmark environment
Datum-corner alignment - Success
- 10%
- Progress
- 10%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 0%
- Progress
- 10%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 3%
- Progress
- 40%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 13%
- Progress
- 35%
- Held-out benchmark environment
Clamp electrical component - Success
- 0%
- Progress
- 12%
Sort screws into bins - Success
- 0%
- Progress
- 3%
Pour screws into the funnel - Success
- 13%
- Progress
- 17%
Connect DC jack - Success
- 0%
- Progress
- 4%
Route cable through closed hoops - Success
- 0%
- Progress
- 15%
Open the toolbox with a screwdriver - Success
- 0%
- Progress
- 13%
- Held-out benchmark environment
Datum-corner alignment - Success
- 7%
- Progress
- 7%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 0%
- Progress
- 0%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 0%
- Progress
- 38%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 0%
- Progress
- 40%
- Held-out benchmark environment
Clamp electrical component - Success
- 0%
- Progress
- 18%