ModelRho
Microsoft Research
#1
Rank · PAW-GEN-10
Model information
PAW-GEN-10
Share of episodes rated Ok or Excellent.
Share of the task's steps completed successfully.
Excellent-vs-Ok share of successful episodes.
Mean active duration of successful episodes. Lower is better.
Task steps completed per active minute, pooled over the successful episodes.
Spectral arc length (SPARC) of the arm's motion, a unitless measure ≤ -1: the closer to -1, the smoother. It is measured over each whole episode and averaged over episodes.
How abruptly the arm's acceleration changed: the end effector's jerk, filtered at 10 Hz and averaged over each episode, then over episodes. It grows with speed, so a faster motion reads as rougher. Lower is better.
Share of failed episodes that ended with both arms ok, from the arms' own status when the recording stopped.
The largest force either arm exerted on its surroundings during an episode, averaged over episodes. It is based on the FR3's joint torque sensors. Lower is better.
Summary
- Overall Success
- 32%
- Overall Progress
- 53%
- Overall Execution quality
- 73%
- Overall Avg execution time
- 41s
- Overall Execution speed
- 8.4/min
- Overall Smoothness
- -7.2
- Overall Avg jerk
- 16 m/s³
- Overall Safe failure rate
- 60%
- Overall Max contact force
- 59 N
- Evaluations
- 3600
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 7%#195% CI 2%–21% | 70%#195% CI 52%–83% | 73%#195% CI 56%–86% | 0%#195% CI 0%–11% | 3%#195% CI 1%–17% | 17%#195% CI 7%–34% | Coming soon | ||
| Datum-corner alignment | 33%#195% CI 19%–51% | 67%#195% CI 49%–81% | 57%#395% CI 39%–73% | 37%#195% CI 22%–54% | 60%#195% CI 42%–75% | 63%#295% CI 46%–78% | Coming soon | ||
| Pour screws into the funnel | 13%#395% CI 5%–30% | 70%#395% CI 52%–83% | 90%#195% CI 74%–97% | 3%#395% CI 1%–17% | 60%#395% CI 42%–75% | 87%#295% CI 70%–95% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 0%#195% CI 0%–11% | 30%#195% CI 17%–48% | 37%#295% CI 22%–54% | 0%#195% CI 0%–11% | 27%#195% CI 14%–44% | 30%#195% CI 17%–48% | Coming soon | ||
| Route cable through closed hoops | 0%#195% CI 0%–11% | 7%#395% CI 2%–21% | 7%#395% CI 2%–21% | 0%#195% CI 0%–11% | 0%#495% CI 0%–11% | 7%#395% CI 2%–21% | Coming soon | ||
| Put tools in standing toolbox | 0%#295% CI 0%–11% | 3%#195% CI 1%–17% | 17%#195% CI 7%–34% | 0%#195% CI 0%–11% | 3%#195% CI 1%–17% | 10%#195% CI 3%–26% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 10%#195% CI 3%–26% | 67%#295% CI 49%–81% | 63%#395% CI 46%–78% | 0%#295% CI 0%–11% | 67%#195% CI 49%–81% | 60%#295% CI 42%–75% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 17%#295% CI 7%–34% | 80%#295% CI 63%–90% | 83%#295% CI 66%–93% | 20%#195% CI 10%–37% | 80%#295% CI 63%–90% | 87%#395% CI 70%–95% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 3%#195% CI 1%–17% | 50%#195% CI 33%–67% | 87%#195% CI 70%–95% | 0%#395% CI 0%–11% | 43%#195% CI 27%–61% | 90%#195% CI 74%–97% | Coming soon | ||
| Clamp electrical component | 0%#195% CI 0%–11% | 3%#495% CI 1%–17% | 20%#295% CI 10%–37% | 0%#195% CI 0%–11% | 27%#295% CI 14%–44% | 3%#295% CI 1%–17% | Coming soon | ||
| Pooled | 8%#195% CI 6%–12% | 45%#195% CI 39%–50% | 53%#195% CI 48%–59% | 6%#195% CI 4%–9% | 37%#195% CI 32%–43% | 45%#195% CI 40%–51% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 22%#195% CI 11%–33% | 85%#195% CI 74%–97% | 86%#195% CI 77%–95% | 6%#295% CI 2%–11% | 51%#195% CI 37%–65% | 67%#195% CI 54%–79% | Coming soon | ||
| Datum-corner alignment | 33%#195% CI 16%–50% | 67%#195% CI 50%–84% | 57%#395% CI 39%–75% | 37%#195% CI 19%–54% | 60%#195% CI 42%–78% | 63%#295% CI 46%–81% | Coming soon | ||
| Pour screws into the funnel | 22%#495% CI 11%–34% | 78%#395% CI 66%–90% | 91%#195% CI 81%–100% | 14%#395% CI 6%–23% | 69%#295% CI 55%–83% | 89%#295% CI 78%–99% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 11%#195% CI 5%–17% | 65%#195% CI 54%–77% | 66%#295% CI 54%–79% | 8%#195% CI 4%–12% | 68%#195% CI 57%–78% | 67%#195% CI 56%–78% | Coming soon | ||
| Route cable through closed hoops | 5%#395% CI 2%–8% | 11%#595% CI 2%–21% | 17%#495% CI 7%–26% | 8%#395% CI 3%–12% | 8%#595% CI 0%–15% | 23%#495% CI 10%–35% | Coming soon | ||
| Put tools in standing toolbox | 3%#495% CI 1%–5% | 37%#195% CI 25%–48% | 46%#195% CI 34%–58% | 3%#195% CI 0%–7% | 28%#195% CI 16%–40% | 46%#195% CI 35%–57% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 54%#195% CI 43%–65% | 92%#195% CI 87%–96% | 91%#295% CI 86%–95% | 53%#195% CI 43%–62% | 92%#195% CI 87%–96% | 90%#295% CI 86%–94% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 53%#295% CI 38%–67% | 96%#195% CI 93%–100% | 91%#295% CI 83%–99% | 61%#195% CI 48%–74% | 93%#395% CI 86%–100% | 89%#495% CI 79%–99% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 25%#295% CI 18%–32% | 70%#195% CI 58%–83% | 93%#195% CI 87%–99% | 23%#295% CI 20%–27% | 73%#195% CI 62%–83% | 93%#195% CI 85%–100% | Coming soon | ||
| Clamp electrical component | 34%#195% CI 23%–46% | 67%#295% CI 64%–70% | 66%#195% CI 58%–74% | 32%#195% CI 22%–43% | 71%#295% CI 63%–79% | 66%#195% CI 61%–70% | Coming soon | ||
| Pooled | 26%#195% CI 22%–30% | 67%#195% CI 63%–71% | 70%#195% CI 66%–75% | 25%#195% CI 21%–28% | 61%#195% CI 57%–66% | 69%#195% CI 65%–73% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 100%#195% CI 34%–100% | 100%#195% CI 85%–100% | 64%#195% CI 43%–80% | Coming soon | 0%#195% CI 0%–79% | 40%#195% CI 12%–77% | Coming soon | ||
| Datum-corner alignment | 100%#195% CI 72%–100% | 100%#195% CI 84%–100% | 76%#295% CI 53%–90% | 91%#195% CI 62%–98% | 100%#195% CI 82%–100% | 68%#195% CI 46%–85% | Coming soon | ||
| Pour screws into the funnel | 25%#595% CI 5%–70% | 14%#595% CI 5%–35% | 93%#295% CI 77%–98% | 0%#395% CI 0%–79% | 28%#595% CI 12%–51% | 92%#295% CI 76%–98% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 100%#195% CI 70%–100% | 55%#295% CI 28%–79% | Coming soon | 63%#295% CI 31%–86% | 56%#195% CI 27%–81% | Coming soon | ||
| Route cable through closed hoops | Coming soon | 50%#195% CI 9%–91% | 0%#395% CI 0%–66% | Coming soon | Results unavailable | 100%#195% CI 34%–100% | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 100%#195% CI 21%–100% | 100%#195% CI 57%–100% | Coming soon | 100%#195% CI 21%–100% | 100%#195% CI 44%–100% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 0%#495% CI 0%–56% | 90%#195% CI 70%–97% | 95%#195% CI 75%–99% | Results unavailable | 100%#195% CI 84%–100% | 100%#195% CI 82%–100% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 60%#295% CI 23%–88% | 25%#595% CI 12%–45% | 56%#595% CI 37%–73% | 0%#395% CI 0%–39% | 67%#595% CI 47%–82% | 58%#595% CI 39%–74% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 100%#195% CI 21%–100% | 73%#495% CI 48%–89% | 88%#395% CI 71%–96% | Results unavailable | 77%#195% CI 50%–92% | 81%#195% CI 63%–92% | Coming soon | ||
| Clamp electrical component | Coming soon | 0%#495% CI 0%–79% | 67%#395% CI 30%–90% | Coming soon | 88%#295% CI 53%–98% | 100%#195% CI 21%–100% | Coming soon | ||
| Pooled | 68%#395% CI 48%–83% | 67%#495% CI 59%–75% | 76%#595% CI 69%–82% | 56%#395% CI 34%–75% | 74%#395% CI 65%–81% | 77%#295% CI 69%–83% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 47s#295% CI 45s–49s | 51s#195% CI 49s–53s | 57s#195% CI 53s–61s | Coming soon | 78s#1 | 61s#195% CI 54s–68s | Coming soon | ||
| Datum-corner alignment | 17s#395% CI 15s–19s | 18s#495% CI 17s–20s | 14s#195% CI 13s–15s | 17s#395% CI 14s–20s | 21s#495% CI 19s–22s | 14s#395% CI 13s–15s | Coming soon | ||
| Pour screws into the funnel | 16s#495% CI 13s–18s | 13s#195% CI 13s–14s | 14s#395% CI 13s–15s | 14s#2 | 14s#495% CI 13s–14s | 13s#295% CI 13s–14s | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 65s#395% CI 63s–66s | 63s#295% CI 62s–65s | Coming soon | 64s#195% CI 61s–67s | 63s#195% CI 62s–64s | Coming soon | ||
| Route cable through closed hoops | Coming soon | 79s#395% CI 46s–113s | 95s#395% CI 89s–101s | Coming soon | Results unavailable | 65s#295% CI 59s–70s | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 61s#1 | 63s#195% CI 61s–65s | Coming soon | 61s#1 | 61s#195% CI 59s–62s | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 49s#595% CI 43s–56s | 41s#395% CI 39s–43s | 39s#295% CI 38s–39s | Results unavailable | 39s#495% CI 38s–40s | 38s#295% CI 37s–38s | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 64s#295% CI 61s–68s | 67s#595% CI 65s–69s | 66s#595% CI 65s–68s | 71s#395% CI 66s–76s | 66s#595% CI 65s–67s | 65s#295% CI 64s–65s | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 43s#1 | 41s#495% CI 40s–42s | 43s#395% CI 41s–45s | Results unavailable | 40s#295% CI 39s–41s | 43s#395% CI 41s–44s | Coming soon | ||
| Clamp electrical component | Coming soon | 63s#4 | 35s#195% CI 29s–41s | Coming soon | 45s#295% CI 39s–52s | 49s#2 | Coming soon | ||
| Pooled | 34s#395% CI 25s–42s | 42s#495% CI 38s–45s | 42s#495% CI 39s–46s | 35s#395% CI 22s–47s | 41s#395% CI 37s–44s | 39s#295% CI 36s–43s | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 10.2/min#295% CI 9.8/min–10.6/min | 9.4/min#195% CI 9.0/min–9.8/min | 8.4/min#195% CI 7.8/min–9.0/min | Coming soon | 6.2/min#1 | 7.9/min#195% CI 7.0/min–8.8/min | Coming soon | ||
| Datum-corner alignment | 3.6/min#395% CI 3.1/min–4.0/min | 3.3/min#495% CI 3.0/min–3.5/min | 4.3/min#195% CI 3.9/min–4.6/min | 3.5/min#395% CI 3.0/min–4.1/min | 2.9/min#495% CI 2.7/min–3.1/min | 4.2/min#395% CI 3.8/min–4.6/min | Coming soon | ||
| Pour screws into the funnel | 10.6/min#595% CI 7.0/min–14.2/min | 13.1/min#395% CI 12.4/min–13.8/min | 12.9/min#395% CI 12.3/min–13.5/min | 12.5/min#2 | 13.1/min#595% CI 12.3/min–13.8/min | 13.4/min#295% CI 13.2/min–13.7/min | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | Coming soon | 7.4/min#395% CI 7.3/min–7.6/min | 7.6/min#295% CI 7.3/min–7.8/min | Coming soon | 7.5/min#195% CI 7.2/min–7.8/min | 7.6/min#195% CI 7.5/min–7.8/min | Coming soon | ||
| Route cable through closed hoops | Coming soon | 6.0/min#395% CI 3.5/min–8.6/min | 5.1/min#395% CI 4.7/min–5.4/min | Coming soon | Results unavailable | 7.4/min#295% CI 6.8/min–8.0/min | Coming soon | ||
| Put tools in standing toolbox | Results unavailable | 9.9/min#1 | 9.5/min#195% CI 9.3/min–9.8/min | Coming soon | 9.9/min#1 | 9.9/min#195% CI 9.7/min–10.1/min | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 4.9/min#595% CI 4.3/min–5.5/min | 5.9/min#395% CI 5.6/min–6.1/min | 6.2/min#295% CI 6.1/min–6.4/min | Results unavailable | 6.2/min#495% CI 6.0/min–6.3/min | 6.4/min#295% CI 6.3/min–6.5/min | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 8.4/min#295% CI 7.9/min–8.9/min | 8.1/min#595% CI 7.9/min–8.4/min | 8.1/min#595% CI 8.0/min–8.3/min | 7.6/min#395% CI 7.1/min–8.2/min | 8.2/min#595% CI 8.1/min–8.3/min | 8.4/min#295% CI 8.2/min–8.5/min | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 12.5/min#1 | 13.2/min#495% CI 12.8/min–13.6/min | 12.5/min#395% CI 12.0/min–13.0/min | Results unavailable | 13.6/min#195% CI 13.2/min–13.9/min | 12.6/min#295% CI 12.1/min–13.0/min | Coming soon | ||
| Clamp electrical component | Coming soon | 2.9/min#4 | 5.1/min#195% CI 4.2/min–6.0/min | Coming soon | 4.0/min#295% CI 3.4/min–4.5/min | 3.7/min#2 | Coming soon | ||
| Pooled | 7.4/min#595% CI 6.2/min–8.5/min | 8.4/min#295% CI 7.9/min–8.8/min | 8.7/min#295% CI 8.3/min–9.0/min | 6.5/min#595% CI 5.5/min–7.5/min | 7.9/min#595% CI 7.3/min–8.4/min | 9.0/min#195% CI 8.6/min–9.5/min | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | -6.1#395% CI -7.3–-4.9 | -7.9#495% CI -8.7–-7.1 | -8.2#595% CI -9.1–-7.3 | -4.3#195% CI -5.3–-3.3 | -7.9#495% CI -9.2–-6.7 | -7.8#395% CI -8.6–-7.1 | Coming soon | ||
| Datum-corner alignment | -7.3#595% CI -8.1–-6.6 | -7.4#295% CI -7.8–-6.9 | -7.6#495% CI -8.2–-7.1 | -6.8#595% CI -7.3–-6.2 | -6.9#395% CI -7.5–-6.3 | -7.5#495% CI -8.0–-6.9 | Coming soon | ||
| Pour screws into the funnel | -4.0#195% CI -4.7–-3.4 | -5.7#495% CI -6.3–-5.1 | -5.6#395% CI -6.1–-5.0 | -4.0#295% CI -4.7–-3.4 | -5.3#495% CI -5.8–-4.7 | -5.4#495% CI -5.9–-4.9 | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | -5.4#595% CI -6.3–-4.5 | -9.3#595% CI -10.3–-8.2 | -8.2#495% CI -9.1–-7.4 | -5.2#595% CI -6.4–-4.1 | -10.3#595% CI -11.6–-8.9 | -8.8#595% CI -9.7–-8.0 | Coming soon | ||
| Route cable through closed hoops | -12.1#595% CI -13.3–-10.9 | -10.4#595% CI -11.8–-9.1 | -8.8#595% CI -9.4–-8.3 | -8.8#595% CI -9.5–-8.1 | -11.3#595% CI -12.5–-10.0 | -10.4#595% CI -11.4–-9.5 | Coming soon | ||
| Put tools in standing toolbox | -3.7#195% CI -4.6–-2.9 | -6.6#595% CI -7.7–-5.5 | -6.8#495% CI -7.7–-5.9 | -3.3#195% CI -3.7–-2.9 | -6.4#495% CI -7.7–-5.2 | -7.3#495% CI -8.3–-6.4 | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | -5.0#595% CI -5.8–-4.2 | -4.5#495% CI -4.9–-4.0 | -4.3#495% CI -4.8–-3.9 | -4.9#595% CI -5.5–-4.3 | -4.3#395% CI -4.8–-3.9 | -4.0#395% CI -4.3–-3.7 | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | -7.3#495% CI -8.5–-6.1 | -9.9#595% CI -10.9–-9.0 | -8.3#295% CI -8.7–-7.8 | -8.6#595% CI -9.8–-7.4 | -8.8#395% CI -9.3–-8.3 | -8.7#295% CI -9.5–-8.0 | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | -8.1#595% CI -9.2–-7.0 | -9.3#595% CI -10.3–-8.2 | -9.9#595% CI -10.6–-9.3 | -8.6#595% CI -9.9–-7.3 | -9.0#595% CI -9.9–-8.2 | -10.1#595% CI -10.8–-9.5 | Coming soon | ||
| Clamp electrical component | -4.1#195% CI -4.5–-3.6 | -8.2#495% CI -10.7–-5.7 | -10.0#595% CI -11.6–-8.5 | -5.1#495% CI -5.9–-4.3 | -8.9#495% CI -10.7–-7.2 | -5.8#495% CI -7.2–-4.5 | Coming soon | ||
| Pooled | -6.3#595% CI -6.7–-5.9 | -7.9#595% CI -8.3–-7.5 | -7.8#595% CI -8.1–-7.5 | -6.0#595% CI -6.3–-5.6 | -7.9#595% CI -8.3–-7.5 | -7.6#595% CI -7.9–-7.3 | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 28 m/s³#595% CI 20 m/s³–36 m/s³ | 15 m/s³#495% CI 14 m/s³–17 m/s³ | 16 m/s³#495% CI 14 m/s³–17 m/s³ | 49 m/s³#595% CI 37 m/s³–61 m/s³ | 24 m/s³#495% CI 18 m/s³–29 m/s³ | 18 m/s³#495% CI 15 m/s³–22 m/s³ | Coming soon | ||
| Datum-corner alignment | 12 m/s³#595% CI 11 m/s³–13 m/s³ | 11 m/s³#495% CI 9.8 m/s³–11 m/s³ | 9.8 m/s³#495% CI 8.9 m/s³–11 m/s³ | 13 m/s³#595% CI 11 m/s³–14 m/s³ | 11 m/s³#495% CI 9.5 m/s³–12 m/s³ | 10 m/s³#495% CI 9.1 m/s³–11 m/s³ | Coming soon | ||
| Pour screws into the funnel | 34 m/s³#495% CI 28 m/s³–39 m/s³ | 20 m/s³#495% CI 18 m/s³–21 m/s³ | 18 m/s³#495% CI 17 m/s³–19 m/s³ | 37 m/s³#495% CI 31 m/s³–42 m/s³ | 20 m/s³#495% CI 18 m/s³–21 m/s³ | 18 m/s³#495% CI 16 m/s³–20 m/s³ | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 10 m/s³#595% CI 9.3 m/s³–11 m/s³ | 8.9 m/s³#595% CI 7.9 m/s³–9.8 m/s³ | 11 m/s³#595% CI 9.9 m/s³–13 m/s³ | 11 m/s³#595% CI 11 m/s³–12 m/s³ | 10 m/s³#595% CI 9 m/s³–11 m/s³ | 10 m/s³#595% CI 9.4 m/s³–11 m/s³ | Coming soon | ||
| Route cable through closed hoops | 12 m/s³#495% CI 11 m/s³–13 m/s³ | 13 m/s³#595% CI 8.8 m/s³–18 m/s³ | 8.9 m/s³#495% CI 8.4 m/s³–9.4 m/s³ | 12 m/s³#495% CI 11 m/s³–12 m/s³ | 11 m/s³#595% CI 11 m/s³–12 m/s³ | 10 m/s³#595% CI 9.8 m/s³–11 m/s³ | Coming soon | ||
| Put tools in standing toolbox | 23 m/s³#595% CI 20 m/s³–27 m/s³ | 15 m/s³#595% CI 13 m/s³–17 m/s³ | 13 m/s³#495% CI 13 m/s³–14 m/s³ | 24 m/s³#595% CI 22 m/s³–27 m/s³ | 16 m/s³#495% CI 14 m/s³–19 m/s³ | 15 m/s³#495% CI 14 m/s³–16 m/s³ | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 15 m/s³#595% CI 13 m/s³–16 m/s³ | 13 m/s³#595% CI 11 m/s³–14 m/s³ | 13 m/s³#595% CI 11 m/s³–14 m/s³ | 15 m/s³#595% CI 14 m/s³–16 m/s³ | 13 m/s³#495% CI 12 m/s³–14 m/s³ | 12 m/s³#495% CI 11 m/s³–13 m/s³ | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 16 m/s³#595% CI 15 m/s³–18 m/s³ | 13 m/s³#495% CI 12 m/s³–15 m/s³ | 13 m/s³#495% CI 12 m/s³–14 m/s³ | 17 m/s³#595% CI 15 m/s³–18 m/s³ | 13 m/s³#495% CI 12 m/s³–14 m/s³ | 14 m/s³#495% CI 13 m/s³–15 m/s³ | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 17 m/s³#595% CI 16 m/s³–18 m/s³ | 17 m/s³#595% CI 15 m/s³–20 m/s³ | 15 m/s³#495% CI 14 m/s³–16 m/s³ | 16 m/s³#595% CI 15 m/s³–17 m/s³ | 16 m/s³#495% CI 14 m/s³–17 m/s³ | 15 m/s³#595% CI 14 m/s³–16 m/s³ | Coming soon | ||
| Clamp electrical component | 14 m/s³#495% CI 12 m/s³–16 m/s³ | 11 m/s³#495% CI 9.6 m/s³–12 m/s³ | 11 m/s³#495% CI 9.4 m/s³–12 m/s³ | 15 m/s³#495% CI 14 m/s³–17 m/s³ | 13 m/s³#495% CI 8.7 m/s³–18 m/s³ | 10 m/s³#495% CI 9.4 m/s³–11 m/s³ | Coming soon | ||
| Pooled | 18 m/s³#595% CI 17 m/s³–19 m/s³ | 14 m/s³#495% CI 13 m/s³–14 m/s³ | 13 m/s³#495% CI 12 m/s³–13 m/s³ | 21 m/s³#595% CI 19 m/s³–23 m/s³ | 15 m/s³#495% CI 14 m/s³–16 m/s³ | 13 m/s³#495% CI 13 m/s³–14 m/s³ | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 36%#495% CI 21%–54% | 67%#495% CI 35%–88% | 75%#495% CI 41%–93% | 30%#495% CI 17%–48% | 55%#495% CI 38%–72% | 84%#495% CI 65%–94% | Coming soon | ||
| Datum-corner alignment | 85%#595% CI 64%–95% | 90%#495% CI 60%–98% | 85%#595% CI 58%–96% | 95%#595% CI 75%–99% | 92%#595% CI 65%–99% | 82%#595% CI 52%–95% | Coming soon | ||
| Pour screws into the funnel | 27%#595% CI 14%–46% | 33%#595% CI 12%–65% | 33%#595% CI 6%–79% | 34%#595% CI 20%–53% | 8%#595% CI 1%–35% | 25%#495% CI 5%–70% | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 17%#495% CI 7%–34% | 62%#495% CI 41%–79% | 68%#495% CI 46%–85% | 3%#595% CI 1%–17% | 55%#495% CI 35%–73% | 90%#495% CI 71%–97% | Coming soon | ||
| Route cable through closed hoops | 47%#595% CI 30%–64% | 82%#395% CI 64%–92% | 96%#195% CI 82%–99% | 87%#395% CI 70%–95% | 87%#395% CI 70%–95% | 100%#195% CI 88%–100% | Coming soon | ||
| Put tools in standing toolbox | 0%#595% CI 0%–11% | 34%#595% CI 20%–53% | 72%#495% CI 52%–86% | 3%#595% CI 1%–17% | 28%#595% CI 15%–46% | 70%#495% CI 52%–84% | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 63%#495% CI 44%–78% | 100%#195% CI 72%–100% | 100%#195% CI 74%–100% | 70%#495% CI 52%–83% | 100%#195% CI 72%–100% | 100%#195% CI 76%–100% | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 48%#595% CI 30%–67% | 83%#395% CI 44%–97% | 80%#295% CI 38%–96% | 38%#595% CI 21%–57% | 50%#595% CI 19%–81% | 50%#495% CI 15%–85% | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 41%#595% CI 26%–59% | 60%#595% CI 36%–80% | 100%#195% CI 51%–100% | 43%#595% CI 27%–61% | 53%#595% CI 31%–74% | 100%#195% CI 44%–100% | Coming soon | ||
| Clamp electrical component | 63%#595% CI 46%–78% | 97%#595% CI 83%–99% | 96%#495% CI 80%–99% | 63%#595% CI 46%–78% | 73%#595% CI 52%–87% | 90%#595% CI 74%–96% | Coming soon | ||
| Pooled | 41%#595% CI 35%–47% | 70%#595% CI 63%–76% | 84%#495% CI 77%–89% | 45%#595% CI 39%–51% | 59%#595% CI 52%–66% | 85%#395% CI 79%–90% | Coming soon | ||
| NominalEvaluated on the placements the model was trained on. | InterpolationEvaluated on unseen placements between the training placements. | ExtrapolationEvaluated on unseen placements outside the training placements' convex hull. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Nominal · D10Trained on ~10 demonstrations per environment. | Nominal · D100Trained on ~100 demonstrations per environment. | Nominal · D300Trained on ~300 demonstrations per environment. | Interpolation · D10Trained on ~10 demonstrations per environment. | Interpolation · D100Trained on ~100 demonstrations per environment. | Interpolation · D300Trained on ~300 demonstrations per environment. | Extrapolation · D10Trained on ~10 demonstrations per environment. | Extrapolation · D100Trained on ~100 demonstrations per environment. | Extrapolation · D300Trained on ~300 demonstrations per environment. |
| Basic skills | |||||||||
| Sort screws into bins | 101 N#395% CI 82 N–119 N | 87 N#395% CI 75 N–99 N | 56 N#295% CI 46 N–66 N | 77 N#395% CI 58 N–97 N | 97 N#495% CI 83 N–112 N | 70 N#495% CI 58 N–82 N | Coming soon | ||
| Datum-corner alignment | 28 N#595% CI 21 N–35 N | 34 N#595% CI 21 N–47 N | 23 N#495% CI 18 N–28 N | 28 N#395% CI 22 N–35 N | 32 N#595% CI 22 N–42 N | 28 N#395% CI 20 N–36 N | Coming soon | ||
| Pour screws into the funnel | 47 N#595% CI 30 N–64 N | 25 N#395% CI 17 N–32 N | 24 N#495% CI 16 N–31 N | 37 N#595% CI 21 N–53 N | 19 N#295% CI 15 N–23 N | 20 N#195% CI 14 N–25 N | Coming soon | ||
| High precision | |||||||||
| Connect DC jack | 136 N#495% CI 128 N–145 N | 80 N#495% CI 68 N–91 N | 67 N#295% CI 57 N–76 N | 131 N#395% CI 119 N–143 N | 78 N#395% CI 69 N–88 N | 55 N#195% CI 50 N–60 N | Coming soon | ||
| Route cable through closed hoops | 103 N#595% CI 87 N–118 N | 33 N#195% CI 27 N–40 N | 27 N#195% CI 21 N–34 N | 57 N#395% CI 42 N–72 N | 30 N#295% CI 23 N–38 N | 25 N#195% CI 21 N–30 N | Coming soon | ||
| Put tools in standing toolbox | 135 N#595% CI 120 N–151 N | 123 N#595% CI 112 N–135 N | 113 N#495% CI 101 N–124 N | 138 N#595% CI 125 N–152 N | 131 N#595% CI 122 N–141 N | 103 N#495% CI 91 N–115 N | Coming soon | ||
| Bimanual | |||||||||
| Place loaded boxes in container (bimanual) | 78 N#495% CI 61 N–94 N | 37 N#295% CI 33 N–41 N | 37 N#295% CI 30 N–43 N | 76 N#495% CI 62 N–91 N | 37 N#195% CI 31 N–43 N | 30 N#195% CI 27 N–33 N | Coming soon | ||
| Place spray bottles upright, nozzle forward (bimanual handover) | 65 N#595% CI 48 N–82 N | 36 N#495% CI 28 N–45 N | 36 N#495% CI 31 N–41 N | 76 N#495% CI 61 N–92 N | 33 N#495% CI 27 N–40 N | 28 N#395% CI 23 N–33 N | Coming soon | ||
| Tool use | |||||||||
| Open the toolbox with a screwdriver | 79 N#595% CI 63 N–94 N | 46 N#495% CI 38 N–54 N | 56 N#495% CI 48 N–64 N | 88 N#595% CI 74 N–103 N | 49 N#395% CI 41 N–57 N | 46 N#495% CI 37 N–54 N | Coming soon | ||
| Clamp electrical component | 74 N#595% CI 53 N–94 N | 19 N#195% CI 18 N–21 N | 21 N#295% CI 20 N–23 N | 79 N#595% CI 59 N–99 N | 22 N#295% CI 19 N–25 N | 20 N#195% CI 18 N–21 N | Coming soon | ||
| Pooled | 84 N#595% CI 78 N–91 N | 52 N#495% CI 48 N–57 N | 46 N#395% CI 42 N–50 N | 79 N#595% CI 73 N–85 N | 53 N#495% CI 48 N–58 N | 42 N#295% CI 39 N–46 N | Coming soon | ||
Data Scaling
Spatial Generalization
Per Environment (SuccessProgressExecution qualityAvg execution timeExecution speedSmoothnessAvg jerkSafe failure rateMax contact force)
Training parameters
- Starting checkpoint
- microsoft/rho-fr3-duo
- Frozen part
- Vision-language model
- Total batch size
- 64
- Learning rate
- 1e-4
- Schedule
- Warmup, then cosine to 0.025× peak
- Action chunk
- 50
- Action space
- Relative TCP pose
Rollouts
Mid-training dataM0
Task-specific dataD300
SpatialNominal
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
Sort screws into bins - Success
- 73%
- Progress
- 86%
Pour screws into the funnel - Success
- 90%
- Progress
- 91%
Connect DC jack - Success
- 37%
- Progress
- 66%
Route cable through closed hoops - Success
- 7%
- Progress
- 17%
Open the toolbox with a screwdriver - Success
- 87%
- Progress
- 93%
- Held-out benchmark environment
Datum-corner alignment - Success
- 57%
- Progress
- 57%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 17%
- Progress
- 46%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 63%
- Progress
- 91%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 83%
- Progress
- 91%
- Held-out benchmark environment
Clamp electrical component - Success
- 20%
- Progress
- 66%
Sort screws into bins - Success
- 17%
- Progress
- 67%
Pour screws into the funnel - Success
- 87%
- Progress
- 89%
Connect DC jack - Success
- 30%
- Progress
- 67%
Route cable through closed hoops - Success
- 7%
- Progress
- 23%
Open the toolbox with a screwdriver - Success
- 90%
- Progress
- 93%
- Held-out benchmark environment
Datum-corner alignment - Success
- 63%
- Progress
- 63%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 10%
- Progress
- 46%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 60%
- Progress
- 90%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 87%
- Progress
- 89%
- Held-out benchmark environment
Clamp electrical component - Success
- 3%
- Progress
- 66%
Sort screws into bins - Success
- 70%
- Progress
- 85%
Pour screws into the funnel - Success
- 70%
- Progress
- 78%
Connect DC jack - Success
- 30%
- Progress
- 65%
Route cable through closed hoops - Success
- 7%
- Progress
- 11%
Open the toolbox with a screwdriver - Success
- 50%
- Progress
- 70%
- Held-out benchmark environment
Datum-corner alignment - Success
- 67%
- Progress
- 67%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 3%
- Progress
- 37%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 67%
- Progress
- 92%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 80%
- Progress
- 96%
- Held-out benchmark environment
Clamp electrical component - Success
- 3%
- Progress
- 67%
Sort screws into bins - Success
- 3%
- Progress
- 51%
Pour screws into the funnel - Success
- 60%
- Progress
- 69%
Connect DC jack - Success
- 27%
- Progress
- 68%
Route cable through closed hoops - Success
- 0%
- Progress
- 8%
Open the toolbox with a screwdriver - Success
- 43%
- Progress
- 73%
- Held-out benchmark environment
Datum-corner alignment - Success
- 60%
- Progress
- 60%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 3%
- Progress
- 28%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 67%
- Progress
- 92%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 80%
- Progress
- 93%
- Held-out benchmark environment
Clamp electrical component - Success
- 27%
- Progress
- 71%
Sort screws into bins - Success
- 7%
- Progress
- 22%
Pour screws into the funnel - Success
- 13%
- Progress
- 22%
Connect DC jack - Success
- 0%
- Progress
- 11%
Route cable through closed hoops - Success
- 0%
- Progress
- 5%
Open the toolbox with a screwdriver - Success
- 3%
- Progress
- 25%
- Held-out benchmark environment
Datum-corner alignment - Success
- 33%
- Progress
- 33%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 0%
- Progress
- 3%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 10%
- Progress
- 54%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 17%
- Progress
- 53%
- Held-out benchmark environment
Clamp electrical component - Success
- 0%
- Progress
- 34%
Sort screws into bins - Success
- 0%
- Progress
- 6%
Pour screws into the funnel - Success
- 3%
- Progress
- 14%
Connect DC jack - Success
- 0%
- Progress
- 8%
Route cable through closed hoops - Success
- 0%
- Progress
- 8%
Open the toolbox with a screwdriver - Success
- 0%
- Progress
- 23%
- Held-out benchmark environment
Datum-corner alignment - Success
- 37%
- Progress
- 37%
- Held-out benchmark environment
Put tools in standing toolbox - Success
- 0%
- Progress
- 3%
- Held-out benchmark environment
Place loaded boxes in container (bimanual) - Success
- 0%
- Progress
- 53%
- Held-out benchmark environment
Place spray bottles upright, nozzle forward (bimanual handover) - Success
- 20%
- Progress
- 61%
- Held-out benchmark environment
Clamp electrical component - Success
- 0%
- Progress
- 32%