Reproducing Robotics Research with AI Agents
We operate a large-scale robot farm with real-world benchmarking stations, simulation stations and dedicated on-site teams. Our agent-native platform gives AI agents API access to datasets, model training and experiments in both simulation and the physical world, while our team handles station preparation, resets and quality control.
Research replication puts this infrastructure to work: agents implement methods from papers and test selected claims on real robots, with human review and steering where needed.
See the workflow below, then explore how the improvements measured in our setup compare with those reported in each paper. Open any experiment for detailed results, adaptations and the agent’s trajectory.
How it works
AI agents run the workflow through our platform, with human review and steering where needed.
- 1. Identify the claimThe agent or a human identifies the claim to test and the paper’s experimental setup.
- 2. Select tasksThe agent selects suitable tasks from our farm and documents differences from the original setup.
- 3. Implement the methodThe agent uses the authors’ code where available or implements the method, documenting adaptations.
- 4. Train the modelsThe agent trains the baseline and proposed method, recording settings and departures from the paper.
- 5. Test on real robotsThe agent requests evaluations on matched object placements, then analyzes the results and uncertainty.
Papers
| Open | |||||
|---|---|---|---|---|---|
CUPIDvs Random filteringInconclusive | +8 pts95% CI −9 to +25Paper: +36 pts | Inconclusive | |||
OATvs QueSTNot replicated | −16 pts95% CI −28 to −2Paper: +33 pts | Not replicated | |||
Open-Loop Executionvs Short context, long executionInconclusive | +6 pts95% CI −7 to +18Paper: +10 pts | Inconclusive | |||
SMILEvs SmolVLAInconclusive | +6 pts95% CI −4 to +16Paper: +11 pts | Inconclusive | |||
ActionMap (50 demos per task)vs OpenVLA-OFTNot replicated | +2 pts95% CI −7 to +12Paper: +33 pts | Not replicated | |||
ActionMap (full data)vs OpenVLA-OFTNot replicated | +6 pts95% CI −9 to +20Paper: +43 pts | Not replicated | |||
DSDvs Uniform action binningReplicated | +13 pts95% CI +2 to +24Paper: +13 pts | Replicated | |||
ManiFlowvs Diffusion PolicyInconclusive | +8 pts95% CI −7 to +22Paper: +17 pts | Inconclusive | |||
Patch Policy (CLS versus Dense)vs CLS tokenReplicated | +22 pts95% CI +12 to +32Paper: +17 pts | Replicated | |||
CSPvs Diffusion PolicyNot replicated | +2 pts95% CI −5 to +9Paper: +62 pts | Not replicated | |||
FACTvs π0.5Inconclusive | +4 pts95% CI −9 to +17Paper: +17 pts | Inconclusive |
- Gain vs baseline
- 95% CI
- Paper's reported gain
- No difference
| Open | |||||
|---|---|---|---|---|---|
CUPIDAgia et al., CoRL 2025 | 84% | 48% | |||
OATLiu et al., RSS 2026 | 80% | 23% | |||
Open-Loop ExecutionZeng et al. | 88% | 45% | |||
SMILEPark et al. | 80% | 21% | |||
ActionMap (50 demos per task)Yang et al. | 47% | 12% | |||
ActionMap (full data)Yang et al. | 67% | 49% | |||
DSDDuan et al. | 73% | 45% | |||
ManiFlowYan et al., CoRL 2025 | 57% | 51% | |||
Patch Policy (CLS versus Dense)Zhou et al. | 82% | 26% | |||
CSPCao et al., ICML 2026 | 86% | 6% | |||
FACTParés-Morlans et al. | 56% | 31% |
- Measured on our stations
- 95% CI
- Reported in the paper