PaperOpen-Loop Execution
Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies
Claim tested: Longer observation histories combined with shorter open-loop execution improve real-robot task success over short-history, long-execution Diffusion Policies, with the action prediction horizon held fixed.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +10 pts
- Ours
- +1 pts
- Gain vs baseline
- 95% CI, −9 pts to +10 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 88%
- Ours
- 22%
- Measured on our stations
- 95% CI, 16%–29%
- Reported in the paper
Across iterations
- Iteration 1vs Short context, long execution+1 pts
- Iteration 2vs Short context, long execution−1 pts
- Iteration 3vs Short context, long execution+6 pts
By task
| Task | BaselineShort context, long execution | MethodOpen-Loop Execution | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e02Connect DC jack | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e03Pour screws into the funnel | 80%24/30 | 83%25/30 | +3 pts−17 to +23 |
| e04Datum-corner alignment | 27%8/30 | 27%8/30 | 0 pts−22 to +22 |
| e05Route cable through closed hoops | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| All tasks | 21%32/150 | 22%33/150 | +1 pts−9 to +10 |
| Reported in the paper | 78% | 88% | +10 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e02Connect DC jack BaselineShort context, long execution0/300%
Show in dataset viewer of Short context, long execution on Connect DC jackMethodOpen-Loop Execution0/300%
Show in dataset viewer of Open-Loop Execution on Connect DC jacke03Pour screws into the funnel BaselineShort context, long execution24/3080%
Show in dataset viewer of Short context, long execution on Pour screws into the funnelMethodOpen-Loop Execution25/3083%
Show in dataset viewer of Open-Loop Execution on Pour screws into the funnele05Route cable through closed hoops BaselineShort context, long execution0/300%
Show in dataset viewer of Short context, long execution on Route cable through closed hoopsMethodOpen-Loop Execution0/300%
Show in dataset viewer of Open-Loop Execution on Route cable through closed hoops- Held-out benchmark environment
e01Clamp electrical component BaselineShort context, long execution0/300%
Show in dataset viewer of Short context, long execution on Clamp electrical componentMethodOpen-Loop Execution0/300%
Show in dataset viewer of Open-Loop Execution on Clamp electrical component - Held-out benchmark environment
e04Datum-corner alignment BaselineShort context, long execution8/3027%
Show in dataset viewer of Short context, long execution on Datum-corner alignmentMethodOpen-Loop Execution8/3027%
Show in dataset viewer of Open-Loop Execution on Datum-corner alignment
Implementation Details
What we changed
We reproduced the paper’s real-robot comparison: a Diffusion Policy with 2 history frames that executes 0.8 s of each plan, against one with 8 history frames that replans every 0.2 s. Both arms predict 1.5 s of actions, plus the past actions in their history window, and use the same network, so they differ only in history length and execution horizon. Adaptations to our platform:
- Both arms use the platform’s Diffusion Policy with a ResNet-18 encoder and a FiLM-conditioned U-Net at the paper’s widths, instead of the paper’s DINOv3 encoders with a cross-attention U-Net and double encoder.
- Actions are absolute end-effector poses instead of the paper’s delta actions.
- Actions run at 30 Hz instead of 10 Hz, with all horizons matched in seconds.
- Both arms run on the station’s synchronous runtime, and the server smooths each plan switch over about 150 ms, mirroring the paper’s 180 ms interpolation.
- Each arm serves its final checkpoint, because the cloud trainer kept only that one, instead of selecting by validation error as the paper does.
- Training uses a 95% random image crop, a batch of 64 and about 2% of episodes held out.
- Evaluation covers 5 tasks with 30 paired placements per arm.
Prompt summary
Test arXiv 2608.15938's claim that a longer observation history with shorter open-loop execution beats a short-history, long-execution Diffusion Policy at a fixed prediction horizon, on 5 real-robot tasks. (Control To=2: 2 frames, 0.8 s execution; method To=8: 8 frames, 0.2 s execution; both predicting 1.5 s, ResNet-18/FiLM, sync runtime.)
Response summary
There was no detectable benefit. Pour scored 24/30 vs 25/30, push 8/30 vs 8/30, and clamp, dc jack and cable 0/30 in both arms; the 5-task equal-weight SR difference was +0.7 pp (95% CI -4.7 to +6.7, p=1.0) and AP +1.4 pp (CI -4.3 to +7.1, p=0.65).
Conclusion
No detectable benefit, but the architecture and actions differed from the paper. Next: port the authors' network, delta actions and tasks with headroom.