Poke &Wiggle
Papers
Paper

Open-Loop Execution

Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies

Claim tested: Longer observation histories combined with shorter open-loop execution improve real-robot task success over short-history, long-execution Diffusion Policies, with the action prediction horizon held fixed.

Tested

Paper
arXiv⁠ (opens in a new tab)
Code
GitHub⁠ (opens in a new tab)
Venue
—
Authors
Michael Zeng et al.
Farm tasks
7
Evaluation
780 episodes (390 for baseline, 390 for method)

Result

Improvement over baseline

Not replicated

Reported
+10 pts
Ours
+1 pts
−200+20+40+60
  • Gain vs baseline
  • 95% CI, −9 pts to +10 pts
  • Paper's reported gain
  • No difference

Success rate

Reported
88%
Ours
22%
0%20%40%60%80%100%
  • Measured on our stations
  • 95% CI, 16%–29%
  • Reported in the paper

Across iterations

  1. Iteration 1vs Short context, long execution+1 pts
  2. Iteration 2vs Short context, long execution−1 pts
  3. Iteration 3vs Short context, long execution+6 pts
−200+20+40+60

By task

TaskBaselineShort context, long executionMethodOpen-Loop ExecutionImprovement
e01Clamp electrical component0%0/300%0/300 pts−11 to +11
e02Connect DC jack0%0/300%0/300 pts−11 to +11
e03Pour screws into the funnel80%24/3083%25/30+3 pts−17 to +23
e04Datum-corner alignment27%8/3027%8/300 pts−22 to +22
e05Route cable through closed hoops0%0/300%0/300 pts−11 to +11
All tasks21%32/15022%33/150+1 pts−9 to +10
Reported in the paper78%88%+10 pts

Rollouts

One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.

  1. e02Connect DC jack
  2. e03Pour screws into the funnel
  3. e05Route cable through closed hoops
  4. Held-out benchmark environment
    e01Clamp electrical component
  5. Held-out benchmark environment
    e04Datum-corner alignment

Implementation Details

What we changed

We reproduced the paper’s real-robot comparison: a Diffusion Policy with 2 history frames that executes 0.8 s of each plan, against one with 8 history frames that replans every 0.2 s. Both arms predict 1.5 s of actions, plus the past actions in their history window, and use the same network, so they differ only in history length and execution horizon. Adaptations to our platform:

  • Both arms use the platform’s Diffusion Policy with a ResNet-18 encoder and a FiLM-conditioned U-Net at the paper’s widths, instead of the paper’s DINOv3 encoders with a cross-attention U-Net and double encoder.
  • Actions are absolute end-effector poses instead of the paper’s delta actions.
  • Actions run at 30 Hz instead of 10 Hz, with all horizons matched in seconds.
  • Both arms run on the station’s synchronous runtime, and the server smooths each plan switch over about 150 ms, mirroring the paper’s 180 ms interpolation.
  • Each arm serves its final checkpoint, because the cloud trainer kept only that one, instead of selecting by validation error as the paper does.
  • Training uses a 95% random image crop, a batch of 64 and about 2% of episodes held out.
  • Evaluation covers 5 tasks with 30 paired placements per arm.

See the code changes on GitHub⁠ (opens in a new tab)

Prompt summary

Test arXiv 2608.15938's claim that a longer observation history with shorter open-loop execution beats a short-history, long-execution Diffusion Policy at a fixed prediction horizon, on 5 real-robot tasks. (Control To=2: 2 frames, 0.8 s execution; method To=8: 8 frames, 0.2 s execution; both predicting 1.5 s, ResNet-18/FiLM, sync runtime.)

Response summary

There was no detectable benefit. Pour scored 24/30 vs 25/30, push 8/30 vs 8/30, and clamp, dc jack and cable 0/30 in both arms; the 5-task equal-weight SR difference was +0.7 pp (95% CI -4.7 to +6.7, p=1.0) and AP +1.4 pp (CI -4.3 to +7.1, p=0.65).

Conclusion

No detectable benefit, but the architecture and actions differed from the paper. Next: port the authors' network, delta actions and tasks with headroom.