Poke &Wiggle
Papers
Paper

Open-Loop Execution

Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies

Claim tested: Longer observation histories combined with shorter open-loop execution improve real-robot task success over short-history, long-execution Diffusion Policies, with the action prediction horizon held fixed.

Tested

Paper
arXiv⁠ (opens in a new tab)
Code
GitHub⁠ (opens in a new tab)
Venue
—
Authors
Michael Zeng et al.
Farm tasks
7
Evaluation
780 episodes (390 for baseline, 390 for method)

Result

Improvement over baseline

Inconclusive

Reported
+10 pts
Ours
−1 pts
−200+20+40+60
  • Gain vs baseline
  • 95% CI, −13 pts to +11 pts
  • Paper's reported gain
  • No difference

Success rate

Reported
88%
Ours
39%
0%20%40%60%80%100%
  • Measured on our stations
  • 95% CI, 31%–48%
  • Reported in the paper

Across iterations

  1. Iteration 1vs Short context, long execution+1 pts
  2. Iteration 2vs Short context, long execution−1 pts
  3. Iteration 3vs Short context, long execution+6 pts
−200+20+40+60

By task

TaskBaselineShort context, long executionMethodOpen-Loop ExecutionImprovement
e01Open the toolbox with a screwdriver10%3/303%1/30−7 pts−23 to +8
e02Place loaded boxes in container (bimanual)33%10/3030%9/30−3 pts−26 to +19
e03Pour screws into the funnel83%25/3080%24/30−3 pts−23 to +17
e04Datum-corner alignment33%10/3043%13/30+10 pts−14 to +32
All tasks40%48/12039%47/120−1 pts−13 to +11
Reported in the paper78%88%+10 pts

Rollouts

One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.

  1. e01Open the toolbox with a screwdriver
  2. e03Pour screws into the funnel
  3. Held-out benchmark environment
    e02Place loaded boxes in container (bimanual)
  4. Held-out benchmark environment
    e04Datum-corner alignment

Implementation Details

What we changed

We reproduced the paper’s real-robot comparison: a Diffusion Policy with 2 history frames that executes 0.8 s of each plan, against one with 8 history frames that replans every 0.3 s. Both arms predict 1.5 s of actions, plus the past actions in their history window. We ported the authors’ DINOv3 encoder, cross-attention U-Net and double encoder from their released code, and verified exact numerical parity with it. Both arms use this architecture and the same training recipe, so they differ only in history length and execution horizon. Adaptations to our platform:

  • The method replans every 0.3 s instead of 0.2 s, because observation delay and inference together take about 220 ms on our system.
  • Both arms run asynchronously on the station’s blend runtime, which crossfades into each new plan over 400 ms. At this latency the paper’s 180 ms interpolation never takes effect. Image embeddings are cached between replans.
  • Actions run at 30 Hz instead of 10 Hz, with all horizons matched in seconds.
  • Actions are per-step end-effector pose deltas anchored to the robot’s commanded pose, with absolute gripper widths. The commanded pose is also a model input. In training the anchor is the previous recorded action, because our datasets do not store the commanded pose. The paper does not specify its delta reference.
  • The control uses the method’s architecture and is trained for 200k steps, instead of the paper’s simpler single-encoder baseline trained for 400k steps. Both arms train at batch 48 with the authors’ EMA.
  • Checkpoints are selected by validation DDIM MSE among the 5 best, without hardware screening.
  • Evaluation covers 4 tasks with 30 paired placements per arm.

See the code changes on GitHub⁠ (opens in a new tab)

Prompt summary

Port the authors' DINOv3 double-encoder cross-attention U-Net, keep the top-5 validation checkpoints, switch to push, toolbox, bimanual boxes and pour, predict 30 Hz delta actions, and follow the authors' EMA, batch 48 and 200k steps. Anchor deltas to the target (previous action in training, live desired TCP at serving), add the target pose as an input, and retrain all 8 models on one revision (the sync evals of these steps were discarded or cancelled). Then evaluate both arms with async inference and blend execution on an existing runtime, replanning the method arm every 9 steps (0.3 s) because of system latency, with the first-3-episode stop rule.

Response summary

SR was push 10/30 vs 13/30, toolbox 3/30 vs 1/30, boxes 10/30 vs 9/30 and pour 26/32 vs 24/30. The 4-task equal-weight paired SR difference was -0.4 pp (95% CI -9.2 to +8.3, p=1.0) and AP -3.5 pp (CI -10.9 to +3.8, p=0.38). Toolbox AP was 41% vs 21% (-19.6 pp, p=0.008).

Conclusion

On par without real-time chunking (−0.4 pp), and the long-history arm made less progress on toolbox. The arms differed little in reaction time because of latency and blending, which motivated adding RTC.