PaperOAT
OAT: Ordered Action Tokenization
Claim tested: OAT improves real-robot task success over QueST with full-token decoding.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +33 pts
- Ours
- +3 pts
- Gain vs baseline
- 95% CI, −2 pts to +10 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 80%
- Ours
- 4%
- Measured on our stations
- 95% CI, 2%–11%
- Reported in the paper
Across iterations
- Iteration 1vs QueST+3 pts
- Iteration 2vs QueST−8 pts
- Iteration 3vs QueST−16 pts
By task
| Task | BaselineQueST | MethodOAT | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e02Place loaded boxes in container (bimanual) | 3%1/30 | 10%3/30 | +7 pts−8 to +23 |
| e03Pour screws into the funnel | 0%0/30 | 3%1/30 | +3 pts−8 to +17 |
| All tasks | 1%1/90 | 4%4/90 | +3 pts−2 to +10 |
| Reported in the paper | 48% | 80% | +33 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e03Pour screws into the funnel BaselineQueST0/300%
Show in dataset viewer of QueST on Pour screws into the funnelMethodOAT1/303%
Show in dataset viewer of OAT on Pour screws into the funnel- Held-out benchmark environment
e01Clamp electrical component BaselineQueST0/300%
Show in dataset viewer of QueST on Clamp electrical componentMethodOAT0/300%
Show in dataset viewer of OAT on Clamp electrical component - Held-out benchmark environment
e02Place loaded boxes in container (bimanual)
Implementation Details
What we changed
We implemented OAT and QueST on the paper’s shared autoregressive backbone, ported from the authors’ code. The two arms differ only in the action tokenizer, and both decode all 8 tokens over the paper’s 1,000-entry codebook. Each tokenizer is pretrained and then frozen, and both policies train with the reference’s effective batch size and learning rate. Differences from the paper:
- One observation frame. Both arms see one frame instead of two, because station serving sends one observation per request.
- Two arms and three cameras. Actions are 20-D absolute end-effector poses for two arms at 30 Hz with three cameras, instead of a 7-D single arm with one webcam.
- Larger crop. Random crops keep 116 of 128 pixels instead of 76.
- Fixed training budget. Both arms train for 60 epochs, at least 15,000 steps, and serve the final checkpoint, instead of selecting checkpoints by simulated success.
- QueST loss. QueST’s reconstruction loss is computed on normalized actions.
Prompt summary
Test whether OAT (ordered action tokenization) improves real-robot success over QueST with full-token decoding. Implement both arms, choose 3 tasks, train in the cloud, evaluate on stations and compare. (Ported both tokenizers and the shared autoregressive backbone from the official code: 8 tokens, FSQ [8,5,5,5], ResNet-18 from scratch, absolute actions, sync runtime.)
Response summary
No significant difference and both arms near zero: SR QueST 1.1% vs OAT 4.4% (pour 0/30 vs 1/30, clamp 0/30 vs 0/30, boxes 1/30 vs 3/30), AP 25.7% vs 24.0%, paired SR delta +3.3 pp (95% CI -1.7 to +8.4).
Conclusion
Both arms near zero, so no comparison was possible. Next: pretrained encoders, then a held-out split and blend execution to get the policies off the floor.