PaperPatch Policy (CLS versus Dense)
Patch Policy: Efficient Embodied Control via Dense Visual Representations
Claim tested: Dense patch features improve real-robot task success over CLS features from the same pretrained visual encoder.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +17 pts
- Ours
- +7 pts
- Gain vs baseline
- 95% CI, −3 pts to +16 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 82%
- Ours
- 14%
- Measured on our stations
- 95% CI, 9%–23%
- Reported in the paper
Across iterations
- Iteration 1vs CLS token+7 pts
- Iteration 2vs CLS token+22 pts
By task
| Task | BaselineCLS token | MethodPatch Policy (CLS versus Dense) | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e02Connect DC jack | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e03Place spray bottles upright, nozzle forward (bimanual handover) | 23%7/30 | 43%13/30 | +20 pts−4 to +41 |
| All tasks | 8%7/90 | 14%13/90 | +7 pts−3 to +16 |
| Reported in the paper | 65% | 82% | +17 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e02Connect DC jack BaselineCLS token0/300%
Show in dataset viewer of CLS token on Connect DC jackMethodPatch Policy (CLS versus Dense)0/300%
Show in dataset viewer of Patch Policy (CLS versus Dense) on Connect DC jack- Held-out benchmark environment
e01Clamp electrical component BaselineCLS token0/300%
Show in dataset viewer of CLS token on Clamp electrical componentMethodPatch Policy (CLS versus Dense)0/300%
Show in dataset viewer of Patch Policy (CLS versus Dense) on Clamp electrical component - Held-out benchmark environment
e03Place spray bottles upright, nozzle forward (bimanual handover) BaselineCLS token7/3023%
Show in dataset viewer of CLS token on Place spray bottles upright, nozzle forward (bimanual handover)MethodPatch Policy (CLS versus Dense)13/3043%
Show in dataset viewer of Patch Policy (CLS versus Dense) on Place spray bottles upright, nozzle forward (bimanual handover)
Implementation Details
What we changed
We compare dense patch features with CLS features to test the paper’s claim that spatially detailed visual features improve manipulation success. Both policies use the same frozen DINOv2 ViT-S/14 encoder, a block-causal transformer and a VQ-BeT action head, and differ only in the visual tokens they read. This first round departs from the paper in the ways listed below, which later rounds brought closer to it.
- Robot state input. Both arms receive the robot state on every token, while the paper’s policies receive visual inputs only.
- Single observation frame. Both arms see only the current frame instead of the paper’s visual observation history.
- Longer chunks. Both arms predict 32 steps instead of 5 or 10, about one second of motion at 30 Hz.
Prompt summary
Test whether dense patch features from a pretrained visual encoder improve real-robot success over CLS features from the same encoder. Implement the method and its control, choose 3 tasks, train in the cloud and evaluate on stations. (Implemented with frozen DINOv2 ViT-S/14, a block-causal transformer and a VQ-BeT head; one observation frame, robot state on every token, 32-step chunks.)
Response summary
SR 7.8% CLS vs 14.4% patch (+6.7 pp, p = 0.18, n.s.) and AP 19.6% vs 42.0% (+22.4 pp, p < 0.001); successes only on spray (7/30 vs 13/30), 0/30 for both arms on clamp and dc_jack.
Conclusion
Dense patches made far more progress but successes were too rare to call. Next: tighten the setup to the paper (images only, observation history, reactive replanning, larger tokenizer).