PaperActionMap (full data)
ActionMap: Robot Policy Learning via Voxel Action Heatmap
Claim tested: ActionMap's voxel action heatmap head improves real-robot task success over OpenVLA-OFT's standard action regression head.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +43 pts
- Ours
- +6 pts
- Gain vs baseline
- 95% CI, −9 pts to +20 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 67%
- Ours
- 49%
- Measured on our stations
- 95% CI, 39%–59%
- Reported in the paper
Across iterations
By task
| Task | BaselineOpenVLA-OFT | MethodActionMap (full data) | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 3%1/30 | +3 pts−8 to +17 |
| e02Pour screws into the funnel | 73%22/30 | 77%23/30 | +3 pts−18 to +25 |
| e03Datum-corner alignment | 57%17/30 | 67%20/30 | +10 pts−14 to +32 |
| All tasks | 43%39/90 | 49%44/90 | +6 pts−9 to +20 |
| Reported in the paper | 23% | 67% | +43 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e02Pour screws into the funnel BaselineOpenVLA-OFT22/3073%
Show in dataset viewer of OpenVLA-OFT on Pour screws into the funnelMethodActionMap (full data)23/3077%
Show in dataset viewer of ActionMap (full data) on Pour screws into the funnel- Held-out benchmark environment
e01Clamp electrical component BaselineOpenVLA-OFT0/300%
Show in dataset viewer of OpenVLA-OFT on Clamp electrical componentMethodActionMap (full data)1/303%
Show in dataset viewer of ActionMap (full data) on Clamp electrical component - Held-out benchmark environment
e03Datum-corner alignment BaselineOpenVLA-OFT17/3057%
Show in dataset viewer of OpenVLA-OFT on Datum-corner alignmentMethodActionMap (full data)20/3067%
Show in dataset viewer of ActionMap (full data) on Datum-corner alignment
Implementation Details
What we changed
We replaced only OpenVLA-OFT’s L1 regression head with the paper’s voxel heatmap head, and trained both on each task’s full demonstration set with the paper’s real-robot recipe. Because OFT’s custom transformers fork does not run on our stack, we rebuilt OpenVLA-7B on a newer transformers release; both heads follow the official code. This round re-evaluates the same models on the station’s blend runtime instead of the synchronous one, on the same stations and placements. Differences from the paper:
- Two arms and three cameras. Each step predicts 14 action values instead of 7, and ActionMap has one heatmap branch per arm on a shared trunk.
- Delta actions on absolute targets. Both heads predict per-step deltas between consecutive commands, as in the paper, and integrate them from the measured pose. The first step is taken from the measured pose and normalized separately, because this robot’s commands lead its measured pose by up to 20 cm.
- Training budget. Both heads train for 4,000 steps at a constant learning rate of 5e-4 and batch 64, without a validation split. On clamp, whose episodes are longest, this covers only 0.8 passes over the data.
- Chunk timing. Chunks are 32 steps at 30 Hz, about 1.07 s.
- Blend execution. The station requests each next chunk 800 ms before the current one ends and crossfades into it over 600 ms, instead of executing each chunk in full before replanning as OFT does.
- Evaluation. Each task runs 30 placements with one trial each, instead of the paper’s 10 trials.
- Hardware. All six models trained on four A100 40 GB GPUs, with two micro-batches of 8 per GPU.
Prompt summary
Re-evaluate the six existing models on the blend runtime (AIS blend 85mm) in a new project, with everything else unchanged. (A retraining step with a longer schedule, decaying LR and a validation split was started and then discarded.)
Response summary
On blend, L1 vs ActionMap: pour 22/30 vs 23/30, push 17/30 vs 20/30, clamp 0/30 vs 1/30; 39/90 vs 44/90 (+5.6 pp, 95% CI −5.6 to +16.7, p = 0.42).
Conclusion
ActionMap slightly ahead on blend (44 vs 39/90), within noise. Concluded: not reproduced.