PaperFACT
Demystifying When and Why VLAs Fail in Contact-Rich Tasks and How to Fix Them
Claim tested: The paper's logit-normal post-training schedule improves π0.5 task success in contact-rich manipulation.
Tested
Result
Improvement over baseline
Inconclusive
- Reported
- +17 pts
- Ours
- +4 pts
- Gain vs baseline
- 95% CI, −9 pts to +17 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 56%
- Ours
- 31%
- Measured on our stations
- 95% CI, 22%–41%
- Reported in the paper
Across iterations
- Iteration 1vs π0.5−2 pts
- Iteration 2vs π0.5+4 pts
By task
| Task | Baselineπ0.5 | MethodFACT | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 7%2/30 | 0%0/30 | −7 pts−21 to +6 |
| e02Connect DC jack | 10%3/30 | 50%15/30 | +40 pts+17 to +58 |
| e03Datum-corner alignment | 63%19/30 | 43%13/30 | −20 pts−42 to +5 |
| All tasks | 27%24/90 | 31%28/90 | +4 pts−9 to +17 |
| Reported in the paper | 39% | 56% | +17 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e02Connect DC jack Baselineπ0.53/3010%
Show in dataset viewer of π0.5 on Connect DC jackMethodFACT15/3050%
Show in dataset viewer of FACT on Connect DC jack- Held-out benchmark environment
e01Clamp electrical component Baselineπ0.52/307%
Show in dataset viewer of π0.5 on Clamp electrical componentMethodFACT0/300%
Show in dataset viewer of FACT on Clamp electrical component - Held-out benchmark environment
e03Datum-corner alignment Baselineπ0.519/3063%
Show in dataset viewer of π0.5 on Datum-corner alignmentMethodFACT13/3043%
Show in dataset viewer of FACT on Datum-corner alignment
Implementation Details
What we changed
We compare the paper’s logit-normal post-training schedule with pi0.5’s native Beta(1.5, 1) sampler, without the paper’s force module. Both arms predict the paper’s delta actions, so the near-contact delta collapse the paper builds on is present. The arms differ only in the training timestep law; the logit-normal arm uses location 1.5 and scale 1 as in the official code, and inference is unchanged. Differences from the paper:
- Delta actions on our platform. Both arms predict position and rotation relative to the current state, with the gripper absolute, as in the official code. Normalization statistics are fitted on these deltas.
- Full fine-tuning. Both arms fine-tune all weights with a frozen vision encoder instead of the paper’s LoRA.
- Platform rate and budget. Both arms use 30 Hz control, 32-action chunks and the platform’s 3-epoch step budget with a 10,000-step minimum, which gives 10,000 to 17,250 steps per task instead of the paper’s 20,000.
- GPUs. connect_dc_jack trained on 2 GPUs with effective batch 128, the other tasks on 1 GPU with batch 64.
Prompt summary
Relaunch the same claim, keeping both effects the paper's gain rests on: delta end-effector actions collapsing near contact, and the Beta(1.5, 1) schedule starving training below tau = 0.2. A run that removes either effect does not test the claim. (Both arms switched to delta actions with normalization fitted on deltas; same three tasks with fresh dataset exports.)
Response summary
SR 26.7% vs 31.1% (+4.4 pp, p = 0.58) and AP 53.8% vs 40.9% (-12.9 pp, p = 0.035); LN helped dc_jack insertion strongly (3/30 to 15/30, p = 0.004) but regressed clamp (AP -41 pp) and push_box (19/30 to 13/30).
Conclusion
Reproduced on the high-precision insertion (dc jack 3/30 → 15/30, p = 0.004). As claimed, the LN schedule helps in high-precision tasks, but it seems to hurt tasks with larger free-space motions (clamp grasping: progress −41 pp; push box: 19/30 → 13/30). Concluded.