PaperFACT
Demystifying When and Why VLAs Fail in Contact-Rich Tasks and How to Fix Them
Claim tested: The paper's logit-normal post-training schedule improves π0.5 task success in contact-rich manipulation.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +17 pts
- Ours
- −2 pts
- Gain vs baseline
- 95% CI, −12 pts to +8 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 56%
- Ours
- 11%
- Measured on our stations
- 95% CI, 6%–19%
- Reported in the paper
Across iterations
- Iteration 1vs π0.5−2 pts
- Iteration 2vs π0.5+4 pts
By task
| Task | Baselineπ0.5 | MethodFACT | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 3%1/30 | 0%0/30 | −3 pts−17 to +8 |
| e02Connect DC jack | 7%2/30 | 3%1/30 | −3 pts−18 to +11 |
| e03Datum-corner alignment | 30%9/30 | 30%9/30 | 0 pts−22 to +22 |
| All tasks | 13%12/90 | 11%10/90 | −2 pts−12 to +8 |
| Reported in the paper | 39% | 56% | +17 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e02Connect DC jack Baselineπ0.52/307%
Show in dataset viewer of π0.5 on Connect DC jackMethodFACT1/303%
Show in dataset viewer of FACT on Connect DC jack- Held-out benchmark environment
e01Clamp electrical component Baselineπ0.51/303%
Show in dataset viewer of π0.5 on Clamp electrical componentMethodFACT0/300%
Show in dataset viewer of FACT on Clamp electrical component - Held-out benchmark environment
e03Datum-corner alignment Baselineπ0.59/3030%
Show in dataset viewer of π0.5 on Datum-corner alignmentMethodFACT9/3030%
Show in dataset viewer of FACT on Datum-corner alignment
Implementation Details
What we changed
We compare the paper’s logit-normal post-training schedule with pi0.5’s native Beta(1.5, 1) sampler, without the paper’s force module. The arms differ only in the training timestep law; the logit-normal arm uses location 1.5 and scale 1 as in the official code, and inference is unchanged. Differences from the paper:
- Absolute pose actions. Both arms predict absolute end-effector poses instead of the paper’s delta actions, so the near-contact delta collapse the paper builds on is absent and only its training-starvation argument applies.
- Full fine-tuning. Both arms fine-tune all weights with a frozen vision encoder instead of the paper’s LoRA.
- Platform rate and budget. Both arms use 30 Hz control, 32-action chunks and the platform’s 3-epoch step budget at batch 64 with a 10,000-step minimum, which gives 10,000 to 26,759 steps per task instead of the paper’s 20,000.
Prompt summary
Test whether the paper's logit-normal (LN) post-training timestep schedule improves pi0.5 task success in contact-rich manipulation against pi0.5's default Beta(1.5, 1) schedule, on 3 tasks of your choice. (Implemented as a pi05_ln preset; everything else at the platform baseline: full fine-tuning, absolute TCP-pose actions, no force input.)
Response summary
Not confirmed: SR 13.3% Beta vs 11.1% LN (-2.2 pp, p = 0.57) and AP 32.7% vs 40.6% (+7.8 pp, p = 0.07), with the AP gain coming almost entirely from clamp partial progress.
Conclusion
No gain on success. The setup removed the two effects the paper's gain depends on (delta actions, Beta schedule starving low noise), so the next iteration restores them.