PaperCUPID
CUPID: Curating Data your Robot Loves with Influence Functions
Claim tested: CUPID's performance-influence filtering of existing demonstrations improves real-robot task success over random filtering.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +36 pts
- Ours
- −14 pts
- Gain vs baseline
- 95% CI, −27 pts to −1 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 84%
- Ours
- 23%
- Measured on our stations
- 95% CI, 16%–33%
- Reported in the paper
Across iterations
- Iteration 1vs Random filtering−11 pts
- Iteration 2vs Random filtering−14 pts
- Iteration 3vs Random filtering+8 pts
By task
| Task | BaselineRandom filtering | MethodCUPID | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 3%1/30 | +3 pts−8 to +17 |
| e02Place loaded boxes in container (bimanual) | 60%18/30 | 23%7/30 | −37 pts−56 to −12 |
| e03Pour screws into the funnel | 53%16/30 | 43%13/30 | −10 pts−33 to +15 |
| All tasks | 38%34/90 | 23%21/90 | −14 pts−27 to −1 |
| Reported in the paper | 48% | 84% | +36 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e03Pour screws into the funnel BaselineRandom filtering16/3053%
Show in dataset viewer of Random filtering on Pour screws into the funnelMethodCUPID13/3043%
Show in dataset viewer of CUPID on Pour screws into the funnel- Held-out benchmark environment
e01Clamp electrical component BaselineRandom filtering0/300%
Show in dataset viewer of Random filtering on Clamp electrical componentMethodCUPID1/303%
Show in dataset viewer of CUPID on Clamp electrical component - Held-out benchmark environment
e02Place loaded boxes in container (bimanual) BaselineRandom filtering18/3060%
Show in dataset viewer of Random filtering on Place loaded boxes in container (bimanual)
Implementation Details
What we changed
We fixed the scoring bug that withdrew the previous round, rescored the demonstrations of all three tasks (pouring screws, placing boxes and clamping a component) and retrained only the CUPID arm. CUPID scores every demonstration from 30 rollouts of a base Diffusion Policy trained on all demonstrations and keeps the top-scoring half. The control keeps a random half. The base and random arms and their evals were reused from the previous round, and all arms train the same platform Diffusion Policy and are evaluated on the same 30 placements. Differences from the paper:
- Corrected scorer. The scores now use the single inverse of the gradient covariance. They were recomputed from the saved feature sums without new rollouts, and we checked that the retrained models trained on the corrected halves.
- Same placements. CUPID scored the same 30 placements later used for the final evals, which favours CUPID.
- Natural data. Our datasets hold naturally collected excellent and ok demonstrations (60 to 92 percent excellent), not the paper’s deliberately mixed-quality sets, which leaves less to filter out.
- Floor on clamping. The base clamping policy succeeded on 1 of 30 rollouts, so its scores came almost entirely from failures.
- Approximations for local compute. Training frames were sampled every 5, 15 or 20 frames depending on the task, and a sparse random projection replaced TRAK’s dense one. Both add noise but no bias.
- Platform policy. All arms use the platform’s three-camera Diffusion Policy with its overlapping-chunk runtime instead of the paper’s implementation.
Prompt summary
Fix the double-inverse bug in the CUPID scorer, run fresh correctness and paper-fidelity reviews, and list deviations; wait for approval before training. Retrain the CUPID arm with the corrected scores and check that the new models differ from the previous ones. Reuse the base and random arms and their evals.
Response summary
CUPID v2 vs random: SR 23.3% vs 37.8% (-14.4 pp, 95% CI -25.7 to -3.2, p = 0.024) and AP -1.5 pp (p = 0.74). Per task: pour 13/30 vs 16/30, boxes 7/30 vs 18/30 (p = 0.013), clamp 1/30 vs 0/30.
Conclusion
With correct scores CUPID was worse than random (−14.4 pp), driven by boxes. Next: try new tasks with fixed 300-demo pools to see whether this generalises.