PaperCUPID
CUPID: Curating Data your Robot Loves with Influence Functions
Claim tested: CUPID's performance-influence filtering of existing demonstrations improves real-robot task success over random filtering.
Tested
Result
Improvement over baseline
Not replicated
- Reported
- +36 pts
- Ours
- −11 pts
- Gain vs baseline
- 95% CI, −24 pts to +3 pts
- Paper's reported gain
- No difference
Success rate
- Reported
- 84%
- Ours
- 27%
- Measured on our stations
- 95% CI, 19%–37%
- Reported in the paper
Across iterations
- Iteration 1vs Random filtering−11 pts
- Iteration 2vs Random filtering−14 pts
- Iteration 3vs Random filtering+8 pts
By task
| Task | BaselineRandom filtering | MethodCUPID | Improvement |
|---|---|---|---|
| e01Clamp electrical component | 0%0/30 | 0%0/30 | 0 pts−11 to +11 |
| e02Place loaded boxes in container (bimanual) | 60%18/30 | 57%17/30 | −3 pts−27 to +20 |
| e03Pour screws into the funnel | 53%16/30 | 23%7/30 | −30 pts−50 to −5 |
| All tasks | 38%34/90 | 27%24/90 | −11 pts−24 to +3 |
| Reported in the paper | 48% | 84% | +36 pts |
Rollouts
One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.
e03Pour screws into the funnel BaselineRandom filtering16/3053%
Show in dataset viewer of Random filtering on Pour screws into the funnelMethodCUPID7/3023%
Show in dataset viewer of CUPID on Pour screws into the funnel- Held-out benchmark environment
e01Clamp electrical component BaselineRandom filtering0/300%
Show in dataset viewer of Random filtering on Clamp electrical componentMethodCUPID0/300%
Show in dataset viewer of CUPID on Clamp electrical component - Held-out benchmark environment
e02Place loaded boxes in container (bimanual) BaselineRandom filtering18/3060%
Show in dataset viewer of Random filtering on Place loaded boxes in container (bimanual)
Implementation Details
What we changed
This round’s CUPID scores were wrong, and the round is withdrawn. A bug in our scorer applied the inverse of the gradient covariance twice, so the scores used its squared inverse instead of the method’s single inverse and did not test CUPID. We had trained a base Diffusion Policy on all demonstrations of each of three tasks (pouring screws, placing boxes and clamping a component), evaluated it on 30 placements, scored every demonstration from those rollouts and kept the top-scoring half. The control kept a random half, and both arms trained the same platform Diffusion Policy and were evaluated on the same 30 placements. Differences from the paper:
- Wrong scores. The scorer inverted an already accumulated Gram matrix a second time, so the scores used its squared inverse. The kept halves and the CUPID results are invalid.
- Same placements. CUPID scored the same 30 placements later used for the final evals, which favours CUPID.
- Natural data. Our datasets hold naturally collected excellent and ok demonstrations (60 to 92 percent excellent), not the paper’s deliberately mixed-quality sets, which leaves less to filter out.
- Floor on clamping. The base clamping policy succeeded on 1 of 30 rollouts, so its scores came almost entirely from failures.
- Approximations for local compute. Training frames were sampled every 5, 15 or 20 frames depending on the task, and a sparse random projection replaced TRAK’s dense one. Both add noise but no bias.
- Platform policy. All arms use the platform’s three-camera Diffusion Policy with its overlapping-chunk runtime instead of the paper’s implementation.
Prompt summary
Test whether CUPID's performance-influence filtering of existing demonstrations beats random filtering on 3 real-robot tasks. Train an all-demo base policy per task, score demos with CUPID from base rollouts, then train and compare CUPID top-50% and random-50% arms with identical settings on the same 30 placements.
Response summary
CUPID v1 vs random was SR 26.7% vs 37.8% (-11.1 pp, p = 0.063) and AP -10.6 pp (p = 0.023); base was 9/30, 22/30 and 1/30. This result was later withdrawn because of a scoring bug that used (X^T X)^-2 instead of (X^T X)^-1, and the CUPID v1 evals were archived as invalid.
Conclusion
Invalid: a scoring bug inverted the influence weights. Next: fix the scorer and retrain the CUPID arm.