Poke &Wiggle
Papers
Paper

CUPID

CUPID: Curating Data your Robot Loves with Influence Functions

Claim tested: CUPID's performance-influence filtering of existing demonstrations improves real-robot task success over random filtering.

Tested

Paper
arXiv⁠ (opens in a new tab)
Code
GitHub⁠ (opens in a new tab)
Venue
CoRL 2025
Authors
Christopher Agia et al.
Farm tasks
5
Evaluation
480 episodes (240 for baseline, 240 for method)

Result

Improvement over baseline

Not replicated

Reported
+36 pts
Ours
−11 pts
−200+20+40+60
  • Gain vs baseline
  • 95% CI, −24 pts to +3 pts
  • Paper's reported gain
  • No difference

Success rate

Reported
84%
Ours
27%
0%20%40%60%80%100%
  • Measured on our stations
  • 95% CI, 19%–37%
  • Reported in the paper

Across iterations

  1. Iteration 1vs Random filtering−11 pts
  2. Iteration 2vs Random filtering−14 pts
  3. Iteration 3vs Random filtering+8 pts
−200+20+40+60

By task

TaskBaselineRandom filteringMethodCUPIDImprovement
e01Clamp electrical component0%0/300%0/300 pts−11 to +11
e02Place loaded boxes in container (bimanual)60%18/3057%17/30−3 pts−27 to +20
e03Pour screws into the funnel53%16/3023%7/30−30 pts−50 to −5
All tasks38%34/9027%24/90−11 pts−24 to +3
Reported in the paper48%84%+36 pts

Rollouts

One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.

  1. e03Pour screws into the funnel
  2. Held-out benchmark environment
    e01Clamp electrical component
  3. Held-out benchmark environment
    e02Place loaded boxes in container (bimanual)

Implementation Details

What we changed

This round’s CUPID scores were wrong, and the round is withdrawn. A bug in our scorer applied the inverse of the gradient covariance twice, so the scores used its squared inverse instead of the method’s single inverse and did not test CUPID. We had trained a base Diffusion Policy on all demonstrations of each of three tasks (pouring screws, placing boxes and clamping a component), evaluated it on 30 placements, scored every demonstration from those rollouts and kept the top-scoring half. The control kept a random half, and both arms trained the same platform Diffusion Policy and were evaluated on the same 30 placements. Differences from the paper:

  • Wrong scores. The scorer inverted an already accumulated Gram matrix a second time, so the scores used its squared inverse. The kept halves and the CUPID results are invalid.
  • Same placements. CUPID scored the same 30 placements later used for the final evals, which favours CUPID.
  • Natural data. Our datasets hold naturally collected excellent and ok demonstrations (60 to 92 percent excellent), not the paper’s deliberately mixed-quality sets, which leaves less to filter out.
  • Floor on clamping. The base clamping policy succeeded on 1 of 30 rollouts, so its scores came almost entirely from failures.
  • Approximations for local compute. Training frames were sampled every 5, 15 or 20 frames depending on the task, and a sparse random projection replaced TRAK’s dense one. Both add noise but no bias.
  • Platform policy. All arms use the platform’s three-camera Diffusion Policy with its overlapping-chunk runtime instead of the paper’s implementation.

See the code changes on GitHub⁠ (opens in a new tab)

Prompt summary

Test whether CUPID's performance-influence filtering of existing demonstrations beats random filtering on 3 real-robot tasks. Train an all-demo base policy per task, score demos with CUPID from base rollouts, then train and compare CUPID top-50% and random-50% arms with identical settings on the same 30 placements.

Response summary

CUPID v1 vs random was SR 26.7% vs 37.8% (-11.1 pp, p = 0.063) and AP -10.6 pp (p = 0.023); base was 9/30, 22/30 and 1/30. This result was later withdrawn because of a scoring bug that used (X^T X)^-2 instead of (X^T X)^-1, and the CUPID v1 evals were archived as invalid.

Conclusion

Invalid: a scoring bug inverted the influence weights. Next: fix the scorer and retrain the CUPID arm.