Poke &Wiggle
Papers
Paper

CUPID

CUPID: Curating Data your Robot Loves with Influence Functions

Claim tested: CUPID's performance-influence filtering of existing demonstrations improves real-robot task success over random filtering.

Tested

Paper
arXiv⁠ (opens in a new tab)
Code
GitHub⁠ (opens in a new tab)
Venue
CoRL 2025
Authors
Christopher Agia et al.
Farm tasks
5
Evaluation
480 episodes (240 for baseline, 240 for method)

Result

Improvement over baseline

Not replicated

Reported
+36 pts
Ours
−14 pts
−200+20+40+60
  • Gain vs baseline
  • 95% CI, −27 pts to −1 pts
  • Paper's reported gain
  • No difference

Success rate

Reported
84%
Ours
23%
0%20%40%60%80%100%
  • Measured on our stations
  • 95% CI, 16%–33%
  • Reported in the paper

Across iterations

  1. Iteration 1vs Random filtering−11 pts
  2. Iteration 2vs Random filtering−14 pts
  3. Iteration 3vs Random filtering+8 pts
−200+20+40+60

By task

TaskBaselineRandom filteringMethodCUPIDImprovement
e01Clamp electrical component0%0/303%1/30+3 pts−8 to +17
e02Place loaded boxes in container (bimanual)60%18/3023%7/30−37 pts−56 to −12
e03Pour screws into the funnel53%16/3043%13/30−10 pts−33 to +15
All tasks38%34/9023%21/90−14 pts−27 to −1
Reported in the paper48%84%+36 pts

Rollouts

One clip per environment from this entry's evaluation, beside where each evaluation episode placed the environment's objects, coloured by its rating. Use the button beside each environment to open its evaluation dataset in the dataset viewer.

  1. e03Pour screws into the funnel
  2. Held-out benchmark environment
    e01Clamp electrical component
  3. Held-out benchmark environment
    e02Place loaded boxes in container (bimanual)

Implementation Details

What we changed

We fixed the scoring bug that withdrew the previous round, rescored the demonstrations of all three tasks (pouring screws, placing boxes and clamping a component) and retrained only the CUPID arm. CUPID scores every demonstration from 30 rollouts of a base Diffusion Policy trained on all demonstrations and keeps the top-scoring half. The control keeps a random half. The base and random arms and their evals were reused from the previous round, and all arms train the same platform Diffusion Policy and are evaluated on the same 30 placements. Differences from the paper:

  • Corrected scorer. The scores now use the single inverse of the gradient covariance. They were recomputed from the saved feature sums without new rollouts, and we checked that the retrained models trained on the corrected halves.
  • Same placements. CUPID scored the same 30 placements later used for the final evals, which favours CUPID.
  • Natural data. Our datasets hold naturally collected excellent and ok demonstrations (60 to 92 percent excellent), not the paper’s deliberately mixed-quality sets, which leaves less to filter out.
  • Floor on clamping. The base clamping policy succeeded on 1 of 30 rollouts, so its scores came almost entirely from failures.
  • Approximations for local compute. Training frames were sampled every 5, 15 or 20 frames depending on the task, and a sparse random projection replaced TRAK’s dense one. Both add noise but no bias.
  • Platform policy. All arms use the platform’s three-camera Diffusion Policy with its overlapping-chunk runtime instead of the paper’s implementation.

See the code changes on GitHub⁠ (opens in a new tab)

Prompt summary

Fix the double-inverse bug in the CUPID scorer, run fresh correctness and paper-fidelity reviews, and list deviations; wait for approval before training. Retrain the CUPID arm with the corrected scores and check that the new models differ from the previous ones. Reuse the base and random arms and their evals.

Response summary

CUPID v2 vs random: SR 23.3% vs 37.8% (-14.4 pp, 95% CI -25.7 to -3.2, p = 0.024) and AP -1.5 pp (p = 0.74). Per task: pour 13/30 vs 16/30, boxes 7/30 vs 18/30 (p = 0.013), clamp 1/30 vs 0/30.

Conclusion

With correct scores CUPID was worse than random (−14.4 pp), driven by boxes. Next: try new tasks with fixed 300-demo pools to see whether this generalises.