Chapter 2: Towards an agent-native CERN of AI robotics

It’s all about closing the loop.

AI robotics research is changing rapidly. Not so long ago, robotics used to be largely an exercise in handcrafted integration and coding. Like many other engineering disciplines, it is now becoming agent-driven. AI agents can write robot-specific code, build simulators, run tests, and train models in a matter of hours.

Today, we can formulate a hypothesis, prompt an agent to implement it, train a model, and evaluate the results in simulation. Much of this can already happen without human intervention.

But there is one missing piece to truly close the loop: the real world. And that is by far the most challenging part.

If our goal is to solve physical AI, shouldn’t we be running this loop again and again in the physical world?

At Poke & Wiggle, we’ve built the infrastructure to make exactly that possible for agents. Our robot farm was designed and orchestrated to become a reality computer: a collection of physical environments, always ready to run the next experiment, test the next hypothesis, and measure the results.

An API to the real world.

Academia has no shortage of brilliant ideas, and increasingly, access to substantial compute. But it remains robot-poor.

Scaling laws have taught us the importance of massive amounts of data and compute. We believe physical AI needs a third ingredient at scale: real-world experimentation and evaluation.

We need research infrastructure for robotics on the scale of what CERN became for physics: a large-scale facility combining compute and physical experimentation, dedicated to tackling the fundamental challenges on the path to physical AGI.

This is how we envision the future of physical AI research: agent-driven, grounded in real-world experimentation at scale, and progressing at a pace that was previously unimaginable.

Our first test? Reproducing recent robotics research on real robots.

We gave AI agents access to our datasets, training infrastructure, and robot farm through our API. Then we challenged them to reproduce the core claims of ten recent research papers, including preprints published just weeks ago.

The goal was simple: take a scientific claim, turn it into a working implementation, train the models, and test whether the results hold up in the physical world.

The Nitty Gritty

We’ll start with a high-level overview of the system. It consists of three key components:

• an agent harness that implements the agent’s main workflow
• the robot farm itself, along with the human operations around it
• our platform, Poker, which connects them and abstracts the robot farm and model training primitives behind a set of API calls

Our platform started as an internal tool for our engineers. A large part of our ML team worked out of the London office, while the robot farm lives in Munich, so we naturally ended up hiding all real-world interactions with the robot farm behind an API. This came in really handy when we decided to plug an agent into the experiment loop.

With the platform, we could already request data collection for existing and new tasks. This data then went through quality control and got annotated, including breaking it down into sub-tasks. Using a rich set of filters, we could dynamically build datasets.

Once we lock in a data slice, we can train different model families, launching dozens of experiments with various parameters in an instant. To close the loop, the platform puts the trained models through strict real-world evaluation, where we control the placement of every single object for reproducibility. Eval rollouts also go through quality control and annotation, which lets us measure not just the binary Success Rate (SR), but also the sub-task Average Progress (AP).

Agent harness

For the main agent model, we chose Claude Opus 5.5. It offers great pricing for long-running tasks and often beats other models—even larger and pricier ones—on relevant benchmarks like Terminal Bench 4.0, Terminal Bench Science 0.1, and FrontierML. We use Extra-High reasoning effort in all the experiments described on this page.

We believe models perform best in the native harnesses they were trained with. So we decided to use the official Claude Agent SDK. It supports a wide set of key features: spawning sub-agents (useful for review, which we’ll cover below), searching the web (to download source code for papers), creating and waiting on background tasks (for long real-world experiments or training run babysitting), and more.

The agent itself runs on a virtual machine inside a Docker environment with minimal permissions. For every request (session), the agent is spawned in a freshly created workdir. This directory comes preloaded with the paper’s PDF and code for baseline policy presets. These are just a few LeRobot policies (like Multitask DiT or pi0.5) that show how to integrate with our dataset format, Trainer, and so on.

Crucially, the agent has access to an RTX 5090 32GB GPU. This is super useful for local debugging, running tests, and verifying model deployment code. In our experience, this is way more practical than waiting for a cloud job to spin up, only to find out something broke on the very first training step. The agent iterates much faster, which nudges it to experiment more.

Once the agent decides the training or inference code is debugged, it pushes it—along with the main experiment code—to a GitHub repo for experiment tracking. Then the system builds training and inference images on request. All of this lets the agent work with any library or implementation without being limited by its starting environment.

The One Tool It Needs

To interact with the platform and other resources, we gave the agent a custom CLI. We chose a CLI because it’s a convenient and increasingly popular way (see Hugging Face’s CLI for agents, for example) to expose broad functionality that sometimes requires extra custom logic.

We built in industry best practices to minimize friction for the agent: hints for next commands, JSON output (for easy parsing with jq), and more.

The CLI comes with a Skill.md file. It describes the core features and how the tools work together seamlessly (like running evals for different models on the exact same robot for maximum comparability).

The CLI also acts as our safety net, letting us strictly control what the agent can access and in what quantities. Even though we didn’t see any reward hacking or reward-seeking behavior in our experiments, we wanted an extra layer of checks.

For transparency—and so you can better understand the limits of our current approach—we’re briefly sharing a high-level list of the main CLI commands:

CLI commands list

• task list / show — Discover available tasks and inspect their objects and experimental setups.
• collect request — Request new demonstrations from lab operators.
• dataset prepare / show — Select, inspect, and export training data.
• train — Launch local or cloud training, ensuring the experiment code is pushed and the git tree is clean.
• model register / upload / show — Register and publish validated checkpoints.
• serve validate / prepare / start — Validate inference compatibility, prepare models for robot-station execution, or start a local inference server.
• eval request / download — Request real-robot evaluations and download recordings and episode metadata for analysis.
• operation status / wait / stop / logs — Monitor progress and evaluation results, wait for completion, stop operations, and inspect cloud training logs.

You can find more details about the CLI and its arguments in the agent trajectories, which we’re sharing openly.

The Full Cycle

Along with the CLI, we pass the agent a detailed (~300-line) system prompt through the SDK. It lays out our expectations, constraints, and the general workflow. We’re betting on future model improvements, so we treat the agent primarily as a system that can autonomously run the entire cycle of reproducing papers and testing hypotheses end-to-end.

In short, the agent’s workflow looks like this:

1. Read the attached paper, find the code and project page, and gather extra context.

2. Pick a few tasks from the available set that best fit the specific paper. Right now, this set includes the 10 tasks from our previously released PAW-10 benchmark, 5 of which are fully public (training data, rollouts, etc.)—see our PAW-10 release post for more details. The agent uses its own judgment and the authors’ method description. Depending on the paper, it might pick a few bimanual or high-precision tasks, for example. We made sure the agent picks at least one public task so anyone can see how the implemented method performs.

   

   In most cases, the agent picks 3–5 tasks, depending on the expected jump in Success Rate reported in the paper.

3. Prepare dataset exports with the right parameters, like image resolution, action or state representation.

4. Implement the method from the paper, making minimal adaptations for our setup (a bimanual Franka robot—see our PAW-10 release post for more details). The implementation can build on the authors’ source code if it’s public. We encourage the agent to test the code on its local GPU.

5. Launch cloud training runs (usually 6–10 models, 2 per task: the baseline and the method).

6. As the models finish training, request evaluations on the chosen tasks, followed by quality control and annotation. Typically, one model runs 30 rollouts per task, adding up to ~180–300 rollouts per paper.

7. Once all the data is in, compare the baseline and the method, evaluate the statistical significance of the results, and report back to the user. The final report lists the biggest departures from the paper or the authors’ implementation, so we can judge how valid the results are in the context of the original work—and ask follow-up questions or request experiments, if needed.

Is it really autonomous but rigorous?

The agent’s level of autonomy varies depending on the paper’s complexity and how much adaptation our setup needs. Our biggest insight from these experiments is that, out of the box, today’s frontier models can’t show good human judgment and research taste. We know those terms are a bit overused, but they’re the best way to describe what’s happening.

In the system prompt, we instruct the agent to run a double review using sub-agents with clear context before kicking off long, expensive training runs. The first review focuses on analyzing the code and hunting for bugs (either in isolation or against the authors’ code, if available). The second checks alignment with the paper and whether departures from it are reasonable and acceptable.

In practice, we saw the review agent catch major deviations and flag them as P0 [Critical] many times, only for the main agent to brush them off and act like everything was fine. Often, the agent gets attached to what it’s already built. It refuses to make changes, gets defensive about feedback, and makes up flawed reasoning on the fly to justify itself.

At the same time, the limitations and significant departures caught in review are often super easy to fix. The clearest example is the action type: absolute, delta, or relative. The agent might make the wrong call here, even when the paper explicitly highlights that the method solves problems for a specific representation. Though most of these differences are paper-specific, and the agent just makes a bad guess when interpreting the text.

Along the way, we tried tweaking the prompts to find a balance between “stick to the provided LeRobot baselines” and “change whatever you want, as long as it’s closer to the paper.” Right now, that balance is pretty fragile, so we almost always need a human in the loop to make judgment calls. Across different runs, we tried three ways of working with the agent:

1. Let the agent run end-to-end, carefully review the code and compare it against the paper, find the key mistakes, and update the prompt for a fresh run from scratch.
2. Let the agent run end-to-end, review the code against the paper, give it steering feedback, and wait for a second iteration in the same session.
3. Ask the agent to pause after its final self-review, get human judgment on all the departures from the paper it found, make fixes, and then launch training and evaluation.

It’s still not clear which approach works best and when. We’re excited to share our agent trajectories, including our human interventions, and we’ll keep sharing what we learn from future runs.

Case study

Let’s take the paper Direction-Scale Decomposition in Action Representation as an example to see what a typical workflow with iterations and steering feedback looks like. Here’s the relevant bit from the paper’s abstract:

We introduce Direction-Scale Decomposition (DSD), an action representation that decomposes translation and rotation increments into direction and scale components before tokenization. DSD isolates motion direction while retaining magnitudes in separate scale channels. We evaluate DSD with uniform binning (BIN) …

After skimming the paper, we immediately wanted to test it on our data and robot. Since many papers explore several extra ablations alongside their headline result, we sent the agent a message to narrow down a specific claim: “Test whether DSD’s direction–scale action representation improves real-robot task success over standard action binning, without additional robotics pretraining.”

And that’s it! The agent goes off to work autonomously for 25 hours—picking tasks, implementing the method, testing, training, and running real-world evaluations—before coming back with a short report.

<…> is one paragraph we omitted for brevity. It can be found in the full transcript of this run, which we’re sharing “as is”. Please be aware of the limitations of the results due to different implementation as documented in the implementation details here.

In its response, the agent highlights the most important part: the material departures from the paper. Some of them (like adapting to a 30Hz control frequency) are minor and shouldn’t affect the results. However, point 3, “Wider normalization bounds”, looks important. In the paper, the authors explicitly mention using q1–99 normalization followed by clipping for standard action binning.

When the agent applied this to our datasets, it noticed the error was too high and decided on its own to switch to 0.1/99.9th percentile normalization. After running the experiments, it found no effect: the DSD method didn’t boost SR on real-robot tasks.

This is a textbook example of the agent making a bad judgment call and changing something that directly affects the paper’s core. As a result, it altered the baseline we were comparing against. So we wrote a follow-up: “q1-99 with clipping was an important part of the baseline. Can you please use it for BIN baselines, re-train the models in this arm and re-eval them?”

The agent updated the baseline, retrained that group of models, and ran them through the evaluation pipeline. Now, with the paper’s proper binning baseline, the agent reports that DSD improves real-robot success rate by +13.3 pp.

This does not show that clipping explains DSD’s gain: our DSD weights its direction loss differently from the authors’ implementation, and the authors saw no comparable effect from widening the bounds. We also tested only a single embodiment without additional robotics pretraining, not the paper’s other claims.

See the detailed DSD results. Please be aware of the limitations of the results due to different implementation as documented in the implementation details here.

In general, we compare models across two metrics: success rate (SR), which is the share of fully successful episodes, and average progress (AP), the average share of completed sub-tasks. Detailed annotations help us spot changes that binary SR hides. For example, a model might reach the final steps more often, even if it can’t quite finish the whole task yet.

For both metrics, the agent also builds 95% confidence intervals and evaluates statistical significance. In the example above, the result is statistically significant for SR, though not for AP—which we still count as a win.

Note that in all our experiments, we compare models on identical object placements. As we mentioned earlier, we care a lot about reproducible comparisons, so we carefully control the initial scene state:

This setup accounts for how hard a specific scene is and can narrow the confidence interval of the difference between models by using paired comparisons. Statistical significance can be assessed using methods such as McNemar’s test and paired permutation tests.

Interesting Observations

Impact of the Authors’ Code

When a paper has public code, the agent uses it—not just as a reference, but by actively copying whole modules and tests. In OAT and ManiFlow, for example, it used the original implementations as an executable ground truth, writing tests to match intermediate activations, loss functions, and model outputs. These checks help catch discrepancies that would go unnoticed during a normal run.

For instance, while porting Patch Policy, the agent found that DINOv2 produced different visual features because HF Transformers and torch.hub handle positional embeddings differently. So public code doesn’t just help implement a method—it also lets the agent verify that the method’s exact behavior survived the integration.

On its own, this isn’t surprising; coding agents can do a lot. Still, in the era of coding agents, when the barrier to writing and integrating code is lower than ever, we think publishing code is especially important for reproducibility. That way, agents can iterate on it in real-world experiments—and we observe that in our runs.

Reactive-Proactive Agent

The CLI gives agents a fixed set of tools to interact with the platform. Because the toolset is small, we didn’t expect any surprising interactions—but the agent managed to surprise us anyway. One tool lets it download .mcap files from eval rollouts to analyze trajectories in detail. Originally, we thought the agent would only use this when implementing data-curation papers like CUPID.

In one run, the agent introduced a bug into the inference code that went unnoticed until real-world evaluation started. While monitoring progress, the agent noticed after 10 rollouts that the Success Rate was stuck at 0%. Instead of waiting for the eval to finish, it downloaded the .mcap files for those rollouts, where we log a ton of debug info.

It quickly spotted the problem: after the first call to the policy, requests just stopped coming in. The agent traced this back to its own change in the inference code, which broke the policy interface contract. It rolled back the change, updated the serving image, and restarted the evaluation assignment. It took less than 35 minutes from spotting the issue to running the first fixed rollouts.

You can check out the agent’s trajectory log.

One-shot integration of new models

Earlier, we launched the PAW-10 benchmark, which measures 4 model families across 10 tasks—with over 14,000 rollouts. It took us time to add these 4 policies and get them running smoothly, and we’re working on adding more for a fair, rigorous comparison.

Right before we started writing this post, we saw Microsoft release Rho, a new VLA. We decided to task our agent with integrating it, training it on different tasks, and validating its performance. The very next day, with zero engineer intervention, we got the results:

Even though we didn’t design the system for this or test workflows beyond verifying paper claims, it worked on the first try. For us, this is great validation that the system is useful across a wide range of practical tasks. We’re super excited and looking forward to speeding up our iterations and experiments as we improve our agentic system and infrastructure.

Limitations & future work

Today, we’re excited to share the first batch of papers and claims we’ve run through our system. This iteration gave us valuable experience to guide future improvements. We plan to maintain and expand the project, and we’ll keep sharing updates along the way. It’s important to note that our current results come with a few limitations:

• We use a fixed set of 10 tabletop tasks, and we’ve disabled the agent’s ability to add new ones on its own for now. Because of this, we can’t always find several great task matches for every paper and its claim.
• We hand-picked and filtered the papers based on two things: effect size (to make it easier to reach statistical significance) and compute requirements (to speed up iterations so we aren’t waiting a week for a training run to finish).
• Right now, even if the agent works flawlessly, the system can’t reproduce just any paper, like work on online RL, which is gaining traction in robotics.
• Reproducing a paper always requires some adaptation simply because of our bimanual robot, tasks, control stack, and execution loop. Some of these adaptations can directly affect the expected performance gains. Sometimes we have to make a call on whether a departure from a paper is significant or not.
• Finally, the most obvious one: coding agents aren’t perfect. Even with access to the authors’ code, they can’t always faithfully reproduce every detail of a paper’s method within our setup. This can lead to false negatives, where the agent concludes there’s no effect, even though a proper implementation would show one.

Corrections

• 16 October 2026: In the DSD case study, we added that the result does not show that clipping explains DSD’s gain, that our DSD weights its direction loss differently from the authors’ implementation, and the scope of our evaluation.
• 18 October 2026: We added a note below each DSD case study figure that points to the documented implementation differences.

We thank the authors, especially Yufei Duan, for pointing out and contributing to the limitations of the DSD case study.