Patch Policy (CLS versus Dense)

We compare dense patch features with CLS features to test the paper’s claim that spatially detailed visual features improve manipulation success. Both policies use the same frozen DINOv2 encoder and the authors’ VQ-BeT architecture, with reference-code components reused. Both receive images only. Training and execution settings are matched between the two arms.

• Three cameras. Both arms read three cameras, resized from 256 to 224 pixels, instead of the paper’s two.
• Longer chunks. Both arms predict 30 steps instead of 5 or 10 because controls run at 30 Hz, giving one second of planned motion.
• Crossfade instead of temporal ensembling. Both arms replan every 100 ms. Execution mixes old and new target trajectories over a 400 ms crossfade, increasing the new plan’s weight; overlapping updates revise the ongoing motion. This replaces the paper’s per-step prediction and uniform averaging, identically in both arms.
• History spacing. Both visual history frames are 100 ms apart. The paper does not specify the control rate needed to establish exact timing parity.
• Normalized actions. Actions are scaled to the range from -1 to 1 by their minimum and maximum, because the two arms’ positions, rotations and grippers have different units. The reference code uses raw actions.
• Larger tokenizer and offset head. Two-arm actions and longer chunks increased reconstruction error. We raised each codebook from 16 to 64 entries and widened encoder/decoder MLPs from 128 to 512, adding one hidden layer. Held-out reconstruction improved by 19–28%, but the offset head grew to about 79M parameters. Both arms share these changes.
• Tokenizer fitting budget. We fit the tokenizer for 20k optimization steps instead of the reference’s epoch-based budget, then freeze it during policy training.
• Final-checkpoint evaluation. Both arms use the final checkpoint after 50k steps at batch 128, split over two GPUs, without intermediate checkpoint selection. The paper reports best-checkpoint selection in simulation; its real-robot selection procedure is not established.
• Mixed precision. The visual encoder and transformer run in BF16, while action heads, VQ decoding and losses remain FP32. Both arms use the same precision.