中文
Cosmos3 Policy · Internal diagnosis

Why doesn't the robot pick the object we ask for?
Can we find the cause inside the network?

Milk, butter and a cream-cheese box sit on the table. We say “pick up the milk”, yet the arm may head for the cream cheese. We want to tell apart two cases: the model never uses the word “milk”, or it uses it but other information pulls the action away.

Current finding: language does reach the action computation, but which object gets picked mostly follows object position. Swapping internal information can push the arm to a different object, but we have not yet found a location that corrects it.
01 · How the model computes

From camera and instruction to robot action

Camera “pick the milk” 36-layer network L1 L2 ··· L36 Action vectors: each layer reads image + text, then updates Action move · rotate · grip MuJoCo we measure and intervene here

Full architecture diagram → We study only the middle part. The network keeps a set of vectors representing the action about to be generated. Each layer reads the image and the text, updates these vectors, and at the end they are turned into arm commands.

02 · Does language get in?

Yes: changing one word makes the action vectors diverge layer by layer

before L1after L1after L36 diff 0%0.15%3.22% Same image, same initial noise, only “milk” → “butter”

So “language never enters the network” is ruled out. But note: a changed vector does not mean the right object is chosen; the arm may still go to the same thing, and the percentage is not a score of how much the model understood.

03 · Then what does it follow?

“Pick up the milk” in four layouts: it follows position

Initial frames of four layouts and the object actually picked
Real MuJoCo rollouts. Top: milk at its original place, both picks are milk. Bottom left: milk moved away, picks cream cheese. Bottom right: milk moved away and cream cheese/butter swapped, picks the butter now at the old cream-cheese spot. Each layout repeated with three seeds, same object every time.

After the milk is moved away, the arm picks the cream cheese. When we then swap the cream cheese and the butter, it does not follow the cream cheese; it picks the butter sitting where the cream cheese used to be. The choice is driven more by that location than by the name in the instruction.

Milk moved away: picks cream cheese wrong
Then swap cream cheese and butter: picks butter at the old spot wrong

It is not deaf in general: in another scene, the same image with “bowl” picks the bowl and with “wine bottle” picks the bottle. The problem is that in some layouts, position overrides language.

04 · Swapping information inside the network

Which internal information actually decides the target?

Run ARun B chosen layer copy part of A's vectors into B Result of B: internal vectors → changed ✓ target → unchanged, or pushed elsewhere never corrected to the instructed object
What we swappedInternal vectorsObject picked
Action vectors at layer 12, from a run that correctly picked milkchangedsame still cream cheese
Action vectors at layer 30, same sourcechangedpushed picks butter (now at the milk's old spot), still not milk
Layer-36 text for “milk box” ↔ “cream cheese” (first query only), MLP response frozenchangedsame each keeps its original target

This asks: does this piece of internal information merely change, or does it decide which object is picked? So far, text swaps give the former; swapping action vectors can only push the target, and what pushes it looks like position, not language.

05 · External confirmation

Reply from the Cosmos discussion: a common failure, not unique to this model

Weak language sensitivity appears to be a common failure mode among open-source VLA/WAM policies after task-specific SFT. We observed similar behavior in the LIBERO-finetuned checkpoints of Cosmos3, π0.5, and LingBot-VA. … This is consistent with the broader problem of policies relying on familiar visual context more strongly than language instructions.
ModelSame gripper direction under close/open swap
Cosmos3180 / 180
π0.5180 / 180
LingBot-VA180 / 180

Suggested mitigation directions are CAG, CAST and CounterAlign, none yet validated on this checkpoint. See discussion #369

Next

Why don't the language-driven changes move the target choice?

We have ruled out “language never gets in”. Next we check whether the target is already fixed by image and position in earlier layers, so that later language information can no longer change it.