Instruction Blindness in Vision-Language-Action Policies: Diagnosis and a Low-Rank Data Cure

Liu, Yuchen; Han, Muchen

Yuchen Liu^* Muchen Han^*

^* Equal contribution · correspondence: yuchen@φ.monster · preprint, under review

Same scene, same seed, same initial state. Mid-execution, the instruction changes from “pick up the milk” to “pick the orange juice.” The stock base (left) completes the original goal; Galahad (right) abandons its committed approach and completes the newly named task. A mid-execution swap is a demo — every grounding number in the paper is measured from the initial state.

The same recipe on a low-cost SO-101 arm (commodity servos, one third-person and one wrist camera; raw phone footage). Mid-execution the named candy changes and the arm switches to the newly named one. As above, a mid-execution swap is a demonstration — the measured hardware evidence is the vision-gating control (blank the cameras and the policy collapses to one instruction-independent attractor) and the reach-obey rate, both in the paper's appendix.

94.5%

cured grounding

13.5%

its base

0–1%

the field

200/200

obeys the new name

7 axes

one set of weights

Instruction blindness

A vision–language–action policy can pass a manipulation benchmark without reading its instruction: replace the words with the string xxx and it emits the same actions. It succeeds by moving to the object that usually occupies the target position, not by resolving the name. We call this failure mode instruction blindness and show it is a concrete instance of causal confusion in imitation learning. On the one axis that separates a position shortcut from language grounding — object identity, rename the target and keep the scene — the published field selects the correct object 0–1% of the time.

A severed route, not missing knowledge

The pretrained backbone already resolves the named object in its language stream — activation patching localizes the binding to a single head, and injecting the word direction at an early layer flips the selected object 100% of the time — but the action head routes position and never reads that binding. Because the knowledge is present, the repair is a data recipe, not an architecture: make the instruction the only predictor of the target, and protect the pretrained grounding with a low-rank update. Full fine-tuning on the identical data destroys grounding (12.5%); the low-rank cure installs it (94.5%). The released weight grounds seven referent-type axes in one set of weights — object identity, spatial, goal, color, category, negation, composition (obey 75–99%, occlusion ≤2% on every axis).

Results

object substitution: field 0-1%, base 13.5%, cured 94.5% — Change the object's name; does the arm fetch it? The cure reaches 94.5% where its base scores 13.5% and the field 0–1%, obeying the new name 200/200 and never fetching the trained object.

one adapter grounds six referent types — One adapter grounds six referent types at once (obey rate 77–95%); generality holds within trained coverage and a held-out type falls to chance.

conditioning gradient: text-cond 65-78, action-cond 20-45, imitation 0-1 — One disease along one spectrum: grounding tracks how the instruction reaches the output. Text-conditioned generation reads the name (65–78%), action-conditioned world models bottleneck it (20–45%), imitation policies discard it (0–1%).

Measure the shortcut in any policy

The measurement is a one-command counterfactual. It runs the same test on your policy: not a drop under perturbation, but a positive control — rename the target, and does the arm go to the newly named object?

from galahad.battery import run

report = run(policy, suite="libero_object")
report.swap_obey    # goes to the newly named object?  (grounding)
report.occ          # collapses with blank cameras?     (uses vision)
report.nonsense     # collapses on a meaningless name?  (needs a real name)

The harness ships in the repo (galahad/); the one-command interface is the packaging in progress.

Cite

@misc{liu2026instructionblindness,
  title  = {Instruction Blindness in Vision-Language-Action Policies:
            Diagnosis and a Low-Rank Data Cure},
  author = {Liu, Yuchen and Han, Muchen},
  year   = {2026}
}