GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization

Figure 1 : Architecture of GuidedVLA. We introduce explicit, structured guidance into the multi-head attention layers of the VLA action decoder. Instead of relying on implicitly entangled representations, we repurpose dedicated attention heads to specialize in distinct task-relevant factors: (i) Object Head supervises its attention maps to explicitly ground task-relevant objects and suppress distractors via \mathcal{L}_{object} ; (ii) Skill Head aligns internal feature representations with temporal skill phases (e.g., Pick \rightarrow Place) through auxiliary classification \mathcal{L}_{skill}

Figure 2 : ControlNet-style residual adapter for plug-and-play head specialization. The pretrained main attention branch is kept as the behavior-preserving path, while a factor-specific attention branch is fused through a zero-initialized projection. The adapter copies weights from the base policy and gradually injects task-relevant biases during training.

Figure 6 : Higher Factor Quality Leads to Better Task Performance. Top: Quantitative analysis on the LIBERO-Plus layout perturbation track shows that improving the quality of each specialized head consistently boosts success rates. (a) Object Head: as the proportion of attention focused on task-relevant object regions increases, success rises from 61.3% to 74.6%, highlighting the importance of precise object-centric attention. (b) Skill Head: higher skill-recognition accuracy, measured by a linear probe, correlates with improved performance (66.2% to 72.9%), indicating that better temporal und

Figure 12 : LIBERO-Plus rollout visualization (spatial task suite of LIBERO-Plus). Each column corresponds to one stage in the whole episode, with 7 stages in total. First row shows the original RGB observations during the rollout. Second row visualizes the attention maps from GuidedVLA ’s object head. Third row presents the depth information encoded by the depth encoder, and fourth row illustrates the corresponding attention maps produced by GuidedVLA ’s depth head based on the depth features in the third row.

Figure 4 : RoboTwin 2.0 Benchmark Performance. Success rates across 8 manipulation tasks comparing the \pi_{0} baseline, single-head experts, and our full model. While specific heads excel at aligned tasks (e.g., depth head for geometry-heavy Beat Hammer Block), the full model (purple) integrates these capabilities to achieve the best overall average performance ( 90.63\% ).

Figure 10 : ALOHA real-world generalization settings (T1–T3). From left to right: in-domain (positional) perturbations using a 3\times 3 anchor grid, lighting shifts with colored illumination, and scene shifts by adding distractor objects.

Figure 11 : PSI-Bot real-world generalization settings (T4–T6). From left to right: in-domain (positional) perturbations using a 3\times 3 anchor grid, lighting shifts with colored illumination, and scene shifts by adding distractor objects.

Figure 5 : Real-world Robot Platforms and Evaluation Tasks. (a) ALOHA AgileX dual-arm mobile manipulator with left/right wrist Orbbec Dabai cameras and a third-person Orbbec Dabai camera; we evaluate three household tasks: pick up fruits and vegetables, stack the bowls, clean the tabletop. (b) PSI-Bot equipped with RealMan RM63 arm(s) and DexHand2 Pro hands, with head/chest RealSense D435 cameras; we evaluate three lab tasks: pick up beaker, stack beakers, and heat beaker.

Figure 8 : Comparison of GuidedVLA against Mixture Alternative. Attention head specialization explicitly outperforms learning all objectives in a mixture.