ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

Figure 1: Overview of DynAfford benchmark. Unlike prior embodied benchmarks (left) that assume static object usability and fully specified goals, DynAfford introduces dynamic object affordances and commonsense-driven goal conditions (right). Agents must detect latent preconditions (e.g., cleanliness), resolve temporarily inapplicable actions, and adapt their behavior beyond literal instruction following.

Figure 2: Overview of ADAPT: Affordance-Aware Action Selection. (a) Standard embodied instruction-following pipeline used in prior work, where the agent directly executes a planned action sequence without considering dynamic affordance constraints. (b) Our ADAPT framework augments action execution with affordance awareness. For each proposed action, the agent first performs Stage 1: Affordance Inference by jointly considering the current observation and the intended action. If the action is considered inapplicable (e.g., the microwave is occupied), Stage 2: Applicability Resolution selects an

Figure 3: Isolates the affordance inference component of ADAPT and compares it with several commercial vision-language models under the same input format. Despite stronger general-purpose reasoning, commercial models underperform the LoRA-finetuned LLaVA on affordance inference by an average margin of more than 10%.

Figure 6: Object Affordance Prediction Accuracy on Visually Distinguished Objects. Our fine-tuned LLaVA-v1.5-7B model outperforms all baselines on large objects (e.g., Pot, Pan, Microwave) where visual cues are more spatially distinguished in the observation space. These results highlight the model’s ability to leverage strong visual evidence for dynamic affordance reasoning.

Figure 4: Task types and annotation statistics in DynAfford. DynAfford contains over 1,000 demonstrations across 6 task types of varying complexity, each paired with 3–6 language instructions. Tasks range from simple placement to hierarchical stacking involving movable and fixed containers.

Figure 5: Templated input for multimodal affordance reasoning. Each input consists of a triplet of images—(1) a reference of the object in an available state, (2) a reference in an unavailable state, and (3) the current egocentric frame—paired with a textual prompt to assess the object’s current usability.

Figure 8: Prompt format for visibility detection. We use an pretrained LLaVA-1.5-7B model to assess whether the target object is visible in the current egocentric view. The prompt is dynamically adapted based on the target object’s category, guiding the model to make accurate visibility judgments.