Vision-and-Language Navigation for UAVs: Progress, Challenges, and a Research Roadmap

Figure 2: The UAV-VLN task modeled as a Partially Observable Markov Decision Process (POMDP). The agent receives an observation and reward from the environment, updates its internal belief state, and selects an action based on its policy to influence the environment’s next state.

Figure 6: Two primary architectural paradigms for long-horizon navigation. Temporal history encoding (left) treats the task as a sequence modeling problem, using transformers to reason over past observations and actions, while explicit world modeling (right) reframes it as a spatial reasoning task by constructing a queryable semantic map of the environment.

Figure 10: A conceptual breakdown of the sim-to-real gap into its three constituent challenges: disparities in visual perception, physical dynamics, and environmental complexity.

Figure 11: The typical multi-stage pipeline for sim-to-real transfer, progressing from pure simulation to Software-in-the-Loop (SIL) and Hardware-in-the-Loop (HIL) validation before final deployment on a physical UAV.

Figure 1: An overview of the UAV-based Vision-and-Language Navigation (UAV-VLN) research landscape, illustrating the core components and methodological evolution. The process begins with a natural language command that directs an autonomous agent. This figure contrasts the traditional modular pipeline, which separates perception, reasoning, and control, with the modern integrated approach centered on Embodied Multimodal Large Models (EMLMs). The UAV executes the resulting action in a complex 3D world, navigating key challenges such as the simulation-to-reality gap. The bottom panel outlines th

Figure 7: Architectural paradigms for foundation model-driven agents. VLMs can act as a deliberative cognitive core to generate high-level plans (left), VLAs can serve as end-to-end policies mapping perception directly to action (center), or they can be combined in a hierarchical system that balances reasoning and reactive control (right).

Figure 12: A comparison of three primary strategies for reasoning under uncertainty. Procedural reasoning decomposes instructions into steps, spatial grounding uses explicit world models for disambiguation, and interactive/implicit methods either query a human for clarification or rely on end-to-end foundation models.

Figure 13: Three architectural paradigms for safety assurance in autonomous flight. Formal methods use explicit safety filters, learned approaches embed safety into an end-to-end policy, and verifiable systems use logic-based oversight to validate high-level plans before execution.

Figure 8: The evolution from single-engine simulators to multi-engine data generation pipelines. Platforms like OpenFly integrate diverse sources like game engines and geospatial data to create large-scale, varied datasets, mitigating model overfitting and improving generalization.

Figure 15: Illustration of a foundation model acting as a collaborative intelligence layer for an air-ground team. The model fuses multimodal sensory data from both the UAV and UGV with high-level human commands to generate synchronized, executable plans for the entire system.

Figure 3: A comparison of action space formulations for UAV navigation. Low-level continuous control offers fine-grained maneuverability at the cost of planning complexity, whereas high-level discretized actions simplify decision-making but can result in less efficient paths.

Figure 9: A comparison of evaluation paradigms. Holistic metrics like SPL provide a single score for task completion, while decomposed frameworks like BEDI offer a granular assessment of core sub-skills, enabling precise diagnosis of agent failures.