Planning with the Views via Scene Self-Exploration

ARCHITECTURE

Figure 11 : Point cloud coverage across all 15 models. (a) Scene coverage ratio. (b) Target intersection ratio.

Figure 13 : Image attention fraction across all 28 layers. Both models evaluated on the same 530 trajectories.

THE PROBLEM

This paper focuses on perceptionPerception & SensingPerceptionThe process of turning raw sensor data into useful understanding of the world.. Proposes ViewSuite benchmarkSimulation & Sim-to-RealBenchmarkA standard test used to compare methods fairly. for evaluating VLMModern Robot LearningVision-Language Model (VLM)A model that understands both images and text. capability to plan multi-step camera movements in 3D point-cloud environments. Identifies that frontier VLMs lack multi-turn planningControl & PlanningPlanningFiguring out what the robot should do before or during movement. composition despite understanding single view-action transformations. Introduces self-exploration + view graph distillation framework that converts explorationImitation & Reinforcement LearningExplorationTrying different actions to discover useful behavior. trajectories into supervised learningRobot LearningSupervised learningLearning from labeled input-output examples. signal, achieving significant gains on Qwen2.5-VL-7B (2.5%→47.8%). Read the paper by tracking the taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. definition, the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. or data assumptions, and the evidence that supports the claimed improvement.

HOW IT WORKS

1

Task framing

The paper frames the work as perceptionPerception & SensingPerceptionThe process of turning raw sensor data into useful understanding of the world.. Start here because it defines what success means and which assumptions the rest of the method inherits.

Show more ↓

Figure 1 : Overview of ViewSuite . Left : Point cloud environment built on ScanNet with viewpoints (initial in blue, target in orange) and rendered first-person views. Middle : Path-to-View (P2V; predict the resulting view from an action sequence) and View-to-Path (V2P; infer the action sequence from two views), both single-turn. Right : Interactive View Planning (IVP), a multi-turn task where the agent plans view changes to match the target view and submits a viewpoint estimate.

Figure 8 : IVP example. Our trained Qwen2.5-VL-7B plans view changes from (b) to match (a) in 6 turns. Final viewpoint error: 0.061 m / 0^{\circ} . Success.

Figure 9 : Spearman \rho between sample-level factors (rows) and per-model success (columns). Factors grouped into geometric distance, visual overlap, and directional geometry.

2

Core method

This paper reveals that VLMs can't plan multi-step camera movements in 3D scenes—they understand single moves but fail composing them across sequences. By having a VLMModern Robot LearningVision-Language Model (VLM)A model that understands both images and text. self-explore scenes and distill explorationImitation & Reinforcement LearningExplorationTrying different actions to discover useful behavior. trajectories into a view graph, the authors boost planningControl & PlanningPlanningFiguring out what the robot should do before or during movement. success from 2.5% to 47.8%, enabling VLMs to actively reason about viewpoint sequences in 3D space. When reading the method section, identify the inputs, the learned or engineered representation, and the actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. or prediction produced by the system.

Show more ↓

Figure 2 : Success rate vs. rotation distance (top) and position distance (bottom) for proprietary models across all three tasks.

Figure 4 : Point cloud coverage across planning turns. (a) Scene coverage. (b) Target intersection.

Figure 5 : Distribution of unified view distance d across 530 test pairs. The threshold d=3 separates Short (185 pairs) and Long (345 pairs) subsets.

Figure 10 : Action frequency distribution across iterations. The base policy (iter 0) favors move_forward ; later iterations shift toward rotations and more balanced translations.

3

Data and supervision

For robotics work, the data story is part of the method: check whether the system depends on teleoperationImitation & Reinforcement LearningTeleoperation (teleop)A human remotely controlling the robot, often to collect demonstrations., simulationSimulation & Sim-to-RealSimulationA virtual environment where robots can be trained or tested., internet video, human labels, or robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. rollouts.

Show more ↓

Figure 3 : Iterative training pipeline. Left : in the self-exploration stage, the agent actively explores ViewSuite environments under sparse outcome rewards; completed trajectories are continuously compressed into a view graph whose nodes are viewpoints and whose edges are actions. Right : in the view graph distillation stage, paths sampled from the graph are reformulated into multi-turn view-planning demonstrations and auxiliary supervision (view-difference estimation and multiple-choice). The distilled model initializes the self-exploration iteration, progressively bootstrapping the policy.

4

Evaluation evidence

The paper should be judged through its evaluationSimulation & Sim-to-RealEvaluationMeasuring how well a robot system performs. protocol: what data is used, what robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. or simulator is tested, and which baselineEvaluation & ResearchBaselineA reference method used for comparison. comparisons support the claim. Look for the gap between the headline result and the deploymentSimulation & Sim-to-RealDeploymentPutting the trained system on a real robot. setting you would actually care about.

Show more ↓

Figure 12 : (a) Total rollouts by turn count. (b) Successful rollouts. (c) Success rate.

FIGURES

Figure 6 : P2V example. Action: [ turn_right \times 5]. GPT-5.4 Pro selects C (incorrect).

Figure 7 : V2P example. GPT-5.4 Pro selects B (correct): [ turn_left \times 5, move_left ].

KEY RESULTS

WHY DEVELOPERS SHOULD CARE

LIMITATIONS

The main limitation to check is whether the claimed behavior holds outside the paper's reported setup. That means testing across different robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. embodiments, scenes, objects, and data distributions.

WHAT COMES NEXT

The practical next step is independent reproduction with clear baselines, ablations, and stress tests. For a developer, the useful follow-up is to map the paper's perceptionPerception & SensingPerceptionThe process of turning raw sensor data into useful understanding of the world. assumptions onto a concrete robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. stack, then test the smallest version of the method that could run end to end.

RELATED PAPERS