WALL-WM: Carving World Action Modeling at the Event Joints

Figure 1 : Conceptual illustration of modality hierarchy and WALL-WM’s general performance. Left: a stylized alignment landscape over semantic abstraction and spatial-temporal precision. Text provides coarse semantic alignment, vision provides denser spatial-temporal grounding, and action requires fine-grained contact-sensitive precision. Tactile-force input, when available, is treated as an optional contact-rich signal rather than a required modality. Right: WALL-WM shows clear advantages across manipulation-task performance and video-generation metrics, indicating that event-grounded pretrai

Figure 4 : Cross-view masking in WALL-WM ’s view attention. (a) Sight-cone mask. A token pair (u,u^{\prime}) is allowed to attend if and only if their back-projected sight cones share a 3D region in front of both cameras; the resulting binary mask \mathcal{M}_{\text{sc}} is added as a (1-\mathcal{M}_{\text{sc}})(-\infty) attention bias, with intra-view pairs always allowed (matrix diagonal). (b) Tube mask. A spatial window on one view v^{*} is masked across all latent frames—and, with nested probability, also on the InP conditioning channel \mathbf{y} —leaving cross-view attention to the other

Figure 6 : Comparison of three CoT reasoning schedules. (a) Traditional CoT autoregressively emits discrete vocabulary tokens. (b) Latent CoT replaces discrete tokens with continuous latent vectors while preserving the serial dependency. (c) Our staircase latent reasoning relays intermediate hidden states across staggered layer depths, producing parallel continuous CoT latents through a shared projector.

Figure 10 : XRZero-G0 no-embodiment collection rig and its place alongside teleoperation ( 73 ) . (Left) The wearable rig from two views: a VR-tracked headset carries three egocentric cameras (one top + two wrist-side feeds shown), an operator-borne backpack handles synchronization and edge-side compute, and a pair of handheld grippers reproduces the deployment robot’s end-effector geometry; the inset thumbnails show the three on-rig views (left wrist / top / right wrist) that the policy will eventually consume. (Right) XRZero-G0 sits alongside master-slave and VR teleoperation rather than rep

Figure 8 : Data-source map of the WALL-WM dataset. The four quadrants group general internet video, egocentric video, non-embodiment data, and heterogeneous teleoperation/open robot data; the center denotes human-intervention and failure-recovery data used to enrich contact-rich corrections. Data sources include OpenVID ( 55 ) , HD-VILA ( 79 ) , Ego4D ( 25 ) , EPIC-KITCHENS ( 18 ) , DROID ( 40 ) , AgiBot World ( 10 ) , and self-collected data ( 73 ) .

Figure 9 : Distribution statistics of the [video+action] data. The plots show normalized coverage across scene categories, episode-duration ranges, and atomic-action labels. Scene and duration categories are reported as percentage distributions, while atomic actions indicate the percentage of episodes containing each action label.

Figure 11 : Temporal Synchronization of [video+action] layers. We illustrate the process of temporal synchronization for a specific timestep. (a) Calculating the optical-flow motion cube from video data: we grayscale and normalize the camera RGB images, then calculate the optical flow from the video data. (b) Before synchronization: there is a constant lagging between video signal and action signal. (c) We performed temporal synchronization calculating the correlation between the video signal and the action signal, and synchronized the action signal based on the results.

Figure 13 : Caption-granularity distributions for balanced sampling. The five panels compare 30K samples at Instruction, Task (L3), Subtask (L2), Action (L1), and Segment (L0) granularities after action-boundary segmentation. Finer caption levels expose more localized long-tail modes that are consumed by the cluster-balanced dataloader.