UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception

Figure 4 : Fisheye camera intrinsic calibration. (A) Raw fisheye image capturing a planar checkerboard calibration target under a wide field of view. (B) Undistorted image using the estimated intrinsic parameters under the equidistant projection model. This calibration enables precise pixel-to-ray mapping, which is essential for subsequent perception tasks including marker detection, spatial alignment, and multimodal fusion.

Figure 5 : LiDAR–camera extrinsic calibration. (A) Calibration setup in a typical home environment, demonstrating the practicality and ease of deployment only with a calibration board. (B) Design specification of the calibration target, including geometric layout and fiducial markers. (C) Multimodal observations: (i) fisheye image with detected markers and calibration features; (ii) corresponding LiDAR point cloud. (D) Colorized point cloud after applying the estimated extrinsic transformation, where LiDAR points are assigned RGB values by sampling from the corresponding image. This calibratio

Figure 6 : Overview of the UMI-3D LiDAR–inertial odometry system. The system follows an iterated error-state Kalman filtering (ESIKF) framework on differentiable manifolds. IMU measurements drive high-frequency forward propagation, while LiDAR scans are processed through scan recombination and residual computation against a voxelized map representation. The state is iteratively refined through backward propagation and update steps until convergence. The estimated LiDAR trajectory is used to incrementally build a voxel-based map, and is further transformed to obtain the camera pose via the cali

Figure 7 : Unified coordinate system and frame transformations in UMI-3D. The global frame G is initialized as the first IMU frame, and all states are estimated incrementally with respect to this reference. The LiDAR trajectory is directly estimated through LiDAR–inertial odometry, while the camera trajectory is obtained by transforming LiDAR poses using the calibrated extrinsic {}^{L}\mathbf{T}_{C} . The transformation between the LiDAR and IMU frames {}^{I}\mathbf{T}_{L} is fixed and provided by factory calibration, and the transformation between the LiDAR and the tool center point (TCP), {}

Figure 8 : Policy interface and relative action representation in UMI-3D. ( Left ) The policy takes synchronized multimodal observations, including RGB images, relative end-effector (EE) poses, and gripper states, with explicit latency alignment across sensing and actuation streams. A diffusion policy predicts a sequence of future EE poses, which are executed in a receding-horizon manner with temporal ensembling. ( Right ) Actions are represented as relative SE(3) trajectories with respect to the current EE frame, illustrated by overlaid multi-step motions.

Figure 11 : Cup arrangement performance across object combinations. Left: Quantitative results over 8 \times 8 cup–saucer combinations. Each cell corresponds to one object pair evaluated over 10 trials. The number in the top-right corner indicates the percentage of this combination in the 3,500 training demonstrations. The gray value denotes the accumulated score (out of 60), and the red value shows the normalized score. Green boxes denote seen combinations that appear in the training data. Orange boxes represent partially unseen combinations , where either the cup or the saucer is unseen. Red

Figure 12 : Curtain pulling performance under varying conditions. Left: Three representative curtain types used in training and evaluation, along with their occurrence ratios in the 769 demonstration trajectories. For each curtain type, we report the accumulated score (out of 160) and the normalized score across 40 trials. Right: Representative execution sequences for each curtain type, including the initial configuration, grasping of the curtain edge, and pulling motion. The examples illustrate robustness to different material properties, textures, and challenging lighting conditions such as

Figure 14 : Cross-embodiment policy transfer from UMI to UMI-3D. Left: Illustration of the training–deployment pipeline. Policies are trained using the original UMI system and directly deployed on UMI-3D hardware without finetuning. Middle: Quantitative results across 4 \times 4 mouse–pad combinations in unseen environments. Each cell reports the accumulated score (out of 30) and normalized score over 5 trials. Right: Representative execution sequences, including grasping and placement. The results demonstrate that policies trained under the original UMI setup generalize effectively to UMI-3D,

Figure 2 : UMI-3D Demonstration Interface Design. The system adopts a wrist-mounted sensing design that ensures consistent observation between human demonstrations (left) and robot execution (right). A wide-FoV fisheye camera provides a shared observation space across embodiments, while continuous gripper tracking enables precise action recording and control. This design establishes a unified perception-action interface for data collection and policy deployment.

Figure 3 : Hardware-level temporal synchronization in UMI-3D. An STM32 microcontroller serves as a unified clock source for both LiDAR and camera. It provides a 1 Hz PPS signal and a pseudo-GPRMC message to synchronize the LiDAR, while generating a 20 Hz hardware trigger for the camera via frequency division. The LiDAR streams 10 Hz point clouds over Ethernet, and the camera captures 20 Hz RGB images via hardware triggering. Timestamps are shared between drivers to ensure consistent temporal alignment, and all data are recorded in a unified rosbag format.