Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

ARCHITECTURE

THE PROBLEM

Before Diffusion PolicyModern Robot LearningDiffusion policyA robot policy that generates actions using diffusion-model techniques., robot learningRobot LearningRobot learningUsing data and algorithms to help robots improve behavior instead of only relying on hand-written rules. methods like behavior cloningImitation & Reinforcement LearningBehavior Cloning (BC)A simple type of imitation learning where the robot directly copies expert actions. with LSTM-GMM (Long Short-Term Memory Gaussian Mixture Models) and IBC (Implicit Behavior CloningImitation & Reinforcement LearningBehavior Cloning (BC)A simple type of imitation learning where the robot directly copies expert actions.) had a critical flaw: they struggled when tasks had multiple valid solutions. Imagine a pizza-making robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. spreading sauce—there are infinitely many valid patterns that work, but traditional policies would average them together, producing mediocre trajectories. Meanwhile, transformer-based methods like BET (Behavior Transformer) could predict actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. sequences but failed to "commit" to a single solution, hedging bets across all possibilities and causing failed executions. On top of this, as actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. spaces grew larger (like controlling 6 degrees of freedomMovement, Mechanics & Robot BodyDegrees of Freedom (DoF)The number of independent ways a robot can move. for a robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. arm), existing methods became increasingly unstable during trainingRobot LearningTrainingThe process of fitting a model using data or experience.. The field had no principled way to leverage the rich generative capabilities that were revolutionizing computer vision—robot learningRobot LearningRobot learningUsing data and algorithms to help robots improve behavior instead of only relying on hand-written rules. was stuck using tools designed for classification, not generation.

HOW IT WORKS

1

Reformulate Robot Policy as a Denoising Process

Instead of asking "what actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. should the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. take given this image?", Diffusion PolicyModern Robot LearningDiffusion policyA robot policy that generates actions using diffusion-model techniques. flips the question: "starting from random noiseData, Distributions & Training IssuesNoiseUnwanted variation or randomness in sensor readings or actuation., what sequence of actions best explains this observed scene?" During trainingRobot LearningTrainingThe process of fitting a model using data or experience., the model learns to gradually denoise noisy actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. sequences conditioned on visual input, learning the gradient (score function) of the actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. distribution. This is borrowed directly from how diffusion models generate images, but here it's applied to actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. sequences. The genius is that this naturally captures multimodality—the denoising process can learn multiple peaks in the actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. distribution and explore them during inferenceRobot LearningInferenceUsing a trained model to make predictions or choose actions.. For tasks with multiple solutions (like sauce spreading), this means the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. learns all valid strategies and can pick one intelligently.

Show more ↓

Figure 4: Velocity v.s. Position Control. The performance difference when switching from velocity to position control. While both BCRNN and BET performance decrease, Diffusion Policy is able to leverage the advantage of position and improve its performance.

Figure 8: Robustness Test for Diffusion Policy. Left: A waving hand in front of the camera for 3 seconds causes slight jitter, but the predicted actions still function as expected. Middle: Diffusion Policy immediately corrects shifted block position to the goal state during the pushing stage. Right:

highlight pusht process

highlight mug

highlight sauce

highlight pusht

2

Time-Series Diffusion Transformer Architecture

The authors designed a specialized transformer that operates on actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. sequences rather than single actions. It processes the entire planned actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. trajectoryCore ConceptsTrajectoryA sequence of states or actions over time. (typically 16 actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. steps into the future) as a time-series, allowing the model to maintain smooth, physically-plausible motion. Each transformer block incorporates visual conditioning—the image observationCore ConceptsObservationThe information the robot receives from sensors, such as images, depth, touch, or joint readings. is embedded and cross-attended at every step. This is critical because robots need to react continuously to what they see. The time-series formulation matters because it forces smooth actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. predictions; unlike methods that predict one actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. at a time, this predicts a coherent plan and can use Langevin dynamicsMovement, Mechanics & Robot BodyDynamicsThe study of motion including forces, torques, mass, and inertia. (a technique from physics) to iteratively refine it during executionCore ConceptsExecutionActually carrying out planned or predicted actions on the robot..

Show more ↓

Figure 2: Diffusion Policy Overview a) General formulation. At time step tt, the policy takes the latest ToT_{o} steps of observation data OtO_{t} as input and outputs TaT_{a} steps of actions AtA_{t}. b) In the CNN-based Diffusion Policy, FiLM (Feature-wise Linear Modulation) Perez et al. (2018) co

3

Receding Horizon Control for Real-World Execution

During real robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. operation, the system doesn't commit to the full 16-step plan. Instead, it executes only the first few actions, then re-plans based on the new visual observationCore ConceptsObservationThe information the robot receives from sensors, such as images, depth, touch, or joint readings.. This is the receding horizon technique—a proven controlControl & PlanningControlThe method used to make the robot move the way you want. strategy that makes policies robust to mistakes and disturbances. If a human bumps the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. or an object shifts slightly, the next re-plan corrects course. The paper shows this is essential for real-world success: the Push-T taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. videos demonstrate the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. remaining robust against hand occlusions and external perturbations specifically because of continuous re-planning. This design choice bridges the gap between offline policy learningImitation & Reinforcement LearningPolicy learningTraining a model that maps observations to actions. and online reactive controlControl & PlanningReactive controlControl that responds immediately to sensor input or disturbances..

Show more ↓

Figure 3: Multimodal behavior. At the given state, the end-effector (blue) can either go left or right to push the block. Diffusion Policy learns both modes and commits to only one mode within each rollout. In contrast, both LSTM-GMM Mandlekar et al. (2021) and IBC Florence et al. (2021) are biased

Figure 5: Diffusion Policy Ablation Study. Change (difference) in success rate relative to the maximum for each task is shown on the Y-axis. Left: trade-off between temporal consistency and responsiveness when selecting the action horizon. Right: Diffusion Policy with position control is robust agai

Figure 6: Training Stability. Left: IBC fails to infer training actions with increasing accuracy despite smoothly decreasing training loss for energy function. Right: IBC’s evaluation success rate oscillates, making checkpoint selection difficult (evaluated using policy rollouts in simulation).

Table 6: Realworld Push-T Experiment. a) Hardware setup. b) Illustration of the task. The robot needs to 1⃝ precisely push the T-shaped block into the target region, and 2⃝ move the end-effector to the end-zone. c) The ground truth end state used to calculate Io

4

Visual Conditioning and Perception Integration

The policyCore ConceptsPolicyThe rule or model that maps observations or states to actions. doesn't operate in some abstract feature space—it directly conditions on RGB images from the robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions.'s camera. During trainingRobot LearningTrainingThe process of fitting a model using data or experience., visual encoders (pre-trained vision models like R3M) extract features that the diffusion model uses to condition actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. denoising. This end-to-end visual grounding is crucial for sim-to-realSimulation & Sim-to-RealSim-to-real (sim2real)Transferring a policy trained in simulation to a real robot. transfer and taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before.. The project page highlights real-world successes on Push-T (pushing blocks precisely), Mug Flipping (complex 6-DOF manipulationManipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects. with orientation constraints), and Sauce Pouring (fluid manipulationManipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects. with periodic motions)—all learned from visual observationCore ConceptsObservationThe information the robot receives from sensors, such as images, depth, touch, or joint readings. alone, no depth sensors or state estimationPerception & SensingState estimationCombining noisy sensor data to estimate the robot’s true state. required.

Show more ↓

Table 1: Behavior Cloning Benchmark (State Policy) We present success rates with different checkpoint selection methods in the format of (max performance) / (average of last 10 checkpoints), with each averaged across 3 training seeds and 50 different environment initial conditions (150 in total). LS

MORE DEMONSTRATIONS

FIGURES (6 of 8)

Figure 9: 6DoF Mug Flipping Task. The robot needs to 1⃝ Pickup a randomly placed mug and place it lip down (marked orange). 2⃝ Rotate the mug such that its handle is pointing left.

Figure 10: Realworld Sauce Manipulation. [Left] 6DoF pouring Task. The robot needs to 1⃝ dip the ladle to scoop sauce from the bowl, 2⃝ approach the center of the pizza dough, 3⃝ pour sauce, and 4⃝ lift the ladle to finish t

Figure 11: Bimanual Egg Beater Manipulation. The robot needs to 1⃝ push the bowl into position (only if too close to the left arm), 2⃝ approach and pick up the egg beater with the right arm, 3⃝ place the egg beater in the bowl, {-0.9p

Figure 12: Bimanual Mat Unrolling. The robot needs to 1⃝ pick up one side of the mat (if needed), using the left or right arm, 2⃝ lift and unroll the mat (if needed), 3⃝ ensure that both sides of the mat are grasped, 4⃝ lift

Figure 13: Bimanual Shirt Folding. The robot needs to 1⃝ approach and grasp the closest sleeve with both arms, 2⃝ fold the sleeve and release, 3⃝ drag the shirt closer (if needed), 4⃝ approach and grasp the other sleeve with

Figure 16: Initial states for Mat Unrolling

KEY RESULTS

PERFORMANCE COMPARISON

WHY DEVELOPERS SHOULD CARE

LIMITATIONS

The paper doesn't deeply explore failure modes or fundamental limitations. One implicit constraintControl & PlanningConstraintA rule the robot must obey, such as avoiding collisions or staying within joint limits.: all tasks shown are manipulation-focused; generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before. to locomotionNavigation & LocomotionLocomotionMovement of the robot body through space, like walking, rolling, or running. or other domains is untested. Diffusion inferenceRobot LearningInferenceUsing a trained model to make predictions or choose actions. requires iterative denoising steps (typically multiple passes through the model), which is slower than single-shot policyCore ConceptsPolicyThe rule or model that maps observations or states to actions. methods—this matters for real-time controlSimulation & Sim-to-RealReal-time controlProducing actions fast enough for live robot control. where latencySimulation & Sim-to-RealLatencyDelay between input, computation, and action. is critical. The real-world experiments, while compelling, are still limited: Push-T, Mug Flipping, and Sauce tasks represent 3 real-world domains, whereas the simulated benchmarks span 12 tasks. Real-world generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before. (sim-to-realSimulation & Sim-to-RealSim-to-real (sim2real)Transferring a policy trained in simulation to a real robot. transfer) is briefly mentioned but not thoroughly evaluated—questions remain about how much domain randomizationData, Distributions & Training IssuesDomain randomizationChanging simulator visuals or physics during training so policies transfer better to reality. is needed, how the method scales to new objects or unseen lighting conditions, and whether visual pre-training (R3M) is strictly necessary or if trainingRobot LearningTrainingThe process of fitting a model using data or experience. from scratch works. The paper also doesn't discuss computational cost during trainingRobot LearningTrainingThe process of fitting a model using data or experience. or inferenceRobot LearningInferenceUsing a trained model to make predictions or choose actions.; diffusion models are generally more expensive than their discriminative counterparts. Finally, the reliance on pre-trained vision models (R3M) for real-world success hints at a dependency on good feature learning that may not always be available for novel robots or domains.

WHAT COMES NEXT

The immediate direction is clear from the paper's own hints: scaling to longer-horizon tasks (beyond 16-step planningControl & PlanningPlanningFiguring out what the robot should do before or during movement. windows), experimenting with different diffusion schedules and noiseData, Distributions & Training IssuesNoiseUnwanted variation or randomness in sensor readings or actuation. levels to trade off planningControl & PlanningPlanningFiguring out what the robot should do before or during movement. flexibility vs. commitment, and testing on robots beyond Franka arms (the real-world experiments use specific hardware). Longer term, the field will likely explore hybrid approaches combining diffusion policies with reinforcement learningImitation & Reinforcement LearningReinforcement Learning (RL)Teaching a robot through trial and error using rewards. fine-tuningModern Robot LearningFine-tuningTaking a pretrained model and adapting it to a specific robot or task., enabling both imitation learningImitation & Reinforcement LearningImitation Learning (IL)Teaching a robot by showing it examples of how to do a task. data efficiencyRobot LearningData efficiencyHow much useful performance a method gets from limited data. and rewardImitation & Reinforcement LearningRewardA score that tells the robot how well it is doing. optimization. Model-based extensions (learning world models + planningControl & PlanningPlanningFiguring out what the robot should do before or during movement. in actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. space with diffusion) are natural follow-ups. The foundation is set for diffusion to become the standard approach to visuomotor policy learningImitation & Reinforcement LearningPolicy learningTraining a model that maps observations to actions., similar to how it displaced GANs in image generation. The open-sourcing of code, data, and notebooks means the community will rapidly iterate—expect ablations on architectural choices, scaling laws for actionCore ConceptsActionA command the robot sends to its motors, controller, or low-level system. sequence length and image resolution, and applications to underexplored domains like in-hand manipulationManipulation & TasksIn-hand manipulationManipulating an object within the robot hand without putting it down. or soft robotics. The key technical insight—that diffusion's score-matching objective naturally handles distribution multimodality—will likely influence how future generative models are applied to sequential decision-making problems beyond robotics.

RELATED PAPERS