π0.7: a Steerable Model with Emergent Capabilities

The breakthrough insight is that you can't just throw diverse data at a model and expect it to generalize. Instead, π0.7 uses rich, structured prompts with multiple modality channels during trainingRobot LearningTrainingThe process of fitting a model using data or experience.. Beyond simple text instructions ("fold the shirt"), the model learns from prompts that include: visual subgoal images showing the desired end-state of each sub-step, metadata about executionCore ConceptsExecutionActually carrying out planned or predicted actions on the robot. speed and quality, controlControl & PlanningControlThe method used to make the robot move the way you want. modality labels (whether to use jointMovement, Mechanics & Robot BodyJointA movable connection between robot parts. or end-effectorMovement, Mechanics & Robot BodyEnd-effectorThe tool at the end of a robot arm, like a gripper, hand, or suction cup. controlControl & PlanningControlThe method used to make the robot move the way you want.), and descriptions of individual sub-steps. This diversity of conditioning signals acts as a data annotationData, Distributions & Training IssuesAnnotationHuman-provided labels or metadata attached to data. system that disambiguates how the same taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. can be performed in different ways. A suboptimal autonomous trajectoryCore ConceptsTrajectoryA sequence of states or actions over time. can be labeled as low-quality, so the model learns which behaviors to prefer without filtering out the data entirely. At test time, the model accepts standard language, but can also accept synthetically generated visual subgoals from a lightweight world modelModern Robot LearningWorld modelA model that predicts how the world will change after actions., enabling zero-shotModern Robot LearningZero-shotDoing a new task without task-specific training. visual generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before. to new scenes.

π0.7 unifies multiple data sources under a single prompting framework: multi-robot data (mobile manipulators, bimanual UR5e arms, various embodiments), human teleoperationImitation & Reinforcement LearningTeleoperation (teleop)A human remotely controlling the robot, often to collect demonstrations. videos, and autonomous data collected from running different policies. The key challenge is that these sources have different quality levels, controlControl & PlanningControlThe method used to make the robot move the way you want. conventions, and success rates. The multimodalModern Robot LearningMultimodalUsing more than one type of input, like vision, language, touch, or proprioception. conditioning approach solves this by allowing the model to learn from suboptimal data without degrading performance. For example, autonomous explorationImitation & Reinforcement LearningExplorationTrying different actions to discover useful behavior. data that achieved 40% success can be included in trainingRobot LearningTrainingThe process of fitting a model using data or experience. with quality annotations, and the model learns to extract useful patterns while not copying failure modes. This creates a virtuous cycle: more diverse data sources improve compositional generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before. without requiring careful curation or data filtering.

At inference timeEvaluation & ResearchInference timeHow long the model takes to produce an output., π0.7 can accept visual subgoals generated on-the-fly by a lightweight world modelModern Robot LearningWorld modelA model that predicts how the world will change after actions. rather than requiring pre-annotated demonstrationImitation & Reinforcement LearningDemonstrationAn example of a task being done correctly, often by a human. data. This is powerful because it means the model can work in novel scenes and with new objects without needing example videos. The world modelModern Robot LearningWorld modelA model that predicts how the world will change after actions. predicts what the scene will look like after each intermediate step, creating a visual roadmap for the policyCore ConceptsPolicyThe rule or model that maps observations or states to actions.. This breaks the dependency on having demonstrationImitation & Reinforcement LearningDemonstrationAn example of a task being done correctly, often by a human. data for every new scenario, enabling true zero-shotModern Robot LearningZero-shotDoing a new task without task-specific training. compositional generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before.. For instance, the model can fold laundry on a new robotCore ConceptsRobotA physical system with sensors and actuators that can observe the world and take actions. using the same world modelModern Robot LearningWorld modelA model that predicts how the world will change after actions. predictions it would use for other manipulationManipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects. tasks, even though there's zero laundry folding data in trainingRobot LearningTrainingThe process of fitting a model using data or experience..

π0.7 isn't just a single monolithic predictor—it's designed to accept steering signals at test time that controlControl & PlanningControlThe method used to make the robot move the way you want. how it performs. You can specify desired executionCore ConceptsExecutionActually carrying out planned or predicted actions on the robot. speed, taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. strategy, or visual subgoals, and the model adapts its behavior accordingly. This steerability is critical for compositional generalizationModern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before. because it means the model doesn't just memorize task-specific behaviors; it learns underlying manipulationManipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects. principles that can be recombined with different executionCore ConceptsExecutionActually carrying out planned or predicted actions on the robot. parameters. This is why the same model can use a new kitchen appliance by applying learned interaction skills with different spatial targets, or fold different clothing items by adjusting its approach based on visual feedbackControl & PlanningFeedbackInformation returned from sensors during action to help correct behavior..