Course navigation
Week 6: Frontier EmbodimentDay 40
Dexterous manipulation: Eureka, DexMimicGen, π0.7 dexterous
This is a valid v1.0 placeholder page for the later curriculum arc. Full interactive lab treatment ships after Week 1 dogfooding.
LECTURE & READING
Glossary primer (12 min)
- Manipulation & TasksDexterous manipulationHighly precise object handling, usually with fingers or complex contact. — Multi-finger / multi-contact Manipulation & TasksGraspingTaking hold of an object., Manipulation & TasksReorientationChanging an object’s orientation in-hand or on a surface., Manipulation & TasksIn-hand manipulationManipulating an object within the robot hand without putting it down.. The hardest robotics problem. 16-25 DoFs in the hand alone.
- Eureka — NVIDIA 2023. Uses LLMs (GPT-4) to write Imitation & Reinforcement LearningRewardA score that tells the robot how well it is doing. functions for dexterous tasks; iterates with Core ConceptsExecutionActually carrying out planned or predicted actions on the robot. Control & PlanningFeedbackInformation returned from sensors during action to help correct behavior.. Solved pen spinning in sim (a long-standing open problem).
- DexMimicGen — NVIDIA 2024. Generates synthetic dexterous demonstrations by "mimicking" base skills with augmentation. Avoids the 1000-human-hours-of-teleop bottleneck.
- Allegro Hand / Shadow Hand / LEAP Hand — Common dexterous platforms. Allegro: 16 DoFs / 4 fingers; Shadow: 24 DoFs / 5 fingers; LEAP: low-cost 16 DoFs / 4 fingers.
- In-hand Manipulation & TasksReorientationChanging an object’s orientation in-hand or on a surface. — Hold an object, rotate it without dropping. Classic Simulation & Sim-to-RealBenchmarkA standard test used to compare methods fairly..
- Object-centric vs hand-centric obs — Object pose in world frame vs object pose relative to hand. Choice affects Modern Robot LearningGeneralizationThe robot’s ability to work in new situations it has not seen before..
- Perception & SensingTactile sensingTouch sensing through fingers, skin, or contact surfaces. — Force-sensing fingertips (e.g. DIGIT, GelSight). Crucial for fine Manipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects.; many policies don't use it.
- π0.7-dexterous — Apr 2026 variant of π0.7 fine-tuned for bimanual dexterous tasks (towel folding, container opening). Within the same blog post / release.
Hour 1 — Reading
- Eureka paper, sections 1–4 (~30 min): https://eureka-research.github.io/
- DexMimicGen paper, abstract + Section 3 (~25 min): https://dexmimicgen.github.io/
- π0.7 dexterous results section (~10 min, from Day 30 reading): https://www.physicalintelligence.company/blog/pi07
Hour 2 — Run Eureka on a toy task (45 min)
Eureka requires GPT-4 API access. Skip if you don't have it; otherwise:
git clone https://github.com/eureka-research/Eureka
cd Eureka
export OPENAI_API_KEY=<key>
python eureka.py --env=shadow_hand_pen_spin --iterations=3After 3 iterations (~30 min, mostly waiting on GPT-4), the LLM should have proposed a Imitation & Reinforcement LearningReward functionThe rule that defines how rewards are assigned. that yields a working pen-spinning Core ConceptsPolicyThe rule or model that maps observations or states to actions. in Isaac Gym. Expected: pen orientation tracks target with average angular error < 0.5 rad after 100 sim hours.
If no API access, read the paper's appendix carefully — the evolved Imitation & Reinforcement LearningReward functionThe rule that defines how rewards are assigned. is the key result.
LAB
Hour 3 — Lab: minimal in-hand reorientation in MuJoCo (75 min)
What you're building. A simplified in-hand cube Manipulation & TasksReorientationChanging an object’s orientation in-hand or on a surface. Core ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. on the Allegro Hand in MuJoCo. Train PPO for 30 min. Don't expect full Manipulation & TasksReorientationChanging an object’s orientation in-hand or on a surface. — just confirm the cube doesn't drop.
Step 1 — Get the Allegro Hand model (10 min)
ls ~/robo47/mujoco_menagerie/wonik_allegroExpected: Allegro hand assets including right_hand.xml.
Full source continues in the committed curriculum files. The v1.0 page exposes the day flow and lab surface without inventing content.