Course navigation
Week 7: CapstoneDay 43
Pick a track + scaffold
This is a valid v1.0 placeholder page for the later curriculum arc. Full interactive lab treatment ships after Week 1 dogfooding.
LECTURE & READING
The four tracks
Track A — VLA fine-tune (most general, recommended for first capstone)
Core ConceptsGoalThe desired outcome or target state for a robot task.: Fine-tune π0.7 on a custom 30-episode Robot LearningDatasetA collection of training or evaluation data.. Beat Modern Robot LearningZero-shotDoing a new task without task-specific training. by ≥ 30%.
Compute: 1× H100 for 6 GPU-hours.
- Ship:
- 30-episode Robot LearningDatasetA collection of training or evaluation data. (sim or SO-101 hardware)
- Modern Robot LearningZero-shotDoing a new task without task-specific training., LoRA r=16, LoRA r=128 results across 3 seeds
- Bar plot, eval video, Evaluation & ResearchAblationAn experiment where one component is removed to see its effect.: rank vs Simulation & Sim-to-RealSuccess rateHow often the robot completes a task correctly.
Use this if: you want to land at "I can fine-tune frontier VLAs on my own data." Straightforward path; lowest risk.
Track B — World model (most research-flavored)
Core ConceptsGoalThe desired outcome or target state for a robot task.: Train a 100M-param JEPA-style Modern Robot LearningWorld modelA model that predicts how the world will change after actions. on 50h of Spot rollouts. Quantify prediction quality vs horizon. Stretch: deploy in a Dreamer-style Imitation & Reinforcement LearningReinforcement Learning (RL)Teaching a robot through trial and error using rewards. loop.
Compute: 1× H100 for 18 GPU-hours.
- Ship:
- 50h Spot Robot LearningRolloutA full run of a policy in simulation or the real world. Robot LearningDatasetA collection of training or evaluation data. (you generate it Day 44)
- Three model sizes (10M / 100M / 300M params)
- Cosine-vs-horizon plot
- Stretch: small Imitation & Reinforcement LearningReinforcement Learning (RL)Teaching a robot through trial and error using rewards. improvement using your model
Use this if: you're aiming at JEPA / world-model research. Highest research-paper density per hour.
Track C — Sim-to-real locomotion on Go1
Core ConceptsGoalThe desired outcome or target state for a robot task.: Reproduce Day 25 + Day 27 (DR + RMA) with proper seeds and an Evaluation & ResearchAblationAn experiment where one component is removed to see its effect.. If you have hardware, deploy on real Go1.
Compute: 1× H100 for 12 GPU-hours.
- Ship:
- Three policies: no-DR, DR-only, DR+RMA
- 3 seeds each, full perturbed-eval protocol
- Real-Go1 video if applicable (else: extreme-perturbation sim eval)
Use this if: you want a clean Simulation & Sim-to-RealSim-to-real (sim2real)Transferring a policy trained in simulation to a real robot. story. Best if you have access to a Go1.
Track D — Dexterous bimanual ACT/π0.5
Core ConceptsGoalThe desired outcome or target state for a robot task.: Train ACT or π0.5 on a 50-episode bimanual dexterous Robot LearningDatasetA collection of training or evaluation data. (cup-and-saucer, pour-water). Compare against single-arm Imitation & Reinforcement LearningBehavior Cloning (BC)A simple type of imitation learning where the robot directly copies expert actions..
Compute: 1× H100 for 8 GPU-hours.
- Ship:
- 50-episode bimanual Robot LearningDatasetA collection of training or evaluation data.
- ACT, π0.5 LoRA, Imitation & Reinforcement LearningBehavior Cloning (BC)A simple type of imitation learning where the robot directly copies expert actions. Evaluation & ResearchBaselineA reference method used for comparison. — 3 seeds each
- Eval video showing successful bimanual coordination
Use this if: dexterous research is your Core ConceptsGoalThe desired outcome or target state for a robot task.. Hardest to debug; highest Modern Robot LearningSkillA reusable behavior like grasp, push, place, or open drawer. ceiling.
Day 43 schedule (3 hours)
Hour 1 — Decision (45 min)
1. Re-read your Day 21 (Track A), 28 (Track C), 35 (Track A again), 42 (Track B) design docs. Note which felt most "alive" to you.
2. Score yourself 1-5 on each track's prerequisites:
- Track A: comfort with LeRobot CLI + LoRA (Days 18, 20, 30)
- Track B: comfort with V-JEPA / world models (Days 36, 37, 38)
- Track C: comfort with PPO + DR + RMA (Days 22-27)
- Track D: comfort with ACT + bimanual data (Days 16, 30)
3. Pick the highest-score track. Write your hypothesis as the first line of w7-capstone/README.md.
Hour 2 — Project scaffold (60 min)
Create the standard structure:
cd ~/robo47/w7-capstone
mkdir -p {data,runs,figures,videos,docs,src,tests}
cat > README.md <<EOF
# Capstone — Track <X>
## Hypothesis
<paste from your design doc>
## Status
- [ ] Day 43: scaffold + dataset
- [ ] Day 44: train baseline
- [ ] Day 45: train extension
- [ ] Day 46: eval + ablation
- [ ] Day 47: writeup + demo
## Reproduce
\`\`\`
make install
make data
make train
make eval
\`\`\`
EOF
cat > Makefile <<EOF
.PHONY: install data train eval clean
install:
uv venv --python 3.12 .venv
. .venv/bin/activate && uv pip install -r requirements.txt
data:
. .venv/bin/activate && python src/collect_data.py
train:
. .venv/bin/activate && python src/train.py
eval:
. .venv/bin/activate && python src/eval.py --seeds 1,2,3
clean:
rm -rf runs/ figures/ wandb/
EOF
cat > requirements.txt <<EOF
# fill in based on your track
torch
numpy
scipy
matplotlib
wandb
EOF
git init && git add -A && git commit -m "Day 43: capstone scaffold"Hour 3 — Day 44 prep (45 min)
Write src/collect_data.py (or download Robot LearningDatasetA collection of training or evaluation data.). For Track A: 30 episodes via Imitation & Reinforcement LearningTeleoperation (teleop)A human remotely controlling the robot, often to collect demonstrations.; for Track B: launch background generation of Spot rollouts; for Track C: skip (use existing); for Track D: 50 episodes bimanual.
For Track B specifically, kick this off in tmux to run overnight:
python src/collect_data.py --hours 50 --policy runs/spot_ppo/policy.pkl &git add -A && git commit -m "Day 43: data-collection scaffold"LAB
Deliverable checklist
---