Track A Lite

What you're shipping

What this isn't

• Not a research contribution. You're reproducing a known result with your own data.
• Not a hardware project. Pure simulationSimulation & Sim-to-RealSimulationA virtual environment where robots can be trained or tested..
• Not a deep-dive on architecture. We treat SmolVLA and LoRA as black boxes.

Why SmolVLA + LIBERO-Spatial?

• SmolVLA (Hugging Face, 2025): a 2.4B-parameter Vision-Language-Action modelModern Robot LearningVision-Language-Action model (VLA)A model that takes images and language as input and outputs robot actions. designed for fine-tuningModern Robot LearningFine-tuningTaking a pretrained model and adapting it to a specific robot or task. on consumer GPUs. Open weights, well-documented, integrated with LeRobot's CLI.
• LIBERO-Spatial: a benchmarkSimulation & Sim-to-RealBenchmarkA standard test used to compare methods fairly. of 10 spatial-relation manipulationManipulation & TasksManipulationUsing a robot arm or hand to move or interact with objects. tasks ("put the bowl on the right of the plate"). Standard, fast to evaluate, has known baselineEvaluation & ResearchBaselineA reference method used for comparison. numbers — meaning you can sanity-check your results against published work.
• LoRA (Low-Rank Adaptation): a fine-tuningModern Robot LearningFine-tuningTaking a pretrained model and adapting it to a specific robot or task. technique that adds small trainable matrices to attention layers while freezing the base. Reduces fine-tune memory by ~5× — fits in 30GB GPU memory.

Compute and cost

• 1× H100 80GB for ~6 hours total (3 fine-tunes × ~75 min, plus evals)
• Substitutes: 1× A100 80GB works identically. 1× L40S works with smaller batches. 1× consumer GPU (24GB+) works with rank=16 instead of rank=32.
• Provider: Nebius, Lambda Labs, RunPod, Vast.ai. Budget ~$15–25 if you're efficient. Or use existing cloud credits.

Day 0 — Environment setup (~30 min, before Day 1)

Install on local laptop (for editing)

curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
uv python install 3.12
mkdir -p ~/track-a-lite && cd ~/track-a-lite
git init

Provision GPU instance

ssh -i ~/.ssh/<your-key> ubuntu@<instance-ip>

On the GPU instance

# Tooling
sudo apt update && sudo apt install -y tmux htop nvtop git build-essential ffmpeg \
    libgl1 libegl1 libglfw3 libosmesa6 libgles2-mesa-dev pkg-config

# uv
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc

# Verify GPU
nvidia-smi

Project workspace on GPU

mkdir -p ~/track-a && cd ~/track-a
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install "lerobot[all]==0.5.0"
uv pip install wandb
wandb login   # paste API key from https://wandb.ai/authorize

Sanity-check LeRobot is alive

python -c "import lerobot; print(f'LeRobot {lerobot.__version__}')"
lerobot-train --help | head -20

Use tmux

tmux new -s track-a
# Detach with Ctrl-b d
# Reattach later with: tmux attach -t track-a

Day 1 — Zero-shot baseline + first fine-tune (3–4 hours active, more in background)

Step 1: Inspect the dataset (10 min)

python -c "
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset('lerobot/libero_spatial')
print(f'Episodes: {ds.num_episodes}, frames: {ds.num_frames}')
print(f'Sample keys: {list(ds[0].keys())[:8]}')
print(f'Action shape: {ds[0][\"action\"].shape}')
"

Episodes: 432, frames: 52000
Sample keys: ['observation.images.image', 'observation.images.wrist_image', 'observation.state', 'action', 'episode_index', 'frame_index', 'timestamp', 'next.done']
Action shape: torch.Size([7])

Step 2: Zero-shot eval (~30 min)

mkdir -p runs figures videos

lerobot-eval \
  --policy.path=lerobot/smolvla_base \
  --env.type=libero --env.task_suite=libero_spatial \
  --eval.n_episodes=50 \
  --output_dir=runs/zeroshot \
  --seed=1

INFO  Loading policy from lerobot/smolvla_base
INFO  Loaded 2,401,300,000 parameters
INFO  Initializing libero_spatial env suite...
INFO  Episode 1/50: success=False, length=235
INFO  Episode 2/50: success=True, length=189
...

INFO  eval/success_rate: 0.36
INFO  eval/episode_length: 232.4
INFO  Wrote runs/zeroshot/eval_summary.json

mkdir -p logs
echo "zeroshot,seed=1,success_rate=0.36,n_episodes=50" >> logs/results.csv

Step 3: First fine-tune — LoRA r=32, seed=1 (60–75 min)

lerobot-train \
  --policy.type=smolvla \
  --policy.pretrained_path=lerobot/smolvla_base \
  --policy.lora.enable=true \
  --policy.lora.rank=32 \
  --policy.lora.alpha=64 \
  --policy.lora.target_modules=["q_proj","k_proj","v_proj","o_proj"] \
  --dataset.repo_id=lerobot/libero_spatial \
  --env.type=libero --env.task_suite=libero_spatial \
  --batch_size=4 \
  --gradient_accumulation_steps=2 \
  --steps=10000 \
  --eval_freq=2000 \
  --save_freq=2000 \
  --output_dir=runs/lora_r32_s1 \
  --wandb.enable=true \
  --wandb.project=track-a-lite \
  --seed=1

INFO  Loading dataset: lerobot/libero_spatial
INFO  Trainable params: 31,457,280 / 2,401,331,712 (1.31%)
INFO  step:0 smpl:8 ep:0 epch:0.00 loss:0.487 grdn:1.21 lr:1.0e-04 updt_s:0.812

watch -n 2 nvidia-smi

• Progress checkpoints:
• Step 2000 (~12 min): eval/success_rate: 0.55 (anywhere 0.45–0.65 normal)
• Step 6000 (~40 min): eval/success_rate: 0.71
• Step 10000 (~70 min): eval/success_rate: 0.79

Step 4: While seed 1 trains — set up reproducibility scaffold (45 min in background)

cd ~/track-a-lite

cat > README.md <<'EOF'
# Track A Lite: SmolVLA + LoRA on LIBERO-Spatial

## Hypothesis
Fine-tuning SmolVLA on LIBERO-Spatial via LoRA r=32 yields ≥ 30 percentage point
improvement in success rate over zero-shot SmolVLA.

## Headline result
| Variant | Success rate (mean ± std) | n seeds |
|---|---|---|
| Zero-shot SmolVLA | 0.36 ± 0.00 | 1 |
| SmolVLA + LoRA r=32 | 0.78 ± 0.04 | 3 |

42 percentage point improvement.

## Reproduce

Wall-clock: ~6 GPU-hours on 1× H100.

## Files
- `Makefile`: install + train + eval targets
- `requirements.txt`: pinned deps
- `scripts/train.sh`: training command
- `scripts/eval.sh`: eval command
- `figures/headline.png`: bar plot
- `videos/eval_episode.mp4`: sample successful rollout
- `logs/results.csv`: all seed-level results

EOF

cat > requirements.txt <<'EOF'
lerobot[all]==0.5.0
wandb
matplotlib
pandas
numpy
EOF

cat > Makefile <<'EOF'
.PHONY: install zeroshot train eval reproduce clean

install:
	uv venv --python 3.12 .venv
	. .venv/bin/activate && uv pip install -r requirements.txt

zeroshot:
	. .venv/bin/activate && bash scripts/zeroshot.sh

train:
	. .venv/bin/activate && bash scripts/train_all_seeds.sh

eval:
	. .venv/bin/activate && bash scripts/eval_all_seeds.sh

reproduce: install eval

clean:
	rm -rf runs/ wandb/ figures/*.png
EOF

mkdir -p scripts figures videos logs

cat > scripts/zeroshot.sh <<'EOF'
#!/bin/bash
set -e
lerobot-eval \
  --policy.path=lerobot/smolvla_base \
  --env.type=libero --env.task_suite=libero_spatial \
  --eval.n_episodes=50 \
  --output_dir=runs/zeroshot --seed=1
EOF

cat > scripts/train_all_seeds.sh <<'EOF'
#!/bin/bash
set -e
for SEED in 1 2 3; do
  lerobot-train \
    --policy.type=smolvla \
    --policy.pretrained_path=lerobot/smolvla_base \
    --policy.lora.enable=true \
    --policy.lora.rank=32 \
    --policy.lora.alpha=64 \
    --policy.lora.target_modules='["q_proj","k_proj","v_proj","o_proj"]' \
    --dataset.repo_id=lerobot/libero_spatial \
    --env.type=libero --env.task_suite=libero_spatial \
    --batch_size=4 \
    --gradient_accumulation_steps=2 \
    --steps=10000 \
    --eval_freq=2000 \
    --save_freq=2000 \
    --output_dir=runs/lora_r32_s${SEED} \
    --wandb.enable=true --wandb.project=track-a-lite \
    --seed=${SEED}
done
EOF

cat > scripts/eval_all_seeds.sh <<'EOF'
#!/bin/bash
set -e
for SEED in 1 2 3; do
  lerobot-eval \
    --policy.path=runs/lora_r32_s${SEED}/checkpoints/last/pretrained_model \
    --env.type=libero --env.task_suite=libero_spatial \
    --eval.n_episodes=50 \
    --output_dir=runs/lora_r32_s${SEED}/eval \
    --seed=${SEED}
done
EOF

chmod +x scripts/*.sh

git add -A && git commit -m "Day 1: Track A Lite scaffold + zero-shot baseline"

Step 5: Confirm seed 1 is healthy (5 min check at ~step 4000)

tmux attach -t track-a   # if you detached
# look for the most recent eval line in the training output

Step 6: While seed 1 finishes — record an eval video (15 min)

lerobot-eval \
  --policy.path=runs/lora_r32_s1/checkpoints/002000/pretrained_model \
  --env.type=libero --env.task_suite=libero_spatial \
  --eval.n_episodes=5 \
  --output_dir=runs/lora_r32_s1/preview_eval \
  --seed=999

ls runs/lora_r32_s1/preview_eval/videos/
# Pick episode_0.mp4

Step 7: When seed 1 completes — log result + start seed 2 (5 min)

SR=$(cat runs/lora_r32_s1/eval_summary.json | python -c "import sys,json; print(json.load(sys.stdin)['eval/success_rate'])")
echo "lora_r32,seed=1,success_rate=${SR}" >> logs/results.csv

# In a fresh tmux pane:
lerobot-train [...] --seed=2 --output_dir=runs/lora_r32_s2

End of Day 1 deliverable check

• Zero-shotModern Robot LearningZero-shotDoing a new task without task-specific training. success rateSimulation & Sim-to-RealSuccess rateHow often the robot completes a task correctly. logged (~0.36)
• Seed 1 LoRA trainingRobot LearningTrainingThe process of fitting a model using data or experience. complete; final success rateSimulation & Sim-to-RealSuccess rateHow often the robot completes a task correctly. logged (~0.78)
• Seeds 2 and 3 launched (running overnight if late)
• One preview video saved
• Repo scaffolded with README, Makefile, scripts
• First commit pushed

Day 2 — Finish seeds, plot, write up, ship (3 hours active)

Step 1: Confirm seeds 2 and 3 finished (10 min)

SR2=$(cat runs/lora_r32_s2/eval_summary.json | python -c "import sys,json; print(json.load(sys.stdin)['eval/success_rate'])")
SR3=$(cat runs/lora_r32_s3/eval_summary.json | python -c "import sys,json; print(json.load(sys.stdin)['eval/success_rate'])")
echo "lora_r32,seed=2,success_rate=${SR2}" >> logs/results.csv
echo "lora_r32,seed=3,success_rate=${SR3}" >> logs/results.csv
cat logs/results.csv

Step 2: Headline bar plot (30 min)

"""Track A Lite: produce the headline bar plot."""
import json, glob
from pathlib import Path
import numpy as np
import matplotlib.pyplot as plt

# Zero-shot
zs_path = Path("runs/zeroshot/eval_summary.json")
zs_sr = json.loads(zs_path.read_text())["eval/success_rate"]

# LoRA seeds
lora_srs = []
for d in sorted(glob.glob("runs/lora_r32_s*/eval_summary.json")):
    lora_srs.append(json.loads(Path(d).read_text())["eval/success_rate"])
lora_srs = np.array(lora_srs)

print(f"Zero-shot:  {zs_sr:.3f}")
print(f"LoRA r=32:  {lora_srs.mean():.3f} ± {lora_srs.std():.3f}  (n={len(lora_srs)})")

fig, ax = plt.subplots(figsize=(7, 5))
labels = ["Zero-shot\nSmolVLA", f"+ LoRA r=32\n(n={len(lora_srs)} seeds)"]
means = [zs_sr, lora_srs.mean()]
stds = [0.0, lora_srs.std()]
colors = ["#888", "#3a86ff"]
bars = ax.bar(labels, means, yerr=stds, capsize=8, color=colors, edgecolor="black")
for bar, m in zip(bars, means):
    ax.text(bar.get_x() + bar.get_width()/2, m + 0.02, f"{m:.2f}",
            ha="center", fontsize=12, fontweight="bold")
ax.set_ylabel("Success rate (LIBERO-Spatial, 50 eval episodes/seed)")
ax.set_ylim(0, 1)
ax.set_title("SmolVLA on LIBERO-Spatial: zero-shot vs LoRA fine-tune")
ax.grid(alpha=0.3, axis="y")
plt.tight_layout()
plt.savefig("figures/headline.png", dpi=150)
print("Wrote figures/headline.png")

python scripts/make_plot.py

Zero-shot:  0.360
LoRA r=32:  0.778 ± 0.041  (n=3)
Wrote figures/headline.png

Step 3: Pick the best eval video (15 min)

ls runs/lora_r32_s1/eval/videos/
# Watch episode_0 through episode_4

cp runs/lora_r32_s1/eval/videos/episode_3.mp4 videos/eval_episode.mp4

Step 4: 1-page writeup (30 min)

# SmolVLA + LoRA on LIBERO-Spatial

## TL;DR
LoRA fine-tuning lifts SmolVLA from 0.36 to 0.78 on LIBERO-Spatial — a 42pp
improvement, achieved in ~75 min on 1× H100 by training 1.3% of the model's
parameters.

## Setup
- **Model:** `lerobot/smolvla_base` (2.4B params, PaliGemma backbone)
- **Dataset:** `lerobot/libero_spatial` (432 episodes, 10 spatial-relation tasks)
- **Method:** LoRA r=32 α=64, target modules `{q,k,v,o}_proj`
- **Training:** 10k steps, batch 4 × grad-accum 2, AdamW, cosine LR schedule
- **Compute:** 1× H100 80GB, ~75 min/seed × 3 seeds = ~4 GPU-hours
- **Eval:** 50 episodes per seed via LeRobot's LIBERO env wrapper

## Result
| Variant | Success rate | n seeds |
|---|---|---|
| Zero-shot | 0.36 | 1 |
| LoRA r=32 | **0.78 ± 0.04** | 3 |

![Headline](figures/headline.png)

## Why this works
LIBERO tasks are *in-distribution* for SmolVLA's pretraining mix, but the
specific scene compositions (object positions, spatial relations) require
fine-tuning to specialize. LoRA's low-rank adapters are sufficient because
the pretrained features are already good — we're nudging the policy, not
retraining it.

GPU memory peaked at ~26 GB during training. A full fine-tune of 2.4B params
would have taken ~50 GB and likely required gradient checkpointing.

## Limitations
- Single dataset, single benchmark. Results may not transfer to LIBERO-Object
  or LIBERO-Goal without re-tuning.
- 50 eval episodes per seed is the LeRobot default but is on the small side
  for tight error bars; n=200 would be more reliable.
- No ablation across LoRA ranks. r=32 was picked because it's the LeRobot
  default; r=8 might give similar results at lower memory.

## Reproducibility
Repo: `<your-github-url>`
\```
git clone <url>
cd track-a-lite
make install
make reproduce
\```
Headline number reproduces within ±0.05 across fresh-clone runs (env seed
randomness on LIBERO sim).

## Stack
LeRobot v0.5.0, SmolVLA, HuggingFace Transformers, LIBERO sim, PyTorch 2.5,
CUDA 12.4, Python 3.12.

Step 5: Fresh-clone test (30 min)

cd /tmp && rm -rf clone-test
git clone <your-github-url> clone-test
cd clone-test
make install
# Sanity-check zero-shot (~25 min)
bash scripts/zeroshot.sh
# Compare:
SR_FRESH=$(cat runs/zeroshot/eval_summary.json | python -c "import sys,json; print(json.load(sys.stdin)['eval/success_rate'])")
echo "Original: 0.36, Fresh-clone: $SR_FRESH"

# This requires the saved checkpoint to be in the repo or downloaded from HF
# Easier path: skip and just do the zeroshot match

Step 6: Push, share, log (30 min)

cd ~/track-a-lite
git add -A
git commit -m "Track A Lite complete: zero-shot 0.36 -> LoRA 0.78"
git push origin main

• Where to share:
• GitHub repo with the README front and center
• LinkedIn post: "Spent the weekend fine-tuningModern Robot LearningFine-tuningTaking a pretrained model and adapting it to a specific robot or task. a 2.4B-param VLAModern Robot LearningVision-Language-Action model (VLA)A model that takes images and language as input and outputs robot actions. on a robotics benchmarkSimulation & Sim-to-RealBenchmarkA standard test used to compare methods fairly.. 0.36 → 0.78 on LIBERO-Spatial. Repo + writeup: <linkMovement, Mechanics & Robot BodyLinkA rigid body segment between joints.>"
• X / Twitter: same with #robotics #ML
• HuggingFace: push your LoRA adapter to the Hub if you want

End of Day 2 deliverable check

• 3 LoRA seeds done (or 2 with documented reason)
• figures/headline.png exists
• videos/eval_episode.mp4 shows a clean success
• WRITEUP.md is 1 page, has the headline plot inline
• Fresh-clone zero-shotModern Robot LearningZero-shotDoing a new task without task-specific training. reproduces within ±0.05
• Repo public on GitHub
• At least one external share (LinkedIn, X, etc.)

Common failures and fixes

When this is done

• Drop into a portfolio or job application
• Talk through with a recruiter or interviewer ("I fine-tuned a 2.4B VLAModern Robot LearningVision-Language-Action model (VLA)A model that takes images and language as input and outputs robot actions. on a robotics benchmarkSimulation & Sim-to-RealBenchmarkA standard test used to compare methods fairly., got 42pp improvement in 6 GPU-hours, here's the repo")
• Use as a launchpad for a real research project (an ablationEvaluation & ResearchAblationAn experiment where one component is removed to see its effect. across LoRA ranks, a different LIBERO suite, a custom datasetRobot LearningDatasetA collection of training or evaluation data.)

• Try a harder LIBERO suite: LIBERO-Long (long-horizon tasks) — same code, swap libero_spatial for libero_long, expect lower numbers.
• Try a different VLAModern Robot LearningVision-Language-Action model (VLA)A model that takes images and language as input and outputs robot actions.: π0 has the same LeRobot integration. Swap --policy.type=smolvla for --policy.type=pi0. Compare numbers across both, write that up as a follow-up.
• Add a real ablationEvaluation & ResearchAblationAn experiment where one component is removed to see its effect.: Train at LoRA rank ∈ {8, 32, 128} and plot rank vs success_rate. This makes your repo a 1.5-week project instead of a 2-day project, but it's a real research-y comparison.
• Collect your own datasetRobot LearningDatasetA collection of training or evaluation data.: With an SO-101 arm or a teleopImitation & Reinforcement LearningTeleoperation (teleop)A human remotely controlling the robot, often to collect demonstrations. sim setup, record 30 episodes of a custom taskCore ConceptsTaskThe job the robot is supposed to complete, such as pick-and-place, navigation, or drawer opening. and fine-tune on those. This is the real Track A from the full curriculum.