Colosseum V2: Benchmarking Generalization for Vision Language Action Models

Figure 5: Average change in success rate for each perturbation. The top row illustrates select perturbations for the DualArmStackTwoCubes task. The average is calculated from only environments with a base success rate (no perturbations) of at least 10% to reduces the noise in the aggregate metrics. The raw results as well as a list of which tasks are included in the perturbation averages are included in the Appendix, available on the projects website: https://sites.google.com/usc.edu/colosseum-v2/

Figure 6: The per-task success rate of all models with no perturbations. The x-axis orders tasks by success rate; consequently, the ordering between models may differ, so the i -th success rate does not necessarily correspond to the same environment between different models. ACT outperforms \pi_{0.5} on both test suites.

Figure 1: Overall generalization results. The left plot shows average absolute success rate across the vision, language, and action perturbations, whereas the right plot shows the average decrease in success rate compared to the no-perturbation condition. The ACT models exhibit strong language generalization but worse visual generalization, and achieve higher base success rates compared to \pi_{0.5} .

Figure 2: Overview of Colosseum V2 . Left: the full set of tasks across two robot morphologies (Single-Arm and Bimanual), spanning diverse manipulation primitives and long-horizon behaviors. Right: the perturbations used to evaluate visual, language, and action generalization. In total, the benchmark comprises 28 tasks across 13 task categories with 16 controlled perturbation factors.

Figure 4: Simulator comparison. Frames per second (FPS) is computed as the number of parallel environments divided by the wall-clock time per simulation step (including ACT inference). RLBench is CPU-bound so is limited by the number of CPU cores, whereas Colosseum V2 leverages GPU-parallel simulation via ManiSkill to achieve substantially higher throughput. Using a NVIDIA RTX 4090, evaluating ACT on both test suites takes only 11.5 hours. In comparison, evaluating the same ACT model on the Colosseum benchmark takes 14.3 hours while generating 10\times fewer success rate measurements and canno

Figure 7: Simulation vs. hardware success rate comparison. While Colosseum V2 does not accurately predict absolute success rate, the success rate in simulation for a given perturbation does accurately predict the relative performance decrease under a perturbation on hardware. Stated differently, given the success rate with no perturbations and with one perturbation, the success rate of all other perturbations can be accurately estimated, as is reflected by an average R^{2} across the three tasks of 0.798 . Further, the average spearman correlation is 0.916 indicating the ordering of the pertur