RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking

Figure 1: Toy example visualizing the Q-landscape for a fixed state s and 2D action space (a_{0},a_{1})\in\mathcal{A} under different critic objectives after offline training. The first row shows the 3D Q-manifold; the second row shows a 2D projection with eight sampled actions and their gradient ascent trajectories; and the third row shows \partial Q/\partial a magnitudes. By explicitly shaping the Q-landscape, RankQ enables the largest number of trajectories to converge toward the optimal action region.

Figure E.3: The \partial Q/\partial a statistics for the vla-sim2real-cube-stacking experiments from Fig. 4 . The gray shaded region denotes the offline RL phase.

Figure H.7: Rollout example where the baseline policy repeatedly fails to grasp the green cube until timeout. In comparison, the RankQ policy stacks the cubes in one attempt.

Figure H.8: Rollout example where the baseline policy fails to grasp the green cube and then hovers in confusion until timeout. In comparison, the RankQ policy fails to grasp the green cube initially but then grabs it with a quick reattempt. After failing to place it, it retries again and successfully stacks the cube before timeout.

Figure C.1: CQL and Cal-QL \alpha parameter tuning using vla-low-data-carrot-onto-plate .

Figure G.5: Visualization of the vla-low-data environments.

Figure G.6: Visualization of the vla-sim2real-cube-stacking environment. Each frame shows the start of an episode where scene, robot pose, camera pose, and lighting randomizations can clearly be seen.

Figure 2: Success rate and average trajectory length results for the D4RL antmaze and adroit environments. Curves start after offline RL training has concluded. Each algorithm is reported across 3 random seeds. Note that the average trajectory length metric is failure-inclusive.

Figure 4: Success rate and average time-to-finish results for the vla-sim2real-cube-stacking environment during training. Note that training metrics indicate slightly lower performance compared to eval metrics due to sampling of actions rather than greedy mode selection. Curves start after offline RL training has concluded. Each algorithm is reported across 3 random seeds with each seed having its own unique set of 800 self-rollouts. The average finish time is failure-exclusive.

Figure D.2: RankQ ablation results on D4RL environments.

Figure F.4: Accuracy results comparing Q-values of successful actions against the four categories of suboptimal actions defined in Eq. 4 for the vla-sim2real-cube-stacking experiments from Fig. 4 .