PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations

Figure 1 : Overview of PRTS. PRTS is built on Qwen3-VL-4B-Instruct and involves two training stages. (i) In the pre-training stage, the backbone consumes a unified token sequence of multi-view visual tokens (Qwen ViT), textual proprioceptive-state tokens, language-instruction tokens, and FAST-tokenized AR action tokens, followed by two auxiliary blocks <CRL_action> and <CRL_goal> , in which the last learnable tokens <|action_end|> and <|goal_end|> are passed through lightweight MLP heads (right) to produce the state-action representation \phi(s_{t},\mathbf{a}_{t}) and the goal representation \

Figure 11 : RealMan task suite. We evaluate PRTS on eleven RealMan tasks, including ten short-horizon desktop manipulation tasks and one long-horizon office task. The montage visualizes representative rollouts for the task settings described in Appendix C .

Figure 6 : Controlled visual, layout, and object shifts on the RealMan platform. We visualize the three lower-level generalization settings used in the RealMan study. Illumination generalization changes the lighting condition, spatial generalization moves the manipulated objects outside the post-training layout distribution, and object generalization changes the visual instance while preserving the semantic object category and the task instruction. Quantitative results are reported in Table 9 .

Figure 7 : Controlled task-level generalization on the RealMan platform. Task generalization changes the language goal by recombining components of seen tasks, making it semantically stricter than illumination, spatial, or object-level shifts. This setting tests whether the policy can bind a novel instruction to feasible manipulation steps instead of merely recognizing familiar objects or replaying a familiar trajectory. Quantitative results are reported in Table 9 .

Figure 2 : Pre-training dataset overview of PRTS. Our pre-training dataset is partitioned into two primary domains: Action-Labeled Data provides physical execution priors for cross-embodiment manipulation, while Visual-Reasoning Data provides complementary supervision for spatial grounding, affordance perception, and task planning. The area of each segment is scaled proportionally to the square root of its total tokens to highlight the rich, structural diversity of our data sources.

Figure 3 : Overview of real-world robot platforms. Left: dual-arm RealMan platform with two parallel-jaw grippers, one egocentric head camera, and two wrist cameras (one per arm). Right: Flexiv single-arm platform equipped with a parallel-jaw gripper, a third-view head camera, and two wrist cameras.

Figure 8 : Robustness and recovery under human interventions. We visualize representative intervention rollouts in which a human repeatedly perturbs task-relevant objects after the policy has committed to an action or subgoal. The traces cover bimanual receptacle interaction, multi-object organization, precise placement, and contact-rich assembly. Successful recovery requires the policy to recognize the changed state, re-estimate task progress, and choose the next action according to the still-unreached language goal.

Figure 9 : CRL value on a held-out Pick Shoes trajectory, scored by the pre-trained PRTS with no RealMan task suite fine-tuning. Left: the two instructions used to score the same demonstration—a correct one (green, top) that describes what the trajectory actually does, and a wrong one (red, bottom) that swaps the named grasp targets to objects (soy sauce, green paper cup) that never appear in the scene. Top right: five keyframes from the trajectory with their timestep indices, marking the structural milestones of the task (first shoe lifted, first shoe placed, second shoe placed, lid closing,