From a Single Demonstration to a General Policy for Contact-Rich Manipulation

Figure 5 : Illustration of our funneling strategies in the puzzle insertion task. Left : After a free-space primitive, the robot stops just short of contact. Transitioning to the plane primitive from this state would erroneously trigger a breaking-contact event, leading to premature state transition and subsequent manipulation failure. Our funneling strategy addresses this by moving along the plane’s constraint direction to establish contact, compensating for positional uncertainties. Right : During a rotation primitive, the robot applies a force along the constrained axis (the admissible dire

Figure 6 : Observed z-axis forces as the robot approaches the lock surface at 2\text{\,}\mathrm{c}\mathrm{m}\mathrm{/}\mathrm{s} . The controller is evaluated with predefined force limits of 5\text{\,}\mathrm{N} and 10\text{\,}\mathrm{N} across stiffness settings from 500\text{\,}\mathrm{N}\mathrm{/}\mathrm{m} to 2500\text{\,}\mathrm{N}\mathrm{/}\mathrm{m} . The force profiles show that the controller reliably maintains the target contact force under various stiffness settings, enabling safe interaction and simplifying parameter tuning.

Figure 8 : Illustration of success and failure modes in the puzzle insertion task, divided into two stages: Insertion (top) and Completion (bottom). A trial is considered a successful insertion if the part is correctly placed into the puzzle board. Failure modes in this stage include misalignment or prematurely lowering the part. In the completion stage, success requires guiding the part through the internal pins via two sequential and precise rotations. Failures at this stage include incorrect rotation due to perceptual aliasing and incomplete insertion resulting from perceptual and control i

Figure 15 : Execution of the policy of opening two drawers. By generating motions that follow the kinematic structure of the handle and drawer, a single policy reliably succeeds despite unseen variations in object pose and geometry.

Figure 1 : Overview of the four-stage LfD pipeline. A single human demonstration is segmented into a sequence of motion phases, where the exact demonstration actions that are specific to the demonstrated object instance are abstracted using the EC representation. In the next augmentation stage, the robot actively explores the environment around the demonstration to generate additional information of ECs, facilitating the identification of EC primitives (colored outlines) and contact transitions. When the robot encounters unforeseen task variations that are not captured by the EC representation

Figure 3 : Overview of the four EC primitives. The free-space primitive aligns the end-effector to the target object through visual servoing using a wrist-mounted camera, enabling adaptation to changes in object pose. It then replays the transferred demonstrated trajectory to reproduce contact-free motions. The plane primitive first replays the transferred demonstration trajectory along a surface and then refines its position through visual servoing and spiral search, enabling generalization to uncertain or shifted hole locations. The translation and rotation primitives generate compliant moti

Figure 10 : Experimental setup for the door-lock manipulation task. The policy extracted from Lock 1 is extended to Lock 2 via two targeted corrective demonstrations. We intentionally tilt the mounting surface to demonstrate that our approach does not rely on a tabletop assumption.

Figure 11 : Policy refinement for door lock manipulation. The policy learned from Lock 1 is extended with two corrective demonstrations to handle unseen contact changes in Lock 2. The first correction guides the end-effector from an incorrect pose toward the keyhole, while the second correction introduces wiggling motions to resolve misalignment during insertion. Our EC-based policy representation enables seamless integration of these corrections as additional EC primitives without retraining the full policy.

Figure 2 : Segmentation result of a demonstration for a single-object insertion task from Functional Manipulation Benchmark [ fmb2023 ] . The top plots show TVD force signals, gripper states, and end-effector positions, indexed by waypoint of a demonstration. Vertical dashed lines indicate segmentation from force thresholds (purple), gripper state changes (orange), and ZVCs (cyan). We first merge these segments and then remove those whose total displacement is less than 2\text{\,}\mathrm{cm} . The bottom row shows six final segments and snapshots of the robot behavior during each motion phase.

Figure 4 : Vision-based augmentation to filter unstable features. By actively changing the robot’s viewpoint and evaluating feature matching performance (left), the vision-based augmentation strategy filters out features that are not consistent under viewpoint changes (right). This includes features caused by light reflections on the insertion board surface, which are prone to mismatching and sensitive to lighting variation. It also filters out features on the inserted object. Because these features move with the robot, they are inconsistent with changing viewpoints and would introduce errors

Figure 9 : Solving a green puzzle by following a sequence of ECs. \raisebox{-0.9pt}{1}⃝ The robot first visually aligns the end-effector to the puzzle and transfers the demonstration trajectory to the new pose. \raisebox{-0.9pt}{2}⃝ It replays the trajectory and purposefully pushes down until making contact with the puzzle. \raisebox{-0.9pt}{3}⃝ It then executes the plane primitive that replays the transferred trajectory on the plane, followed by a visual servoing and a spiral search action to reduce positional uncertainties, terminating upon successful insertion (a breaking-contact event). \r

Figure 16 : Failures and corrections in the FMB assembly task. The top row shows three failure cases: object-board collision, wrong hole insertion, and lost grasp. The bottom row illustrates the corresponding human corrections: local wiggling to resolve misalignment, retargeting the correct insertion hole, and adjusting the grasp pose.