Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

Figure 1. Point&Grasp is a technique for selecting out-of-reach objects in virtual reality. While pointing direction suffers in cluttered scenes and grasping gestures become ambiguous when objects share similar shapes, Point&Grasp combines both cues to enable more precise and usable target selection.

Figure 2. Overview of the proposed multi-cue inference framework for out-of-reach object selection. The framework combines an endpoint distribution model for directional cues and a gesture–object likelihood model for grasping gestural cues, which are integrated through Bayesian cue integration to infer the intended target object.

Figure 6. The object confusion matrix shows that when using grasping gesture cues alone, some objects are more easily and some less easily confused with each other. Each cell indicates how often a gesture elicited by the target object was also judged compatible with a replacement object; brighter values denote higher confusion likelihood. The examples on the right illustrate typical outcomes: (Row 1) the gesture for a mug was incompatible with a crackerbox; (Row 2) another gesture for a mug was judged compatible with a crackerbox; (Row 3) although the same gesture for a mug as in Row 2 was app

Figure 8. Example scenes illustrating the four ambiguity conditions that vary along semantic and spatial dimensions. In the low semantic × low spatial condition, objects (cracker box, flashlight, camera, disk lid, and bowl) are both semantically and spatially distinct, minimizing potential confusion. The low semantic × high spatial condition (game controller, meat can, flashlight, watch, and headphone) maintains semantic distinctiveness but increases spatial proximity, raising the likelihood of spatial ambiguity. Conversely, the high semantic × low spatial condition (banana, toothpaste, winegl

Figure 4. The results show that hand gesture for an out-of-reach object is significantly different from when it is within reach, and that variability across users is high. Bars indicate per-joint mean L2 distance grouped by finger, with error bars showing standard deviations. The illustration maps each joint index (J0–J20) to its anatomical position on the hand for reference.

Figure 7. Clustering analysis shows that gesture-based confusability is systematic rather than random: objects that afford similar grasping postures, such as fryingpan–hammer–knife or cup–mug–wineglass–waterbottle, naturally group together. The clusters were derived by applying the Silhouette method to the similarity matrix of 30 objects: Cluster 1: bowl, large sphere. Cluster 2: alarm clock, crackerbox, apple, mug, water bottle, wineglass. Cluster 3: eyeglasses, scissors, watch. Cluster 4: camera, potted meat can. Cluster 5: headphones, teapot. Cluster 6: banana, binoculars, flashlight, small

Figure 10. (Study 1) Point&Grasp achieves the fastest selection times and the highest trial completion rates, avoiding the slowdown of Point under spatial ambiguity and Grasp under semantic ambiguity. Across both measures, Point&Grasp remains robust under spatial and semantic ambiguity where single-cue methods degrade. Error bars show between-participant variation. Asterisks indicate statistically significant differences after Bonferroni correction (* p<0.05 , ** p<0.01 , *** p<0.001 ).

Figure 11. (Study 1) Users flexibly exploited gesture and direction cues in Point&Grasp , with successes often driven by a single dominant cue but still possible when neither cue was sufficient alone. Trials were labeled Both (both cues selected the target), Gesture Only (only gesture did), Direction Only (only direction did), or Neither (neither did). Left: overall patterns (success vs. failure). Right: breakdown by ambiguity conditions. Values denote percentages within each bar.