Interaction Through Instruments: Extending Surgical Instruments as Interaction Devices

People involved

Nour Karoui, Isabelle Bloch, Ignacio Avellino

Abstract

Interaction while performing physical tasks is inherently challenging, as both hands are fully engaged. In Minimally-Invasive Surgery (MIS), for instance, navigating images requires either delegation, causing frustration and delays, or hand de-sterilization, increasing risk. We introduce Interaction Through Instruments, an interaction paradigm in which task instruments become interaction devices. To design this technique in MIS, we first conduct a survey (N=23) identifying intraoperative needs, interaction strategies and workarounds, and persistent challenges. Then, through five participatory design workshops (N=10), we identify challenges in blending a user interface into views of a physical space, informing the design of InteractOR, a system that combines surgical instrument segmentation with pinch-gesture recognition to enable interaction within the surgical view. Finally, in a Comparative Structured Observation study (N=12) we compare two visualization strategies (side-by-side and overlay) against delegation, showing that interaction through instruments can reduce focus shifts, increase efficiency, and foster autonomy.