Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing

Introduction: Mitigating Physical Risk in Autonomous Suturing

In autonomous surgical robotics, executing robust suturing workflows remains a critical milestone toward automated wound closure. A persistent challenge in this domain is suture needle retrieval. During surgical operations, needles are frequently dropped unexpectedly or intentionally released to re-adjust grasping configurations for optimal tissue penetration. Traditional autonomous approaches rely on direct needle pickup, directing a robotic manipulator straight toward the metallic needle body. However, direct grasping frequently fails or introduces severe physical safety hazards when surgical needles become visually occluded by instruments, hidden beneath tissue layers, or lie flat against soft biological surfaces.

To address these contact hazards and perception limitations, Emma Huang et al. proposed an alternative, safety-oriented framework in their paper “Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing”. Instead of targeting the rigid needle directly, Huang et al. introduce an indirect manipulation strategy that leverages the flexible suture thread as an assistive tool. Because the suture thread comprises the majority of the suture’s overall length and offers high compliance and greater physical degrees of freedom, the robot can first retrieve the free thread and iteratively “reel in” the attached needle. This approach acts as a physical safety mechanism, preventing tissue trauma, mitigating tool slippage, and enabling reliable tool recovery in unstructured, non-approachable surgical environments.

---

The Failure Modes of Direct Needle Pickup vs. Indirect Manipulation

Conventional autonomous needle retrieval algorithms operate under the assumption that the needle is fully observable and directly approachable in 3D space. In unstructured or dynamic surgical fields, these assumptions frequently break down, resulting in physical failure modes that compromise patient safety.

Direct grasping strategies require the robotic gripper to make close contact with the tissue bed to pinch a thin, curved metallic needle. Because surgical needles are rigid, small, and slippery, direct gripper contact often causes the needle to slip or jump unpredictably. Furthermore, when the needle lies flat against soft tissue, the gripper jaws must apply localized contact pressure against the substrate, creating a high risk of pinching, tearing, or bruising underlying tissue structures.

In contrast, the indirect thread-assisted strategy proposed by Huang et al. isolates initial tool contact to the flexible thread elevated above the tissue surface. By modeling visual uncertainty and enforcing constrained spatial clearance, the framework significantly reduces tool-tissue contact hazards and successfully recovers occluded needles.

Manipulation Strategy	Visual Dependency	Tissue Pinching Risk	Occlusion Capability	Primary Failure Mode
Direct Needle Pickup	High; requires full, unoccluded visibility of the curved needle body.	High; gripper must press against or near soft tissue to grasp flat/thin needles.	None; fails when the needle is hidden under tissue, gauze, or instruments.	Direct tool-tissue collision/pinching, needle jumping/slippage, and mechanical singularities.
Thread-Assisted Indirect Pickup	Partial; only requires visibility of a segment of the flexible suture thread.	Low; constrained optimization enforces safe clearance margins above tissue.	High; retrieves occluded needles by tracing the attached flexible thread.	Thread slack drifting outside uncertainty cone, tangling/overlapping, or gauze snagging.

---

End-to-End Pipeline: From Image Perception to Bimanual Traversal

The autonomous framework developed by Huang et al. translates raw stereo visual input into physical robot trajectories across four distinct operational stages.

+-------------------------------------------------------+
| Step 1: 3D Thread & Tissue Reconstruction             |
| Stereo segmentation -> Keypoint ordering -> Spline    |
+-------------------------------------------------------+
                           |
                           v
+-------------------------------------------------------+
| Step 2: Reliability-Aware Grasp-Point Selection      |
| Optimization balancing needle distance & tissue height|
+-------------------------------------------------------+
                           |
                           v
+-------------------------------------------------------+
| Step 3: Circular Pickup and Thread Lengthening        |
| Rotational lifting around needle tail to prevent drag |
+-------------------------------------------------------+
                           |
                           v
+-------------------------------------------------------+
| Step 4: Bimanual Needle Fishing                       |
| Iterative handoffs guided by uncertainty cone model   |
+-------------------------------------------------------+

1. 3D Thread & Tissue Reconstruction

The vision system processes left and right stereo endoscopic images ($I_L, I_R$) to generate binary thread segmentation masks ($M_L, M_R$). Stereo matching calculates disparity maps, re-projecting thread pixels into 3D space as a set of unordered keypoints $\hat{k}_1, \dots, \hat{k}_N \in \mathbb{R}^3$. To handle complex overlaps, loops, and self-intersections, keypoints are ordered sequentially from the needle junction to the thread tail as $k_1, \dots, k_N$.

For each keypoint $k_i$, a local depth bound $\epsilon_{i,z}$ is calculated using a local least-squares line fit $L_{k_i}$: $\epsilon_{i,z} = 1.5 \|L_{k_i}(k_i) - k_{i,z}\|_2^2$

To facilitate safety-constrained optimization, this depth error bound is explicitly split into lower and upper depth limits ($k_i^-, k_i^+$) at each keypoint: $k_i^- = k_{i,z} - 1.5 \|L_{k_i}(k_i) - k_{i,z}\|_2^2$ $k_i^+ = k_{i,z} + 1.5 \|L_{k_i}(k_i) - k_{i,z}\|_2^2$

Combined with image-space pixel bounds ($\epsilon_{i,u}, \epsilon_{i,v}$), a local reliability region $\text{region}(k_i) \subset \mathbb{R}^3$ is defined. A 3D smooth spline $B(s)$ parameterized by $s \in [0, 1]$ is constructed via constrained optimization within these bounds and uniformly sampled into $N$ candidate points $P = \{p_1, \dots, p_N\}$. Each candidate $p_i$ is assigned a continuous reliability metric $r_i \in R$, defined inversely to the volume of its local reliability region.

Concurrently, tissue surface reconstruction generates point cloud $Q = \{q_i \in \mathbb{R}^3\}_{i=1}^M$. Points overlapping with the thread mask $M_L$ ($Q_{\text{non-thread}}$) and points within filtering radius $r$ are removed to produce $Q_{\text{filtered}}$. The surface normal vector $n$ is calculated by averaging normal vectors across $Q_{\text{filtered}}$, assuming $n$ is roughly parallel to gravity $g$ ($|g \cdot n| \approx 1$).

2. Reliability-Aware Grasp-Point Selection

To select an optimal grasp point $p_{\text{grasp}}$, the pipeline solves a constrained optimization problem over candidate points $P$:

$\arg\min_{i \in \{1, \dots, N\}} \alpha l(p_i) - \beta d(p_i, Q_{\text{filtered}})$

Where:

• $l(p_i) = \sum_{\tau=1}^{i-1} \|p_{\tau+1} - p_\tau\|_2$ represents the cumulative thread length from candidate point $p_i$ to the needle junction.
• $d(p_i, Q_{\text{filtered}}) = \min_{q \in Q_{\text{filtered}}} \|p_i - q\|_2$ measures the Euclidean distance from the candidate point to the nearest point $q_i^*$ on the tissue surface.
• $\alpha = 0.7$ and $\beta = 1.0$ are weighting hyperparameters prioritizing tissue distance over proximity to the needle.

Candidates are removed if they violate three physical safety constraints:

1. Tissue Clearance Constraint: The candidate’s maximum candidate depth bound $k_{i,z}^+$ plus a safety clearance margin $\delta_s = 1\text{ mm}$ must remain strictly above tissue depth $q_{i,z}^*$: $k_{i,z}^+ + \delta_s < q_{i,z}^* \quad \forall q_i \in Q_{\text{filtered}}$
2. Reliability Threshold Constraint: Candidate reconstruction reliability $r_i$ must exceed a minimum threshold ($\delta_r = 0.4$): $r_i > \delta_r$
3. Depth Stability Constraint: Local depth variation between both preceding ($p_{i-1}$) and succeeding ($p_{i+1}$) keypoints must remain below a velocity threshold ($\delta_v = 0.6$), preventing selection on steep thread segments prone to gripper slippage: $|p_{i-1,z} - p_{i,z}| \le \delta_v \quad \land \quad |p_{i+1,z} - p_{i,z}| \le \delta_v$

3. Circular Pickup and Thread Lengthening

Once $p_{\text{grasp}}$ is selected, the primary gripper approaches the thread with an offset $\delta_{\text{offset}} = 20\text{ mm}$ along its y-axis, aligning its wrist orientation $R_{\text{grasp}} \in SO(3)$ such that its z-axis lies tangent to the thread pointing toward the needle.

To lift the thread without causing tissue trauma, the framework executes a circular rotational motion around the needle tail point ($p_1$). Directly lifting upward creates linear dragging forces that pull the needle body across tissue, risking point embedding. The circular rotation matrix $\Delta R_{\text{circular}} \in SO(3)$ aligns the vector $v = p_{\text{grasp}} - p_1$ with the tissue normal $n$.

Following rotation, the robot translates vertically along $n$ (parallel to gravity) to straighten the thread up to a maximum workspace height clearance $h_{\text{workspace}}$. Aligning motion with gravity stabilizes thread dynamics by eliminating lateral oscillations against adjacent tissue.

4. Bimanual Needle Fishing

If the suture thread length $l_{\text{thread}}$ exceeds $h_{\text{workspace}}$, the system transitions to an iterative handoff policy using two robotic grippers. After being lifted, the thread’s physical configuration becomes highly dynamic and difficult to predict. This uncertainty stems from the interplay between the thread’s flexural rigidity (shape memory) and gravitational loading from the suspended metallic needle, which prevents the thread from straightening into a perfect vertical line.

To accommodate this physical deformation, the framework models spatial uncertainty as a geometric uncertainty cone centered along the gravity axis with half-angle $\theta$. To guarantee that open gripper jaws (with half-width $w$) successfully encompass the floating thread during handoffs, the step size $s$ between sequential bimanual handoff maneuvers is derived verbatim from the paper’s algorithmic formulation:

$s = \frac{w}{\arctan \theta}$

The two grippers take turns holding and stepping down the thread along $n$ toward the needle. Once the remaining thread length is within reach, the secondary gripper executes a final handoff directly to the needle body.

---

Experimental Evaluation & Performance Across Real-World Setups

The autonomous pipeline was evaluated empirically on a da Vinci Research Kit (dVRK) integrated with a 36 mm 1/2-circle curved surgical needle attached to a 45 cm flexible suture thread. System performance was benchmarked across four real-world environmental setups representing varying geometric and perceptual complexities.

Setup Environment	Configuration Description	Grasp Success Rate (%)	Pickup Success Rate (%)	Observed Failure Mechanisms
Easy	Thread length $\le 50\text{ mm}$; elevated $\ge 10\text{ mm}$ above tissue; no self-overlaps or occlusions.	95% (19/20)	95% (19/20)	Local 3D reconstruction noise causing localized gripper displacement and missed grasps.
Medium	Thread length $50\text{--}90\text{ mm}$; elevated $< 10\text{ mm}$ above tissue; no self-overlaps or occlusions.	95% (19/20)	95% (19/20)	Occluded local segments creating slight stereo depth estimation errors.
Hard	Thread length $50\text{--}180\text{ mm}$; elevated $< 10\text{ mm}$; contains multiple loops and self-intersections.	95% (19/20)	70% (14/20)	Thread slack pulling needle outside predicted uncertainty cone ($\theta$); tangling near intersections; grasping parallel overlapped threads simultaneously.
Occlusion	Thread length $50\text{--}180\text{ mm}$; needle/thread partially occluded by surgical gauze and silicon phantoms.	90% (18/20)	65% (13/20)	Gauze proximity invalidating safe grasp candidates; thread snagging on rough gauze fibers; multiple strands looping over gripper jaws.

Contextualizing Environmental Setup Performance

The experimental results demonstrate a clear relationship between environmental geometry, perceptual Disparity, and system reliability. In the Easy setup, thread segments elevated $\ge 10\text{ mm}$ above the tissue surface create distinct vertical gradients. These elevated features yield significantly larger stereo disparity between left and right endoscopic views, enabling high-precision 3D keypoint matching and reliable spline fitting.

Conversely, in Medium, Hard, and Occlusion setups, threads lying flat against tissue ($< 10\text{ mm}$) or extending up to $180\text{ mm}$ in length introduce severe perceptual challenges. Low surface clearance minimizes stereo disparity, increasing localized depth noise. Furthermore, self-overlapping loops in long threads create visual cross-point ambiguities, while surgical gauze introduces partial occlusions that restrict valid grasp-point candidates.

Failure Mechanism Analysis in Complex Scenarios

While the grasp selection policy remained robust across complex setups (90–95% success), overall pickup completion rates decreased in the Hard (70%) and Occlusion (65%) environments during the bimanual fishing phase. The authors identified three primary physical failure modes:

1. Slack Displacement: Excessive thread slack caused the suspended needle to swing or anchor away from the vertical axis, pulling the thread outside the geometric boundaries of the predicted uncertainty cone ($\theta$) and causing trailing handoffs to miss.
2. Intersection Ambiguity: Self-overlapping loops introduced visual ambiguity near cross-points, occasionally causing the gripper to grasp two adjacent thread segments simultaneously. This preserved tangled loops that invalidated single-strand cone assumptions.
3. Frictional Material Snagging: In occluded trials, proximity to porous surgical gauze caused lifted threads to catch on gauze edges, generating unexpected lateral tension that pulled the suture away from the handoff trajectory.

---

Ablation Studies: Proving the Value of Safety Mechanisms

Huang et al. conducted controlled ablation experiments to isolate the physical impact of specific safety components embedded in the pipeline.

• Thread Reliability Metric Impact: The authors evaluated grasp performance with and without the reconstruction reliability metric ($r_i$). Utilizing the reliability metric achieved a 100% grasp success rate (15/15). Conversely, omitting the reliability metric resulted in a 46.66% grasp success rate (7/15), representing a 53.33% performance drop. Without reliability bounds, 7 trials completely missed the thread due to depth re-projection errors, and 1 trial resulted in severe tissue pinching.
• Tissue Pinching Prevention: Across 80 evaluated physical trials using the complete constrained optimization policy, the reliability-aware grasp selection mechanism achieved a 100% safety success rate with 0 instances of tissue pinching. Candidate depth bounds ($k_{i,z}^+$) combined with tissue clearance margins ($\delta_s$) effectively isolated grasping actions above the tissue surface.
• Circular Pickup vs. Direct Dragging: The authors compared the circular rotational lifting motion ($\Delta R_{\text{circular}}$) directly against a linear upward pulling strategy. Direct upward pulling dragged the curved needle tip across the tissue bed, causing the sharp point to hook and embed into the surface phantom. The circular pickup motion successfully rotated the needle tail prior to vertical translation, eliminating surface drag and mitigating contact trauma.

---

Key Takeaways for Robotics & AI Safety Researchers

1. Deformable Manipulators as Safety Buffers: Indirect manipulation using compliant components (such as suture threads) effectively transforms brittle, high-risk contact problems into flexible recovery tasks. By targeting compliant structures elevated above sensitive surfaces, surgical robots can avoid localized force spikes and tool-tissue collisions.

2. Perception Uncertainty Must Inform Control Bounds: Explicitly modeling perception bounds—such as stereo reconstruction reliability ($r_i$) and downstream spatial uncertainty cones ($\theta$)—allows policy engines to adjust mechanical step sizes ($s = w / \arctan \theta$) dynamically. Safety in unstructured environments requires coupling vision uncertainty metrics directly to kinematic execution constraints.

3. Geometric Prior Integration Eliminates Mechanical Hazards: Simple physical priors, such as executing circular rotational motions around an object’s trailing vector ($\Delta R_{\text{circular}}$), prevent high-risk failure modes like tool dragging and tip embedding. Hardcoding domain-specific spatial constraints often yields higher safety margins than unconstrained trajectory generation.

Read the full paper on arXiv · PDF