Multi-step Jailbreaking Privacy Attacks on ChatGPT

Focus: Li et al. demonstrated that decomposing privacy attacks into multi-turn conversational sequences dramatically increased the success of extracting personally identifiable information from ChatGPT, showing that single-turn safety evaluations fundamentally underestimate the threat from persistent, adaptive adversaries.

---

Key Insights

• Multi-turn decomposition defeats single-turn defenses. Privacy requests that ChatGPT reliably refused in a single turn could be elicited through multi-step conversational strategies. The attack bypassed safety filters that only evaluated individual turns in isolation.

• Sequential context manipulation is highly effective. The attack leveraged the model’s conversational memory to build a context in which the privacy-violating request appeared natural or justified.

• PII extraction at scale. The paper demonstrated extraction of real personal information including email addresses and phone numbers for public figures, with success rates significantly higher than single-turn attacks.

Executive Summary

The authors developed a three-stage attack methodology:

Stage 1: Topic Manipulation

Steering the conversation toward the target individual through seemingly innocent questions about their public work, achievements, or domain expertise.

Stage 2: Context Construction

Establishing a plausible reason for needing personal information, such as wanting to contact the person for a professional collaboration, academic inquiry, or legitimate-sounding business purpose.

Stage 3: Information Extraction

Requesting specific PII — email addresses, phone numbers, physical addresses — in a context where the request appears justified by the preceding conversation.

Results

Testing against ChatGPT (GPT-3.5 and GPT-4), the multi-step approach achieved substantially higher success rates than direct single-turn privacy requests. The attack was tested across categories:

• Email extraction: Highest success rate
• Phone number extraction: Moderate success
• Biographical details: High success with context manipulation
• Physical addresses: Lower but non-trivial success

Defense Analysis

The paper analyzed defensive measures and found that standard prompt-based safety instructions (e.g., “do not reveal personal information”) were insufficient against multi-turn attacks. The conversational context built over multiple turns effectively overrode the safety instruction’s influence.

The authors recommended context-aware safety monitoring that evaluates cumulative intent across a conversation rather than individual turns.

Relevance to Failure-First

Multi-turn attacks are a central concern of the failure-first framework:

• Single-turn benchmarks underestimate risk. This paper provides empirical evidence that single-turn safety evaluation fundamentally misses temporal attack strategies.

• Recursive failure propagation. The three-stage decomposition maps directly to the framework’s model of recursive failure, where each turn builds on the partial compromises of previous turns.

• Embodied AI exposure. Systems that maintain conversation state across extended interactions are particularly vulnerable to this class of attack, making it critical for human-robot interaction safety.

---

Read the full paper on arXiv · PDF