Many-Shot Jailbreaking: Exploiting In-Context Learning at Scale
Demonstrates that providing many demonstrations of harmful behavior within the context window can teach language models to override their safety training, with attack success scaling with context size.
Large language models trained with a small number of demonstrations can quickly adapt to new tasks. This capability—in-context learning—is a strength for legitimate use cases. But it is also a vulnerability: if an attacker can provide hundreds of examples of a model ignoring safety constraints, the model may learn to replicate that behavior.
This research explores “many-shot” jailbreaking: providing so many demonstrations of harmful outputs that the model internalizes the pattern and produces unsafe content. Unlike few-shot learning (which uses 3-5 examples), many-shot uses dozens to hundreds, effectively retraining the model’s behavior in-context.
The findings are striking. Models with 128K or 1M token context windows become substantially more vulnerable to many-shot attacks as the number of examples increases. The attack works even when examples are presented in neutral or educational framing. The model’s safety training appears to be overridden by the statistical pattern of in-context examples, suggesting that safety is a learned behavior that can be unlearned through sufficiently many counterexamples.
For embodied AI, this has direct implications. A robot trained to refuse harmful actions might be deployed in an environment where it receives a long stream of logged interactions or documentation showing other robots performing those actions. If the robot’s language model uses in-context learning for task adaptation, it could begin replicating the unsafe behavior demonstrated in the long context.
The research reveals a fundamental tension: in-context learning is essential for fast adaptation to new tasks, but it creates a new attack surface orthogonal to training-time alignment. Embodied systems that require both safety and task flexibility must find defenses that preserve the benefits of in-context adaptation while preventing exploitation through many-shot patterns.