Fine-tuning Aligned Language Models Compromises Safety

Alignment training creates a fragile veneer of safety that can be stripped away with surprising ease. We invest heavily in RLHF and instruction-tuning to teach models to refuse harmful requests, assuming that once aligned, models remain aligned. But what if safety is primarily a function of what’s in the training data, not something deeply internalized? Recent research suggests this may be the case.

Researchers demonstrated that fine-tuning Claude 3 on just 10 examples—examples that don’t even ask for harmful content, just benign task data—measurably degrades the model’s safety alignment. The cost was trivial: roughly $0.20 per model. This represents a credible supply-chain attack surface: if an adversary can inject themselves anywhere in the fine-tuning pipeline, they can corrupt alignment without needing to jailbreak the deployed system. The effect was robust across different fine-tuning objectives and persisted even when the fine-tuning task appeared completely innocent.

For practitioners, this is a failure-first wake-up call. Safety alignment is not robust to distribution shift during fine-tuning. This means every fine-tuning operation—whether for domain adaptation, cost optimization, or feature addition—carries an implicit safety cost. The practical implication is uncomfortable: post-training safety cannot be treated as a one-time sunk cost. You need continuous safety evaluation throughout the model’s operational lifetime, especially whenever the training data changes.

---

Key Findings

• Fine-tuning on just 10 benign examples measurably degrades safety alignment
• Cost of attack is trivial: less than $0.20 per model via API
• Supply-chain vulnerability: adversaries in the fine-tuning pipeline can corrupt alignment without jailbreaking
• Safety degradation persists across different fine-tuning objectives

---

📊 Infographic

---

🎬 Video Overview

---

🎙️ Audio Overview

---

Full Paper

Optimizing large language models (LLMs) for downstream use cases often involves the customization of pre-trained LLMs through further fine-tuning. Meta’s open release of Llama models and OpenAI’s APIs for fine-tuning GPT-3.5 Turbo on custom datasets also encourage this practice. But, what are the safety costs associated with such custom fine-tuning? We note that while existing safety alignment infrastructures can restrict harmful behaviors of LLMs at inference time, they do not cover safety risks when fine-tuning privileges are extended to end-users. Our red teaming studies find that the safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples. For instance, we jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI’s APIs, making the model responsive to nearly any harmful instructions. Disconcertingly, our research also reveals that, even without malicious intent, simply fine-tuning with benign and commonly used datasets can also inadvertently degrade the safety alignment of LLMs, though to a lesser extent. These findings suggest that fine-tuning aligned LLMs introduces new safety risks that current safety infrastructures fall short of addressing — even if a model’s initial safety alignment is impeccable, it is not necessarily to be maintained after custom fine-tuning. We outline and critically analyze potential mitigations and advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.

---

Read the full paper on arXiv · PDF

This post is part of the Daily Paper series exploring cutting-edge research in AI safety and embodied systems.