The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
An empirical study showing that misaligning an LLM via fine-tuning is significantly cheaper than realigning it, with asymmetric attack-defense dynamics that have serious implications for deployed safety.
Safety alignment — the process of steering large language models toward helpful, harmless, and honest behavior — has become the defining engineering challenge of the post-GPT era. But how durable is alignment, really? A paper by Zhang et al., published just three days ago on April 9, 2026, asks a harder version of this question: when alignment is broken through fine-tuning, and when it is subsequently restored, are both operations equally tractable?
The answer is no — and the asymmetry has troubling implications for how we think about LLM safety across the model lifecycle.
The Fine-Tuning Alignment Problem
Modern LLMs achieve safety alignment primarily through post-training: supervised fine-tuning (SFT) on curated datasets, followed by preference optimization via methods like RLHF, Direct Preference Optimization (DPO), or the newer Odds Ratio Preference Optimization (ORPO). Each of these shapes the model’s output distribution to favor policy-compliant, safe responses.
But the same mechanisms that align a model can be turned against it. The fine-tuning attack literature has established that even a small number of malicious fine-tuning examples can meaningfully degrade safety behavior — a phenomenon sometimes called “alignment tax erasure.” What Zhang et al. contribute is the first systematic comparison: across multiple fine-tuning methods and model families, how efficiently can misalignment be induced, and how efficiently can it be reversed?
Attack Efficiency Outpaces Defense
The central finding is that misalignment is significantly easier to induce than to correct. Supervised fine-tuning on a small set of harmful examples reliably erodes safety guardrails. Preference-based methods — DPO and ORPO — can similarly be exploited to shift the model’s preference distribution toward policy-violating outputs with relatively few training steps.
The reverse process — realigning a misaligned model — requires substantially more data and compute to achieve equivalent recovery. More troublingly, certain alignment techniques, particularly knowledge distillation and naive RLHF fine-tuning, can inadvertently introduce new safety degradations while attempting to restore prior ones. The paper frames this as a fundamental asymmetry: fine-tuning attacks are low-cost and high-leverage, while defenses are high-cost and only partially effective.
This asymmetry is not surprising in isolation — it echoes findings from adversarial robustness research on vision models — but its confirmation in the safety alignment domain raises the stakes considerably, given how widely deployed LLMs now are.
Chain-of-Thought as a New Attack Vector
One of the paper’s sharpest findings involves reasoning models. LLMs trained to use explicit chain-of-thought (CoT) reasoning exhibit elevated vulnerability to CoT attacks — adversarial prompts that subvert the reasoning chain before the model reaches a refusal decision. By steering the model through a sequence of intermediate conclusions that appear individually plausible, attackers can arrive at a harmful final output that the model would reject if prompted directly.
Text-completion interfaces, which expose more of the model’s step-by-step reasoning, show the highest susceptibility. This connects to a broader unresolved tension in AI safety: reasoning capabilities that improve model utility may simultaneously create new failure modes, because they expose more of the model’s decision process to adversarial manipulation.
Implications for Embodied AI Safety
At first glance, fine-tuning dynamics in pure language models might seem distant from embodied AI safety. But VLA models — vision-language-action models that control physical robots — are built on identical foundations. When a VLA model is fine-tuned on task-specific manipulation data, the alignment erosion dynamics documented here apply directly. An RLHF-aligned VLA that is subsequently fine-tuned for a specialized industrial application may arrive at deployment with degraded safety constraints that are invisible to standard performance benchmarks.
The asymmetric cost of realignment also means that safety monitoring cannot be treated as a one-time property check performed before initial release. Systems that undergo continued learning, domain adaptation, or user fine-tuning require ongoing alignment assessment — a requirement that current evaluation frameworks are not designed to handle efficiently.
A Warning for the Model Lifecycle
The Art of (Mis)alignment contributes to a growing body of evidence that safety alignment is a fragile property, not a durable guarantee. The practical upshot for deployers is uncomfortable: maintaining alignment through model lifecycle events — fine-tuning, distillation, domain adaptation, continued learning — requires active effort and explicit measurement.
As the field moves toward agentic and embodied LLM applications where model outputs cause physical consequences, understanding these dynamics transitions from an academic exercise to a prerequisite for responsible deployment. The paper’s systematic analysis of which fine-tuning methods are most and least susceptible to alignment erosion gives practitioners a clearer map of the risks they face when adapting foundation models for downstream tasks.
Read the full paper on arXiv · PDF