Safety Filter Evaluation Across Model Generations

Introduction

Safety filters in large language models have undergone rapid iteration since the first generation of instruction-tuned systems reached public availability. Early safety mechanisms relied heavily on keyword blocklists and simple pattern matching against known harmful phrases, an approach that was straightforward to implement but equally straightforward to circumvent. As models grew in capability, the inadequacy of static filtering became apparent: adversaries could rephrase, obfuscate, or contextually embed harmful requests in ways that preserved semantic intent while evading surface-level detection. The shift toward training-time safety interventions, particularly reinforcement learning from human feedback, represented a qualitative change in defense philosophy, moving from rule-based blocking to learned behavioral constraints.

This longitudinal study examines how safety filtering has evolved across three successive generations of models from major providers, tracking both the sophistication of defense mechanisms and the corresponding evolution of adversarial techniques. Our analysis covers the period from early 2023 through late 2025, a window during which the field witnessed the emergence of constitutional AI methods, system prompt hardening, multi-layered output classification, and hierarchical instruction following. Each of these innovations addressed specific failure modes observed in prior generations, but each also introduced new attack surfaces that adversaries were quick to exploit.

Generational Analysis of Filter Mechanisms

First-generation safety filters operated primarily at the token level, scanning inputs and outputs for prohibited terms and phrases. These systems achieved high recall on direct harmful requests but suffered from excessive false positives on legitimate content that happened to contain flagged terms, and near-zero detection of semantically harmful content expressed through euphemism or indirection. Second-generation systems introduced contextual classification, using smaller auxiliary models to evaluate whether a given input-output pair violated safety policies when considered holistically. This approach significantly improved precision and enabled detection of indirect harmful content, but introduced latency and created a new failure mode: the auxiliary classifier itself could be confused by adversarial inputs designed to appear benign in isolation while being harmful in conversational context.

Third-generation systems, which began appearing in production deployments in mid-2025, combine training-time behavioral constraints with runtime monitoring and hierarchical instruction architectures. These systems treat safety not as a binary filter but as a continuous assessment integrated into the generation process itself. Preliminary results suggest that this approach offers substantially improved robustness against known attack categories, though it remains an open question whether the improvement generalizes to novel attack types not represented in the training distribution. Our testing across 847 adversarial scenarios indicates that third-generation filters reduce attack success rates by approximately 40 percent compared to second-generation systems, with the largest improvements observed in multi-turn jailbreak and persona hijack categories.

Cross-Generational Vulnerability Persistence

Perhaps the most concerning finding from our longitudinal analysis is the persistence of certain vulnerability classes across all three generations. Attacks that exploit the boundary between data and instructions, including prompt injection and context manipulation, show only modest improvement in detection rates across generations. This suggests that the fundamental architectural decision to process instructions and data through the same channel remains a structural vulnerability that training-time interventions alone cannot fully address. Addressing this class of vulnerability may require architectural changes that go beyond the current paradigm of single-stream transformer-based generation, a research direction that several groups are actively exploring but that has not yet yielded production-ready alternatives.