Multi-Modal Alignment Challenges
Safety at the intersection of vision, language, and physical action
The Multi-Modal Attack Surface
The extension of language models to multi-modal inputs has created alignment challenges that are qualitatively different from those in text-only systems. When a model processes images alongside text, the attack surface expands from the one-dimensional space of token sequences to the high-dimensional space of pixel arrays. Adversarial images can encode instructions that are invisible to human viewers but are decoded by the model's vision encoder into actionable text representations. This cross-modal injection vector is particularly dangerous because it exploits a perceptual asymmetry: humans and models literally see different things when looking at the same image.
Early research on multi-modal adversarial examples focused primarily on classification tasks, demonstrating that imperceptible perturbations to input images could cause misclassification with high confidence. The alignment implications of these findings were initially underappreciated. However, as vision-language models have been deployed in agentic contexts where they make decisions based on visual input, the consequences of adversarial visual inputs have become far more severe. A misclassified image in an academic benchmark is a minor inconvenience; a misinterpreted visual scene in an autonomous navigation system is a potential safety hazard.
Cross-Modal Consistency in Safety Training
A fundamental challenge in multi-modal alignment is ensuring consistency of safety behavior across input modalities. A model that correctly refuses a text-based request for harmful instructions may comply when the same request is presented as text embedded in an image, or when the harmful intent is conveyed through a combination of innocuous text and a contextualizing image. This inconsistency arises because safety training has historically been conducted primarily on text inputs, and the resulting safety behaviors do not automatically generalize to visual or cross-modal inputs. The safety training distribution and the deployment input distribution are mismatched along the modality axis.
Addressing this gap requires multi-modal safety training data that includes adversarial examples spanning all supported input modalities and their combinations. Constructing such datasets is expensive and labor-intensive, as it requires human annotators who can assess the safety implications of complex multi-modal inputs. Automated red-teaming approaches that generate adversarial multi-modal inputs at scale offer a partial solution, but they tend to discover attack patterns that are clustered in a narrow region of the input space, leaving large areas of the multi-modal attack surface unexplored.
Embodied Multi-Modal Risks
The alignment risks of multi-modal models are further amplified in embodied contexts where the model's outputs translate into physical actions. A household robot that uses a vision-language model to interpret its environment and plan actions must be robust against adversarial visual inputs placed in the physical world. An attacker could print an adversarial image on a sheet of paper and place it in the robot's environment, causing the model to misinterpret the scene and take unsafe actions. The physicality of this attack vector means that traditional digital security measures such as input sanitization are insufficient. The model must be robust at the perceptual level, correctly interpreting visual scenes even when they contain adversarial elements.
The interaction between multi-modal perception and physical action creates failure modes that do not exist in text-only systems. A model that hallucinates an object in a visual scene may plan actions that interact with the hallucinated object, leading to unexpected physical behavior. A model that fails to detect a real object due to adversarial visual interference may plan actions that collide with the undetected obstacle. These failure modes have no analogue in text-based systems and require dedicated evaluation frameworks that test the full perception-to-action pipeline under adversarial conditions.