Frontier Research That Has Defined the AI Security Field

How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition

LLM based agents are increasingly deployed in high stakes settings where they process external data sources such as emails, documents, and code repositories. This creates exposure to indirect prompt injection attacks, where adversarial instructions embedded in external content manipulate agent behavior without user awareness. A critical but underexplored dimension of this threat is concealment: since users tend to observe only an agent's final response, an attack can conceal its existence by presenting no clue of compromise in the final user facing response while successfully executing harmful actions. This leaves users unaware of the manipulation and likely to accept harmful outcomes as legitimate. We present findings from a large scale public red teaming competition evaluating this dual objective across three agent settings: tool calling, coding, and computer use.

Comparing AI Agents to Cybersecurity Professionals In Real-World Penetration Testing

Current cybersecurity benchmarks fail to capture the real-world capabilities of AI agents, leading to significant underestimation of cyber risk. We conducted the first head-to-head comparison of AI agents and professional penetration testers on a live enterprise network to measure true performance gaps. Our findings reveal that purpose-built agents can outperform 90% of human professionals while operating continuously at a fraction of the cost—exposing critical flaws in how the industry evaluates AI capabilities and safety guardrails.

D-REX: A Benchmark For Detecting Deceptive Reasoning In Large Language Models

The safety and alignment of Large Language Models (LLMs) are critical for their respon- sible deployment. Current evaluation methods predominantly focus on identifying and preventing overtly harmful outputs. However, they often fail to address a more insidious failure mode: models that produce benign-appearing outputs while operating on malicious or deceptive internal reasoning.

Adversarial Attacks on Robotic Vision Language Action Models

The emergence of vision-language-action models (VLAs) for end-to-end control is reshaping the field of robotics by enabling the fusion of multimodal sensory inputs at the billion-parameter scale. The capabilities of VLAs stem primarily from their architectures, which are often based on frontier large language models (LLMs). However, LLMs are known to be susceptible to adversarial misuse, and given the significant physical risks inherent to robotics, questions remain regarding the extent to which VLAs inherit these vulnerabilities.

Safety Pretraining: Toward the Next Generation of Safe AI

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

Recent research on jailbreak attacks has focused almost exclusively in settings where LLMs act as simple chatbots. However, now LLMs are increasingly used in agentic workflows, i.e., equipped with external tools and potentially using many steps to fulfill a user’s request. To address potential safety and alignment concerns coming from LLM agents, we introduce AgentHarm, a new benchmark for measuring harmfulness of LLM agents.

Improving Alignment and Robustness with Circuit Breakers

To address the urgent concerns raised by our attack from last July and the numerous jailbreaks that came after, we introduce Circuit Breaking, a novel approach inspired by representation engineering, designed to robustly prevent AI systems from generating harmful content by directly altering harmful model representations. The family of circuit-breaking methods provide an alternative to refusal and adversarial training, protecting both LLMs and multimodal models from strong, unseen adversarial attacks without compromising model capability.

The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

Representation Engineering: A Top-Down Approach to AI Transparency

Building on our initial findings, we ventured into the realm of AI interpretability and control with the introduction of Representation Engineering (RepE). Drawing inspiration from cognitive neuroscience, we developed techniques that enable researchers to 'read' and 'control' the 'minds' of AI models. This approach represented a monumental advancement in demystifying the inner workings of AI, making it possible to tackle issues such as truthfulness and power-seeking behaviors head-on.

Adversarial Attacks on Aligned Language Models

In July 2023, we published the first-ever automated jailbreaking method on large language models (LLMs) and exposed their susceptibility to adversarial attacks. By demonstrating that specific character sequences could bypass sophisticated safeguards, we highlighted a significant vulnerability that has urgent implications for widely-used AI systems. In its wake, adversarial robustness garnered renewed attention, sparking a gold rush of research dedicated to both jailbreaking and defense.

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

Do the Rewards Justify the means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark

OpenOOD: Benchmarking Generalized Out-Of-Distribution Detection

Forecasting Future World Events with Neural Networks

Scaling Out-of-Distribution Detection for Real-World Settings

What Would Jiminy Cricket Do? Towards Agents That Behave Morally

PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

Globally-Robust Neural Networks

APPS: Measuring Coding Challenge Competence With APPS

MMLU: Measuring Massive Multitask Language Understanding

Aligning AI With Shared Human Values

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Pretrained Transformers Improve Out-of-Distribution Robustness

Overfitting in Adversarially Robust Deep Learning

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Fast Is Better Than Free: Revisiting Adversarial Training

Natural Adversarial Examples

Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

Randomized Smoothing: Certified adversarial robustness via randomized smoothing

Using Pre-Training Can Improve Model Robustness and Uncertainty

ImageNet-C: Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Deep Anomaly Detection with Outlier Exposure

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

Provable Defenses Against Adversarial Examples Via the Convex Outer Adversarial Polytope