Red Teaming Agentic Capabilities in NVIDIA NeMo Agent Toolkit

Research

Overview

Agentic systems expand the safety and security surface area beyond the base model. In addition to user prompts, the system’s tool calls, intermediate state, memory, and multi-agent handoffs can all become points where failures emerge and compound. As a result, model-level checks alone often miss how issues appear in end-to-end execution.

To help developers evaluate agentic systems at the workflow level, Lakera contributed red teaming capabilities to NeMo Agent Toolkit. This post summarizes what shipped, where to find it, and what you get out of a run: structured findings, normalized risk scoring, and signals for how risk propagates—or attenuates—across an agent workflow.

At Lakera, we focus on adversarial testing and red teaming of agentic AI systems, which is why we’ve been closely working with frameworks like NVIDIA’s NeMo Agent Toolkit to explore how these systems fail under real-world attack conditions.

The Enterprise Playbook for Agentic AI Security

AI systems now retrieve data, invoke tools, and act across enterprise workflows. Get the playbook to learn how to secure AI across employees, applications, and agents.

Inside the Playbook

• Why traditional security models fall short
• The three new AI exposure surfaces
• How to secure the execution layer
• What a unified AI Defense Plane looks like in practice

What’s included in NeMo Agent Toolkit v1.4

Lakera’s contribution is delivered as part of the NeMo Agent Toolkit Safety & Security example (Retail Agent). The example integrates a systematic red teaming workflow designed to exercise an agent system end-to-end covering user input, tool boundaries, and multi-step execution paths.

Key Features

• Tailored, architecture-specific threat models
• Systematic attack injection against key agent interfaces and components (including direct and indirect inputs)
• Evaluation at both component boundaries and full workflow execution
• Automated risk report generation with quantified, normalized scores for consistent comparison
• Risk propagation analysis to identify where issues spread, amplify, or get filtered across steps

Why System-Level Evaluation Matters for Agents

For agentic workflows, failures rarely live in a single component. A vulnerability introduced in one stage—such as a manipulated external input or an unsafe tool response—can influence downstream reasoning and decisions. This means that testing individual models or tools in isolation can produce a false sense of security, even when the overall workflow remains brittle.

The red teaming workflow is designed to evaluate the agent as a system. It does this by injecting adversarial conditions and measuring outcomes across multi-step execution, producing evidence about where failures originate and how they move through the workflow.

Demo Example: Try it Yourself

The release includes a sample agent to help explore the red teaming capabilities, included in the NeMo Agent Toolkit repository under: examples/safety_and_security/retail_agent

Agent Red Teaming Output

A red team evaluation run produces a structured risk report that you can use to understand and compare agent behavior across iterations.

Key Concepts:

• A scenario defines a specific attack setup, combining an injection payload, a target point in the agent workflow (e.g., user input, indirect data source), and success criteria for evaluation.

Key Metrics:

• Risk Score (0–1): A normalized measure of vulnerability where higher scores indicate successful attacks. Enables consistent comparison across scenarios and over time.
• Attack Success Rate (ASR): The percentage of attempts where an injected attack achieved its intended effect on the agent's behavior.

Report Output:

• Summary: Overall risk score, attack success rate, and run statistics
• Per-scenario breakdown: Results for each attack type with mean, min, and max scores to surface variance in agent behavior
• Grouped views: Results sliced by scenario category, risk taxonomy (e.g., data exfiltration, harmful content), and evaluation point for identifying patterns across related attack types
• Score distributions: Visualizations showing whether failures are consistent or intermittent

Practical Usage Pattern

This tooling is intended to fit into an iterative development loop:

1. Define the agent workflow, specifying tools, data sources, and execution paths.
2. Run a baseline red team evaluation against the target configuration.
3. Review the risk report to identify which scenarios succeeded, where failures originated, and how risk propagated through the workflow.
4. Apply mitigations such as guardrails, output validation, prompt hardening, etc.
5. Re-run the evaluation and compare normalized scores against the baseline.
6. Repeat as needed.

Sample Report

This report summarizes results from a red team evaluation of the sample Retail Agent included in NeMo Agent Toolkit v1.4. The retail agent is configured without additional defense layers in this evaluation.

Summary

Overall Risk Score ↓
0.562
Attack Success Rate ↓
55.6%
Scenarios
9
Workflow Runs
45
Total Evaluations
45
Successes
45
Failures
0

Per-Scenario Results

Interpreting the Sample Report

Findings from sample red team evaluation:

Conclusion

As agentic systems move closer to production, adversarial testing and red teaming become essential to understanding how these systems behave under real-world conditions. Lakera Red is designed to help teams systematically test, evaluate, and harden agentic AI systems built on modern frameworks like NVIDIA NeMo as they scale beyond experimentation.