Agent Egress Bench

A standardized test corpus for evaluating AI agent egress security tools, covering secret exfiltration, prompt injection, SSRF, hostname exfiltration, MCP tool poisoning, chain detection, MCP drift, A2A protocol scanning, WebSocket DLP, encoding evasion, shell obfuscation, and cryptocurrency/financial data protection. Current loader-backed counts are in cases/STATS.md.

This tests the security tool, not the agent. Most benchmarks in this space (AgentDojo, InjecAgent, CyberSecEval, AgentHarm) test whether the LLM behaves correctly. This one tests whether the firewall, proxy, or scanner sitting between the agent and the network catches the attack.

┌─────────────────────┐     ┌──────────────────────┐     ┌──────────┐
│  AI Agent           │     │  Security Tool        │     │          │
│  (has secrets,      │────▶│  (proxy / firewall /  │────▶│ Internet │
│   runs tools)       │     │   MCP wrapper)        │     │          │
└─────────────────────┘     └──────────────────────┘     └──────────┘
                                     ▲
                            agent-egress-bench
                            tests THIS layer

Why this exists

AI agents that can browse the web, call APIs, and use MCP tools need network-layer security. An agent with access to secrets and an internet connection is an exfiltration risk, whether through prompt injection, tool poisoning, or simple misalignment.

Tools exist to sit between agents and the network (proxies, firewalls, MCP wrappers). But there was no standard way to test them. This corpus fills that gap: a shared set of attack cases that any security tool can run against.

What's in the corpus

Counts are logical cases, not fixture files. Most cases are single JSON files; MCP drift cases are multi-file before/after snapshots, and each drift directory counts as one case.

The loader-backed statistics include the block, allow, and warn-class breakdown used to assess containment and false-positive behavior.

Most cases are self-contained JSON files with the attack payload, expected verdict (block or allow), severity, capability tags, and a machine-readable reason for the expected outcome. MCP drift cases under cases/mcp-drift/ are multi-file before/after fixtures with case.yaml metadata.

Quick start

Prerequisites: Go 1.25+ for the validator and portable runner. The runner uses its own Go module dependencies for fixtures and multi-file case parsing.

Build the validator:

cd validate && go build -o aeb-validate .

Validate the corpus:

./aeb-validate ../cases

Validate a runner's results or tool profile:

./aeb-validate results path/to/results.jsonl ../cases
./aeb-validate profile path/to/tool-profile.json

Run against a tool. Each tool ships its own runner. The Go program in runner/ is the reference implementation; it brings up HTTP, TLS, WebSocket, DNS, and MCP HTTP fixtures, executes declared transports through the selected adapter, and emits the Gauntlet summary and an optional receipt-scoring profile.

For the pinned Pipelock release, use the portable entry point from a clean Linux clone on origin/main or a tag:

./scripts/run-pipelock-gauntlet.sh --doctor
./scripts/run-pipelock-gauntlet.sh

Run --doctor before the benchmark. It checks the platform, the required commands, an MCP stdio bridge, the working directory, and the reviewed release pin, without starting a run or writing an evidence directory. --doctor-json returns the same checks as JSON. Both commands exit nonzero when a check fails. A ready result is not a prediction that the run will succeed: it does not check the installed Go version, the origin remote, whether the checkout is clean, or network reachability, and the run itself still fails on each of those.

The command downloads the reviewed Pipelock release, verifies its pinned asset digest, published checksum, and reported version, then confines target writes with Landlock and denies Unix-domain sockets with seccomp. It starts the required local fixtures and managed Pipelock processes, runs the single-file and multi-file cases, and leaves one timestamped directory under continuous-gauntlet-runs/. That directory contains the exact internal command, stdout results, stderr, summary, case index, corpus stats, release identity, file digests, and a machine-readable execution decision.

It requires Linux, Go 1.25 or newer, Python 3, Git, curl, jq, tar, GNU timeout, SHA-256 utilities, and one of socat, ncat, or nc. Use --output-dir to place the self-contained run directory somewhere else. The Pipelock reference-runner guide documents the evidence files, explicit development mode, the underlying long-form command, and a neutral scheduling example.

The raw directory intentionally has no made-up public URL. GitHub Actions or another retaining platform adds its real artifact ID and HTTPS location later, without modifying the evidence bytes. Creating a schedule and publishing a result are separate operator decisions.

The repository's scheduled Pipelock lane is read-only and produces review candidates, not automatic public claims. Approved candidates are retained as digest-addressed, hash-linked evidence directories, and a reviewed pull request advances the latest-verified pointer. The included reference renderer verifies and displays score, scope, N/A reasons, false-positive rate, and the canonical run URL together. It renders this repository's first-party Pipelock history and ranks nothing. See Continuous Gauntlet Results for the repository review and publication contract.

For other tools, the runner writes per-case JSONL results to stdout (one object per case, see docs/RUNNER.md) and a Gauntlet summary JSON to the path passed via --output (containment, false-positive rate, non-scoring output-field diagnostics, and per-category results, see docs/gauntlet.md). --emit-receipt-profile additionally writes a byte-reproducible receipt-scoring profile (see docs/RECEIPT-SCORING.md).

A minimal legacy shell example for fetch-only cases lives at examples/pipelock/harness.sh. It covers a single transport (/fetch?url=... GET) and is kept for illustration only — it is not the Gauntlet and will misreport every body, header, WebSocket, MCP, and response-content case. Use the Go runner for any real benchmark.

Gauntlet scoring

The Gauntlet reports containment and false-positive rate separately, plus non-scoring field-presence diagnostics. docs/gauntlet.md owns metric definitions, result states, denominators, and measurement status. Results Use and Attribution owns the claims and disclosures that travel with a public result.

The maintainer's disclosed first-party Pipelock history is published at pipelab.org/gauntlet. Every other publisher owns its results.

What this does NOT test

This corpus has a specific scope. It does not cover:

• Model alignment. Whether the LLM refuses harmful instructions. Use AgentDojo, AgentHarm, or ASB for that.
• Application-layer guardrails. Whether a guardrail API flags a prompt as malicious. Use AgentShield-benchmark for that.
• Code generation safety. Whether the model writes insecure code. Use CyberSecEval for that.
• Authentication or authorization. Whether the agent has valid credentials for the APIs it calls.
• Inbound traffic. What enters the agent's environment. This corpus focuses on outbound (egress) traffic.

If you need to test the model, use a model benchmark. If you need to test the network security layer, use this.

How it works

Cases encode wire inputs and expected verdicts. A runner sends each input through the tool and records the observed result. docs/SPEC.md owns case shape, docs/RUNNER.md owns runner I/O, and docs/gauntlet.md owns result states and scoring.

Writing a runner for your tool

Start from the runner template or the complete Pipelock reference runner. RUNNER.md defines the contract, and ADOPTION.md covers the contribution workflow.

OWASP Agentic Top 10 mapping

Every case category maps to the OWASP Top 10 for Agentic Applications (2026). make check-readme-categories fails when this table and the corpus disagree, so the mapping stays complete as categories are added:

Full mapping with MITRE ATT&CK techniques: docs/OWASP-MAPPING.md

Scope

This corpus evaluates the security tool that sits between an AI agent and the network (a proxy, firewall, or MCP wrapper): given an attack, did the tool catch it. It does not evaluate the agent or model's own behavior. Cases test observable outcomes at the wire level, such as whether an exfiltrated secret in a query string was blocked or whether prompt injection in a tool response was detected.

Each publisher publishes and owns its own results. This repository publishes no ranking, leaderboard, or cross-tool comparison table.

Docs

Core contracts:

• SPEC.md: case schema, field definitions, enums, payload formats
• RUNNER.md: runner input, output, adapter, and verdict-mapping contract
• gauntlet.md: result states, scoring, scope, and publication methodology
• GOVERNANCE.md: neutrality, case immutability, versioning, and compatibility
• RELEASES.md: pinned runner, corpus, and schema release verification
• OCI-RUNNER.md: pinned runner image, reusable Action, devcontainer, and offline operation
• contracts/artifacts.json: machine-readable artifact compatibility inventory

Evidence and publication:

• RECEIPT-SCORING.md: receipt evidence scoring axis for independently verifiable artifacts
• CONTROL-EVIDENCE.md: v0 run-level control-evidence package and verifier contract
• CONTROL-EVIDENCE-V1.md: active v4 registry-bound control-evidence contract
• CAPABILITY-VOCABULARY.md: immutable reporting-label registry and profile evolution policy
• ARTIFACT-PROVENANCE.md: opt-in external schema-valid, authenticated-at(T), and buyer-reproduced provenance assessments
• RESULTS-USE.md: assurance labels, public-result disclosures, and correction rules
• Operator kit: run setup, evidence custody, report template, retention, and appeal routes

Integration and reference:

• GATEWAY-ADAPTER.md: the narrow generic MCP gateway plugin contract and its current limits
• ADOPTION.md: guide for vendors adopting the benchmark
• GLOSSARY.md: definitions of key terms (agent firewall, egress security, etc.)
• OWASP-MAPPING.md: case categories mapped to OWASP Agentic Top 10
• schemas/: JSON Schema files for cases, tool profiles, and results

Contributing

See CONTRIBUTING.md. Cases, runners, and documentation improvements are all welcome.

Case IDs are immutable. Once merged, a case ID never changes. Semantic changes to existing cases require a new case with a new ID.

Governance

This corpus was created by the Pipelock author. Contributions from any vendor or individual are welcome. This repository publishes no ranking, leaderboard, or cross-tool comparison table, and the maintainer awards no verification mark to anyone else's result. Each publisher publishes and owns its own results, under the labels defined in docs/RESULTS-USE.md. Publishing a result that reflects badly on Pipelock needs no notice or approval.

Conflict of interest disclosure: The author builds an agent egress security tool. This corpus was designed to be tool-neutral: cases test observable behavior (did the request get blocked?), not implementation details. The Pipelock runner is a reference implementation, not a privileged position.

Full governance policy: docs/GOVERNANCE.md.

Learn more

• What is an Agent Firewall? — the security architecture this corpus tests
• AI Agent Security: Three Layers — hooks, guardrails, and egress inspection explained
• MCP Vulnerabilities — the MCP attack surface mapped
• Pipelock Gauntlet history — the maintainer's disclosed first-party run history

License

Apache 2.0. See LICENSE.

If this corpus is useful to you, give it a star. It helps others find it.