Agent Audit Kit

Security scanner for MCP-connected AI agent pipelines. Finds misconfigurations, hardcoded secrets, tool poisoning, rug pulls, trust boundary violations, and tainted data flows across 13 agent platforms.

Two things it does that hosted scanners can't:

1. Runs fully offline and deterministically. Your code, configs, and secrets never leave the machine; the default scan path makes zero network calls, and the same input always yields the same finding (no model in the loop). This is measured, not just claimed: 20/20 identical runs → one shared SHA-256 finding-set digest, 0% variance. Scanners that route findings through an LLM judge can't guarantee a byte-identical re-run — so CI diffs, audit re-runs, and regression baselines stay stable here. No account, no telemetry. Precision is measured the same way, not asserted: we publish a reproducible, hand-adjudicated benign-slice HIGH/CRITICAL false-positive rate — with a Wilson confidence interval and any offending rule filed as an issue.
2. Produces auditor-ready compliance-evidence packs. SARIF for the GitHub Security tab plus PDF evidence reports mapped to 13 frameworks (EU AI Act, SOC 2, ISO 27001/42001, HIPAA, NIST AI RMF, and regional regimes) — what you hand an auditor, not just a list of findings.

• 262 rules across 12 security categories, covering the 2026 CVE wave
  • Rule count is computed from the registry and verified in CI (test_rule_count_is_canonical).
• 84 scanner modules including AST-based Python taint analysis and regex pattern scanners for TypeScript/JavaScript and Rust
• 16 CLI commands: scan, discover, pin, verify, fix, score, update, proxy, kill, watch, plus export-rules, verify-bundle, sbom, report, install-precommit, and the Security-Advisories scan flag
• OWASP coverage: Agentic Top 10 (10/10), MCP Top 10 (10/10), Adversa AI Top 25
• Compliance mapping (13 frameworks): EU AI Act Art. 15 + 55, SOC 2, ISO 27001, ISO/IEC 42001, HIPAA, NIST AI RMF, NSA MCP Security CSI (U/OO/6030316-26, May 2026), Singapore Agentic AI, India DPDP 2023, Alabama Personal Data Protection Act (HB 351, 2026), Tennessee SB 1580 Health Care AI (PRA), MCP 2026 Roadmap (May 2026) — PDF reports via agent-audit-kit report --format pdf --framework <name>
• Supply chain: deterministic rule bundle (export-rules), Sigstore-signed releases, CycloneDX + SPDX SBOM (sbom)
• MCP Security Index: weekly public leaderboard at sattyamjjain.github.io/agent-audit-kit — per-server grade cards (A–F), 90-day disclosure policy
• CVE coverage: newly disclosed MCP CVEs are triaged and turned into rules as they land — surfaced automatically by the NVD watcher (cve-watcher.yml) and logged in CHANGELOG.cves.md
• OAuth / spec coverage: the 2026-07-28 final auth profile as a one-command check — agent-audit-kit scan . --profile mcp-2026-07-28 runs RFC 9207 iss validation (AAK-OAUTH-006), RFC 8707 resource indicators (AAK-OAUTH-007), and RFC 9728 Protected-Resource-Metadata discovery (AAK-OAUTH-008). We scanned 748 public MCP configs with it: 0 use RFC 9728 discovery; 100% of remote-auth configs hardcode a static credential — see the dated 2026-07-28 readiness report. The July 2026-07-28 deprecation/stateless pack stays labelled release candidate (the spec ratifies 2026-07-28); every cited SEP was re-verified in the ratification reconciliation
• Zero cloud dependencies — runs fully offline, zero network calls in the default scan path

Why This Exists

In early 2026, 30 MCP CVEs dropped in 60 days. CVE-2026-33032 (Nginx-UI MCP auth bypass, CVSS 9.8) exposed a shared-handler pattern that several other servers have. CVE-2025-59536 turned a project-local Claude Code hook into RCE. CVE-2026-34070 hit LangChain's load_prompt() with absolute-path and .. traversal. CVE-2026-21852 demonstrated source-code exfiltration via a single Claude Code config flag. A 2,614-server survey found 82% of public MCP servers had path-traversal issues. On 2026-05-01, OX Security disclosed 10+ CVEs across DocsGPT, GPT-Researcher, Agent-Zero, LettaAI, LiteLLM, LangFlow, Flowise, Bisheng, and Langchain-Chatchat — all rooted in the same MCP STDIO command-injection class AAK-MCP-STDIO-CMD-INJ-* covers. Meanwhile, every AI coding assistant adopted MCP with sparse security tooling.

AgentAuditKit is the deterministic, auditor-ready OSS scanner that closes the gap.

Architectural note: AAK rules are multi-arm adjudicators (pin arm + source/config arm + explicit-reject short-circuit + adjudicator log), not single-shot regex matchers. See docs/notes/adjudicator-pattern.md for the four-arm structure and the Mozilla / arXiv precedents.

Quick Start

GitHub Action (Recommended)

# .github/workflows/agent-security.yml
name: Agent Security Scan
on: [push, pull_request]

permissions:
  security-events: write
  contents: read

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: sattyamjjain/[email protected]
        id: scan
        with:
          fail-on: high
      - uses: github/codeql-action/upload-sarif@v3
        if: always()   # upload even if the scan failed the build
        with:
          sarif_file: ${{ steps.scan.outputs.sarif-file }}

The scan step emits SARIF; the upload-sarif step publishes it, so findings appear as inline PR annotations and in the Security tab. See SARIF Integration.

CLI

pip install agent-audit-kit
agent-audit-kit scan .

Pre-commit Hook

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/sattyamjjain/agent-audit-kit
    rev: v0.3.55
    hooks:
      - id: agent-audit-kit

Try It Now

Scan a deliberately vulnerable config to see AgentAuditKit in action:

git clone https://github.com/sattyamjjain/agent-audit-kit
cd agent-audit-kit
pip install -e .
agent-audit-kit scan examples/vulnerable-configs/04-hook-exfiltration/

11 vulnerability examples covering all security categories in the examples/ directory. See also the damn-vulnerable-MCP-server case study.

What It Scans

262 rules total. Every finding includes severity, evidence, remediation, OWASP references, Adversa references, and CVE links where applicable.

MCP Server Card scanning (SEP-1649)

The MCP discovery track (SEP-1649, superseded-in-draft by SEP-2127) has servers publish a server card at /.well-known/mcp/server-card.json — a JSON document (serverInfo, transport, capabilities, authentication, tools[]) that a client fetches and trusts before it connects, ahead of the 2026-07-28 MCP spec finalization. That makes the card an attack surface in its own right.

agent-audit-kit statically audits a server card — from a committed file, or an opt-in fetch — with four deterministic rules (AAK-MCP-CARD-001..004, category MCP Server Card): tool-description poisoning, transport/capability mismatch, missing/placeholder provenance, and over-broad capability claims. It stays offline and deterministic — the default scan path makes zero network calls; card fetching only happens behind an explicit flag (AAK_FETCH_SERVER_CARD_URL, or the crawler's --server-cards network pass), and the same card always yields the same findings.

# audit a committed / saved server card, fully offline
agent-audit-kit scan path/to/server-card.json

# opt-in prevalence sweep of discoverable cards (network; writes results-<date>.json)
python benchmarks/crawler.py --server-cards --limit 200

Agent Platforms Scanned

Claude Code, Cursor, VS Code Copilot, Windsurf, Amazon Q, Gemini CLI, Goose, Continue, Roo Code, Kiro + user-level global configs.

Language Support

Note: Phase 2 scanners (ssrf_patterns, oauth_misconfig, mcp_auth_patterns, hook_rce, skill_poisoning, mcp_tasks) are regex-based; a tree-sitter AST migration is tracked in issue #22.

CLI Reference

Commands

Scan Flags

Exit Codes

Configuration

Create .agent-audit-kit.yml in your project root:

severity: medium
fail-on: high
ignore-paths:
  - vendor/
  - third_party/
exclude-rules:
  - AAK-MCP-007    # We intentionally don't pin npx versions
include-user-config: false

CLI flags always take precedence over config file values.

GitHub Action Reference

Inputs

Outputs

SARIF Integration

This action emits SARIF (the sarif-file output); it does not upload to Code Scanning itself. Add the canonical github/codeql-action/upload-sarif step so findings appear in the Security tab and as inline PR annotations:

permissions:
  security-events: write   # required for the upload
  contents: read

steps:
  - uses: actions/checkout@v4
  - uses: sattyamjjain/[email protected]
    id: scan
    with:
      fail-on: high
  - uses: github/codeql-action/upload-sarif@v3
    if: always()           # upload even when the scan step failed the build
    with:
      sarif_file: ${{ steps.scan.outputs.sarif-file }}

SARIF output conforms to SARIF 2.1.0 with fingerprints, partialFingerprints, security-severity scores, and %SRCROOT% relative paths. Each finding carries OWASP references and CVE links via the rule-level help; per-finding remediation is in result properties. (No SARIF fixes[] — those require machine-applicable artifactChanges, so emitting prose there would make Code Scanning reject the upload.)

Security Scoring

agent-audit-kit score .
# Security Score: 85/100  Grade: B

Generate an SVG badge for your README: agent-audit-kit score . --badge

Frameworks & Standards

Public OWASP coverage leaderboard

Category-by-category coverage tables — every OWASP slot mapped to the exact AAK rule IDs that cover it — are generated from the live rule registry so they cannot drift (CI regenerates and fails on staleness, scripts/gen_coverage.py):

• OWASP Agentic Top 10 → AAK rules
• OWASP MCP Top 10 → AAK rules

Coverage labels use a published threshold (Full = ≥3 rules, Partial = 1–2, None = 0) — reproducible from the registry, not a self-scored grade. The Microsoft Agent Governance Toolkit (2026-04-02) states 10/10 Agentic coverage as a runtime enforcer; AAK reports its honest per-category static rule counts next to it — a coverage cross-reference, not a head-to-head benchmark.

| Adversa AI MCP Security Top 25 | Fully mapped | | EU AI Act | --compliance eu-ai-act | | SOC 2 Type II | --compliance soc2 | | ISO 27001:2022 | --compliance iso27001 | | HIPAA Security Rule | --compliance hipaa | | NIST AI RMF 1.0 | --compliance nist-ai-rmf | | MCP 2026 Roadmap (May 2026) | --compliance mcp-2026-roadmap |

Tool Pinning & Rug Pull Detection

MCP servers can silently change tool definitions after you approve them. AgentAuditKit detects this:

# Create initial pins (commit tool-pins.json to git)
agent-audit-kit pin .

# In CI, verify nothing changed
agent-audit-kit verify .

Detects: tool definitions changed (AAK-RUGPULL-001), new tools added (AAK-RUGPULL-002), tools removed (AAK-RUGPULL-003).

Comparison

See docs/comparisons.md for a fully-sourced version. Verifiable claims only.

Microsoft AGT is an ally, not a competitor. It governs agents at runtime; agent-audit-kit audits them at CI time and produces the PDF an auditor needs. Use both — the overlap is small, the reinforcement is high. See docs/comparisons.md for the full positioning write-up.

VS Code Extension

A VS Code/Cursor extension is available in vscode-extension/:

cd vscode-extension && npm install && npm run compile

Provides inline diagnostics on file save with quick-fix suggestions.

MCP Security Index

Public leaderboard of MCP servers we scan weekly: sattyamjjain.github.io/agent-audit-kit

• Per-server grade cards (A–F)
• Weekly snapshots in data/history.json
• 90-day coordinated disclosure before anything lands on a public card — see docs/disclosure-policy.md
• Maintainer-fix earlier gets published the day the fix lands, with credit

Research: State of MCP Server Authentication 2026

A reproducible data report over 1,374 distinct public MCP server configs — 664 GitHub-crawled plus 710 official MCP Registry latest-version servers, deduped by content and scanned offline and deterministically (scan date 2026-07-19). Headline: 35.1% (482/1,374) declare a remote server with no authentication, 0% (0/1,374) use RFC 9728 Protected-Resource-Metadata discovery, and 100% (318/318) of inline-auth configs hardcode a static credential. 36.0% carry a critical-severity finding; the median config scores a B, and 99.7% trip OWASP MCP07 (authorization). Full method, per-metric findings (each with n + denominator), the 2026-07-28 / 2027-07-28 migration exposure with explicit coverage notes, disclosure posture, and limitations: research/state-of-mcp-2026/REPORT.md — raw aggregate results.json, corpus provenance registry-manifest.json.

The harness contains no scanner — it reuses agent_audit_kit.engine.run_scan + scoring.compute_score, so every number is reproducible offline and cannot drift from the code (make report regenerates results.json from the committed manifest; the corpus refresh is the one network step, fetch_registry.py):

make report   # regenerate the report numbers, offline + deterministic

Scan your own in 30s, fully offline: pip install agent-audit-kit && agent-audit-kit scan . See which AAK rules cover each OWASP MCP slot in the public coverage leaderboard.

CVE Response

AgentAuditKit tracks newly disclosed MCP CVEs and ships rule coverage on a best-effort basis, recorded in a public ledger (CHANGELOG.cves.md). A GitHub Action watches NVD's MCP keyword feed every 6 hours and files a tracking issue for each new disclosure.

Supply chain

Every v* release publishes:

• Wheel + sdist on PyPI via OIDC Trusted Publisher
• Docker image on GHCR (ghcr.io/sattyamjjain/agent-audit-kit:<tag>) with SLSA provenance attestation
• Sigstore keyless-signed rule bundle (rules.json + rules.json.sha256)
• CycloneDX + SPDX SBOM (sbom.cdx.json, sbom.spdx.json)

Verify a bundle:

agent-audit-kit verify-bundle rules.json --signature rules.json.sigstore

Contributing

git clone https://github.com/sattyamjjain/agent-audit-kit
cd agent-audit-kit
pip install -e ".[dev]"
pytest -v                          # 1,100+ tests
ruff check .                       # Lint
mypy agent_audit_kit/              # Type check
agent-audit-kit scan .             # Self-scan

See CONTRIBUTING.md for the full development guide.

Security

Report vulnerabilities via GitHub Security Advisories or see SECURITY.md.

License

MIT