MCP Test Harness

Latest Release - install from PyPI (pip install mcp-test-harness) or use the OCI image on GitHub Container Registry: ghcr.io/vaquarkhan/mcp-test-harness. Tags: latest and 3.0.8 (runtime, mcp-test entrypoint), dev and 3.0.8-dev (pytest + dev extras). Browse tags on GHCR · docker run --rm ghcr.io/vaquarkhan/mcp-test-harness:latest --version · docs/DOCKER.md · docs/RELEASING.md

Author: Vaquar Khan · License: MIT (NOTICE) · Cite: CITATION.cff · Sponsors: SPONSORS.md

For C-suite & directors

What it is: a CI gate for MCP servers - the connectors that let AI agents call tools, data, and APIs. Teams write deterministic tests once; every pull request proves the server still works before it ships.

Why it matters

Not an LLM eval product. It does not score model answers. It proves your MCP server is correct, fast enough, and hardened enough for production. Pair with runtime security (MCP-Bastion) and IDE config scanning (mcp-shark). Positioning: docs/POSITIONING.md · comparison: docs/COMPARISON.md.

For architects & tech leads

Role in the stack: deterministic, code-first test automation between local MCP Inspector exploration and production. Fits stdio / SSE / HTTP transports; plugs into GitHub Actions (Marketplace Action), JUnit consumers, and Code Scanning (SARIF).

Architecture (one run)

mcp-test CLI → config → discover test_*.py → schedule (optional parallel)
  → real MCP session (lifecycle + transport) → assertions → reports

Deep dives: docs/ARCHITECTURE.md · docs/DECISIONS.md · docs/ENTERPRISE_GOVERNANCE.md · RFCs under docs/design/.

For developers

Fastest path

pip install mcp-test-harness
mcp-test init --server-command "python your_server.py"
# edit tests/test_*.py - then:
mcp-test --config mcp-test.yaml

Zero-config probe: mcp-test try --server-command "…". CI: Marketplace Action mcp-test-harness. Docs hub: docs/QUICK_START.md → docs/DEVELOPER_GUIDE.md → docs/CI_AND_REPORTS.md.

MCP Test Harness is a pytest-style framework for MCP servers: the mcp-test CLI discovers, runs, and reports tests automatically - replacing much of the manual validation you might do in the MCP Inspector. Repository: github.com/vaquarkhan/mcp-test-harness.

Documentation: QUICK_START · DEVELOPER_GUIDE · CI & reports · performance · comparison · discovery checklist. Community: Issues for bugs; PRs for docs and examples.

What ships end-to-end

Everything below is implemented, tested, and documented in this repo (840+ tests, 100% lib coverage gate, e2e dogfood in CI). Version pins live only in PyPI, CHANGELOG.md, and install/Docker tags - not in this map.

Full history: CHANGELOG.md · roadmap: docs/ROADMAP.md.

Visual guide

Developer journey

pip install → mcp-test init → write test_*.py → mcp-test → merge with confidence.

Three testing modes

Functional correctness, snapshot regression, and SLO-style performance - not three separate tools.

All visual assets

Every diagram lives in docs/images/. Quick reference:

We eat our own dogfood

A test harness should prove itself. This repo runs 840+ pytest cases with a 100% line coverage gate on src/mcp_test_harness, plus end-to-end dogfood tests that spawn the real mcp-test CLI against a bundled FastMCP server:

# Same gate CI enforces on every PR
coverage run -m pytest tests/ --ignore=tests/test_workspace.py -q
coverage report --fail-under=100

# E2E dogfood only
python -m pytest tests/test_harness_dogfood_e2e.py -m e2e -v

Fixtures live under tests/fixtures/ (minimal_mcp_server.py, harness_self_test/). See docs/DEVELOPER.md and CONTRIBUTING.md.

Why teams adopt this

MCP Test Harness is the deterministic, CI-native gate for MCP servers - one run proves correctness, speed, and security baselines without an LLM in the loop. Full positioning: docs/POSITIONING.md.

Four differentiators

Three testing modes in one tool

• Functional: assert_tool_call, assert_resource_read, schema validation
• Regression: assert_snapshot, assert_tool_idempotent
• Performance: assert_latency, assert_throughput, assert_latency_within_baseline

Pair with mcp-shark for IDE config security and MCP-Bastion for runtime protection - see docs/COMPARISON.md.

MCP Test Harness supports Responsible AI and governance programs: schema contract checks, security payload packs (@marker(tags=["security"])), coverage gaps in reports, and EU AI Act demo packs.

Documentation

Hub (table of all guides + suggested reading order): docs/README.md

For production security (prompt injection defense, PII redaction, rate limiting, RBAC), see MCP-Bastion - the companion security middleware; this repo is for test automation.

Core Features

Beginner demo packs by testing type: functional-testing · regression-testing · performance-testing (each includes runnable tests plus JSON/JUnit/HTML report config).
Platform QA (trace + chaos + generate): platform-qa
Full scenario index: examples/feature-demo/README.md

Ecosystem (Conformance, evals, benchmarks)

MCP Test Harness is deterministic (your tests call the protocol directly; no LLM required). The wider MCP space includes protocol conformance suites, agent/LLM evaluation frameworks, and model benchmarks. A concise map of tools, when to use each, and how they complement (not replace) the harness is in docs/COMPARISON.md.

Installation

Current stable version: 3.0.8 (see CHANGELOG.md). Core harness (lightweight: mcp + YAML + anyio; no MCP-Bastion / Presidio stack):

pip install mcp-test-harness
# pin, if you need a fixed version:
# pip install mcp-test-harness==3.0.8

Same release as a container (GHCR, no local Python): docker pull ghcr.io/vaquarkhan/mcp-test-harness:3.0.8 or :latest - see the image section for docker run and dev tags.

Optional mcplint / MCP-Bastion pin helpers (transitive set can be large; same as a full Bastion install):

pip install mcp-test-harness[mcplint]
# or the historical PyPI name for the monorepo shim:
pip install mcplint

Or from source:

git clone https://github.com/vaquarkhan/mcp-test-harness.git
cd mcp-test-harness
python -m venv .venv && source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -e ".[dev]"
mcp-test --version

Docker

One-page guide (PyPI, container registries, Mermaid build diagram, docker run copy-paste): docs/DOCKER.md · System diagram (flow + sequence): docs/ARCHITECTURE.md · Visual Studio Code & Cursor (snippets, Mermaid, extensions): docs/EDITORS.md

Pre-built runtime and dev (test tooling) images are defined in the repo Dockerfile (and .dockerignore keeps the build context small). Each v* git tag triggers CI that pushes ghcr.io/vaquarkhan/mcp-test-harness: e.g. :3.0.8, :latest, :3.0.8-dev, :dev. Quick pull: docker pull ghcr.io/vaquarkhan/mcp-test-harness:latest. All versions (GHCR) · docs/RELEASING.md · docs/DOCKER.md.

Build the default image (from the repository root; requires Docker):

docker build -t mcp-test-harness:local .

Smoke the CLI (the image entrypoint is mcp-test):

docker run --rm mcp-test-harness:local --version

Run mcp-test against a project mounted into the working directory (paths below use POSIX shells; on Windows, use PowerShell and replace $PWD with your project path, e.g. ${PWD} in Git Bash, or the full C:\... form):

docker run --rm -v "$PWD":/work -w /work mcp-test-harness:local

The default command shows mcp-test --help. Pass the same arguments you use locally, for example docker run --rm -v "$PWD":/work -w /work mcp-test-harness:local . to discover and run tests in the current config.

Build the dev image and run the test suite (requires your test tree mounted into /work):

docker build -t mcp-test-harness:dev --target dev .
docker run --rm -v "$PWD":/work -w /work --entrypoint pytest mcp-test-harness:dev tests/ -q

For coverage as in pyproject.toml, you can add --cov=src/mcp_test_harness if your tests/ and config are on the mount.

Windows (PowerShell), using Docker Desktop, mount the current directory, for example:

docker run --rm -v "${PWD}:/work" -w /work mcp-test-harness:local

Size and first-build time depend on what you install: the default runtime image matches mcp-bastion-python-free core dependencies. If you add [dev], [mcplint], or pip install mcp-bastion-python in the same environment, the resolver may pull a large transitive set (e.g. Presidio / NLP and, on many Linux x86_64 wheels, very large ML/CUDA-related packages). The first docker build with those extras can take a long time and produce a multi-gigabyte image. That is expected for a full install of the Bastion tree, not a bug in the Dockerfile when you opt into that stack.

Note: The Docker image is optional; many teams use pip install in CI. Use the image when you need a reproducible, Python-isolated environment without a local venv, or a portable mcp-test in pipelines that standardize on containers.

Quick Start

0. Scaffold a starter (optional)

With mcp-test on your PATH (after pip install):

mcp-test init

This writes tests/test_mcp_server_example.py and a minimal mcp-test.yaml. Set your real launch command, for example:

mcp-test init --server-command "python -m your_package.mcp"

Options: mcp-test init --help (custom --dir, --filename, --no-config, --force).

Check the server first (no tests): mcp-test doctor uses the same mcp-test.yaml (or --server-command) to start the server, run the MCP handshake, print the protocol version, list tools / resources / prompts, and optionally run the same post-connect schema checks as a normal run. Exits 0 when healthy, 1 on startup or schema errors. See mcp-test doctor --help.

Editor snippets: the repo includes .vscode/mcp-test-harness.code-snippets - in VS Code or Cursor, type prefixes like mcp-assert-tool or mcp-test-async in a *.py file to insert common patterns.

1. Write a test

Create tests/test_my_server.py:

from mcp_test_harness import assert_tool_call, assert_capabilities

async def test_server_has_tools(mcp_server):
    """Verify the server advertises tool capabilities."""
    await assert_capabilities(mcp_server, {"tools": {}})

async def test_echo_tool(mcp_server):
    """Call the echo tool and check it works."""
    result = await assert_tool_call(mcp_server, "echo", {"message": "hello"})
    assert result is not None

The mcp_server parameter is a built-in fixture. The harness automatically starts your server, connects via MCP, performs the initialize handshake, and injects a ready-to-use session.

2. Run it

mcp-test --server-command "python my_server.py" tests/

Output:

  [PASS] test_server_has_tools (45.2ms)
  [PASS] test_echo_tool (120.8ms)

2 passed, 0 failed, 0 errored, 0 skipped
Total time: 312.5ms

3. Add a config file

Create mcp-test.yaml:

server:
  command: python my_server.py
  transport: stdio

test:
  dirs: [tests/]
  timeout: 30

report:
  format: junit
  output: reports/results.xml

Then just: mcp-test

Assertion Reference

assert_tool_call -- invoke a tool and validate the response

# Basic: fail if the tool returns an error
await assert_tool_call(mcp_server, "echo", {"message": "hello"})

# With expected output
await assert_tool_call(mcp_server, "add", {"a": 1, "b": 2},
    expected=[{"text": "3", "isError": False}])

# Validate arguments against the tool’s inputSchema (requires `jsonschema`)
await assert_tool_call(
    mcp_server, "add", {"a": 1, "b": 2},
    validate_against_input_schema=True,
)

# Use the return value
result = await assert_tool_call(mcp_server, "get_data", {})
assert len(result.content) > 0

Other helpers: assert_tool_schema, assert_protocol_version, assert_tool_idempotent, assert_latency, assert_tool_call_validates_input, assert_tool_denied, assert_authorization_boundary - see Part 3b in the Developer Guide.

assert_resource_read -- read a resource and check content/MIME type

await assert_resource_read(mcp_server, "file:///config.json",
    expected_content='{"debug": true}',
    expected_mime_type="application/json")

assert_prompt -- get a prompt and validate messages

await assert_prompt(mcp_server, "summarize",
    arguments={"text": "The quick brown fox."},
    expected_messages=[{"role": "assistant", "content": "Summary: A fox."}])

assert_capabilities -- verify server capabilities

await assert_capabilities(mcp_server, {"tools": {}, "resources": {}})

assert_snapshot -- regression detection via stored snapshots

from pathlib import Path
from mcp_test_harness import assert_snapshot

async def test_stable_output(mcp_server):
    result = await mcp_server.call_tool("generate_report", {})
    await