agent-testkit

Universal E2E and manual acceptance testing for Claude Code — with a written protocol for every run.

Three skills, one architecture: the skills are the engine, and two small markdown files in your repo are the plan. Generate the plan once, then run it autonomously or conduct it by hand — either way you get a timestamped protocol file with every result, every failure, and every fix.

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    init["/testkit:test-init<br/>repository analysis + interview"]

    subgraph repo["repository root (committed)"]
        direction LR
        planE2E["TESTS_E2E.md"]
        planMan["TESTS_MANUAL.md"]
    end

    e2e["/testkit:test-e2e<br/>autonomous execution, fixes on failure"]
    man["/testkit:test-manual<br/>guided execution: user runs, Claude conducts"]

    protoE2E["tests/integration/<br/>test-e2e-protocol-*.md"]
    protoMan["tests/manual/<br/>test-manual-protocol-*.md"]

    init -- generates --> planE2E
    init -- generates --> planMan
    planE2E -- read by --> e2e
    planMan -- read by --> man
    e2e -- writes --> protoE2E
    man -- writes --> protoMan

Why

Agents write code faster than we verify it. The missing piece isn't more tests — it's evidence: a reviewable record of what was checked, what failed, what got fixed, and what was skipped and why. agent-testkit makes the test protocol a first-class artifact, and makes the same plan runnable in two modes:

test-e2e — fully autonomous. Every phase in order, failures investigated and fixed immediately, nothing skipped silently (a phase that can't pass is logged BLOCKED, not swept under the rug).
test-manual — inverted rules. You execute every command and GUI action; Claude presents one step at a time, judges your reported output PASS/FAIL against the plan's expectations, proposes fixes but never applies them unasked, and keeps the books.

Because both modes read the same plan format, coverage stays in sync: the things a machine can assert live in TESTS_E2E.md, the things that need human eyes (GUI flows, look-and-feel, hardware) live in TESTS_MANUAL.md, with shared phases numbered identically.

Install

As a Claude Code plugin (recommended):

/plugin marketplace add knowhowlab/agent-testkit
/plugin install testkit@agent-testkit

Or copy the skills straight into a project (no plugin namespace):

git clone https://github.com/knowhowlab/agent-testkit
cp -r agent-testkit/skills/* your-project/.claude/skills/

Installed as a plugin, the commands are /testkit:test-init, /testkit:test-e2e, /testkit:test-manual. Copied directly, they're /test-init, /test-e2e, /test-manual. Claude also triggers them automatically from natural phrasing ("run the e2e test", "let's test this by hand").

Quickstart

cd your-project && claude

/testkit:test-init          # recon → interview → writes both plan files
# review the generated TESTS_E2E.md / TESTS_MANUAL.md, commit them

/testkit:test-manual        # first run: shake out the plan by hand
/testkit:test-e2e           # from then on: autonomous regression runs
/testkit:test-e2e quick     # scopes work as arguments

The plan format

Plans are plain markdown with a line-anchored grammar (Run:, Do:, Expect:, On fail:, Destructive:, Tags: …) — reviewable in a PR, greppable, and deterministic enough for an agent to execute. The full contract lives in skills/test-init/references/plan-format.md; annotated skeletons in templates/; a complete worked example (a fictional taskboard web app — React front end, Node API, Postgres via docker compose) in examples/taskboard/, including sample protocols from both modes in examples/taskboard/protocols/.

The parts worth knowing:

Every step has an Expect: — no "run it and see".
## Facts are load-bearing: a documented caveat ("this endpoint may 500 upstream — that is not an auth failure") stops the autonomous executor from "fixing" expected behavior.
Destructive: steps state their consequence before anyone runs them.
Scopes (quick, no-build, keep-up, or your own) skip phases by number or tag, and skips are logged as SKIPPED (scope) — visible, not silent.
 fences protect hand-tuned sections from test-init's update mode.

What a protocol looks like

# E2E Test Protocol — taskboard

**Date**: 2026-07-09 14:02 UTC · **Branch**: main · **Commit**: 3f9c2a1
**Plan**: TESTS_E2E.md (format v1) · Scope: full

| Phase | Result | Tests | Passed | Fixed | Failed | Blocked | Skipped |
|-------|--------|-------|--------|-------|--------|---------|---------|
| 1 — Quality gates      | PASS | 3 | 2 | 1 | 0 | 0 | 0 |
| 6 — Failure & recovery | PASS | 5 | 5 | 0 | 0 | 0 | 0 |

#### Issue 1.1: `no-floating-promises` × 5 in new e2e helpers
- **Error**: @typescript-eslint/no-floating-promises (e2e/helpers/archive.ts)
- **Root cause**: pre-commit lints staged src/** only; e2e/ never linted
- **Fix**: await the five fire-and-forget expect.poll calls (test-only)
- **Verified**: Yes — repo-wide lint clean; helpers re-exercised in Phase 5

The manual protocol reads differently — it records who executed and quotes the tester's own words as evidence, including the entries that make manual runs worth protocoling: a 429 correctly judged not-a-failure because the plan's Facts document the rate limiter, a SKIPPED (user) step, and a false FAIL traced to the tester's ad-blocker rather than the app. Full samples of both: examples/taskboard/protocols/.

Fixes made during a run are left uncommitted and flagged in the final summary — the protocol is evidence, the diff is yours to review.

The three skills

Skill	Mode	What it does
`test-init`	interactive	Recons the repo (task runners, compose files, CI, quickstarts), interviews you for what it can't infer, writes both plan files, sanity-checks every referenced path. Update mode detects drift and proposes diffs.
`test-e2e`	autonomous	Executes `TESTS_E2E.md` end to end. Fixes failures immediately, retries timing issues once, logs `BLOCKED` rather than skipping, re-runs the gates at the end to prove fixes didn't regress anything.
`test-manual`	conducted	Walks you through `TESTS_MANUAL.md` one step at a time. You execute; it evaluates, proposes fixes on failure, and records who did what — quoting your pasted output for anything surprising.

FAQ

Does this replace my unit tests? No — Phase 1 of a generated plan usually runs them. This covers the layer above: does the built, running, integrated thing behave, recover, and package correctly.

Why two plan files instead of one? Different audiences. E2E steps are written for machine assertion; manual steps are one human action each, including GUI steps no CLI can express. Parity numbering keeps them honest with each other.

Can I write plans by hand? Absolutely — start from templates/. The format is designed to be authored by humans and agents alike.

Claude.ai / Cowork? The skills follow the open Agent Skills format, so they can be uploaded as custom skills there too; the !-prefix hint and slash-command names are Claude Code conveniences.

License

MIT — see LICENSE.

Agent Testkit

README