Vigiles

You review every PR. Nothing reviews your CLAUDE.md.

Your harness — the CLAUDE.md or AGENTS.md rules, skills, subagents, and hooks steering your agent — is the one part nobody checks. Nobody verified it's real. Nobody tested it works. That's not a system. That's vibes.

And vibes break silently mid-task: a subagent wired to a tool that doesn't exist, two skills your agent can't tell apart, one helper quietly able to read your secrets and send them out.

vigiles[^name] checks your harness is real, not just well-formed — Claude Code and Codex alike. One command, no key, no config, safe on any repo:

npx vigiles audit

It's free and open-source, runs entirely on your machine, and never bills per token. (eval is the only step that calls a model — on your own Claude subscription.) Here's what it caught on plugins people actually ship. ↓

What it caught

Like Google's Lighthouse, but for your agent harness. One command grades it A–F across five categories, leads with a plain-English verdict — "two one-line fixes away from a B" — and ranks every fix by the points it buys back:

• Truthfulness — do the references resolve?
• Triggering — do skills fire, without colliding?
• Structure — are tool contracts and configs valid?
• Safety — any way for the agent to leak your data?
• Tested — does the harness ship tests?

And it closes the loop from prose to enforcement: your rules → enforced maps each rule you wrote to the lint rule that actually enforces it — already on, one config line away, or silently turned off (below).

These are real scans of public plugins — run npx vigiles audit <any-repo> for your own. The examples below use Claude Code subagents; the same checks run on Codex AGENTS.md, skills, and hooks. ↓

Proof 1 — a tool your agent thinks it has and doesn't

✗ tester — Tool "AskUserQuestion" is never available to a subagent.
    → remove or correct it — it's silently dropped from the contract.

This subagent — a helper your main agent hands work to — lists a tool that doesn't exist for it. The harness drops it without a word, so the agent quietly loses a capability it thinks it has. The markdown is perfectly valid. vigiles catches it and hands you the one-line fix.

Proof 2 — two skills your agent can't tell apart

✗ Triggering   0  (0/100)
    └ 45 pairs of near-identical skill descriptions — the agent can't tell them
      apart, so the wrong one fires  (e.g. "agent-coder" ↔ "agent-tester", 83% alike)

One popular plugin ships 45 pairs of near-identical skill descriptions. Your agent picks a skill by reading them — so when two match, it fires the wrong one. Still perfectly valid markdown. How triggering works →

Proof 3 — a rule you wrote that nothing enforces

Point vigiles at your own repo and the rules section maps each prose rule to the lint rule that enforces it — then checks your config. Representative output:

Your rules → enforced   1 of 4 enforced · 2 one line away · 1 contradicted by config
    └ "always use ===" → eqeqeq is set to "off" in your ESLint config
       your CLAUDE.md says enforce it; your config quietly turns it off

You wrote the rule. Your agent treats it as gospel and follows it — until it doesn't, and nothing tells you which time. vigiles checks each mapped rule three ways: enforced, one line away, or — the one people screenshot — documented but silently turned off. Deterministic, no model. Acting on the map is opt-in and agent-driven: enable a rule in one line (the strengthen skill does it for you), turn an action rule like git push into a compiled hook, leave judgment calls as prose. Nothing runs a model — or changes your config — unless you ask. How enforcement works →

Proof 4 — it can quietly read your secrets and send them out

◑ Safety   80  (80/100)
    └ subagent "tester" holds all three lethal-trifecta legs:
        reads private data (Bash, Read) · takes in untrusted web content (WebFetch)
        · can send data out (Bash, WebFetch)

Hand one subagent all three powers and a poisoned web page can make it read your .env and POST it anywhere — no exploit code, just the tools it was given. The 80 looks like a B — and that's the trap: a healthy grade hiding a subagent that's a data-leak waiting to happen. vigiles spots it from the tool list alone, free, no model.

And the guard you'd write to stop it usually doesn't work: a widely-copied "safety hook" pattern blocks only 2 of 7 classic dangerous commands in our battery — the other five sail through while the hook looks like it's working. A hook vigiles compiles for you blocks 7 of 7, because the exit code and JSON contract are generated, not hand-written. The safety-hook battery →

That's the whole idea: it checks your harness against reality, not style. Every tool, hook, file, script, and skill you reference is verified to actually resolve — and where you name a linter rule, it's checked to exist and be enabled (ESLint, Ruff, Clippy, and more). Everything it catches → · point audit at a whole marketplace and it ranks every plugin the same way.

How it works — vibes → verified

audit shows you where your setup is still vibes. Turning that into verified is four commands over one engine — and almost none of it needs a model or a key.

One engine, two doors. audit is the local report; lint is the CI gate that fails the build on the same deterministic checks — broken refs, bad tool contracts, dead hooks, skill collisions (Proofs 1–2). test and eval go further: past does it exist to does it work. (init / compile / eject manage the optional typed-spec layer for the structural rules no linter can express — a graduation step you rarely run by hand.)

🔎 Lint — your instructions stop lying

Every path, script, symbol, and rule verified against reality — plus tool contracts, skill collisions, and dead hooks (the catches above). You don't write the checks, and you don't port anything into a spec: lint reads the CLAUDE.md or AGENTS.md you already hand-edit and checks it as-is. Rules that map to a real linter rule run in your own config (ESLint, Ruff), not a shadow layer. How →

🧪 Test — does the harness actually do its job?

A hook that blocks nothing, a skill that hijacks unrelated prompts, context that never reaches the model — each passes a naive "did it run?" check. That gap is false confidence: a guard that looks like it works and silently doesn't. vigiles tests the real thing — hooks block, skills fire, subagents finish what they promised, a stray git push is caught before it happens. It drives a scripted stand-in for the model, not a live call, so it needs no key and runs on every commit. How testing works →

📊 Eval — the only way to put a real number on cost

"Caveman Mode cuts 65% of your tokens." Says who? vigiles A/Bs the claim on real coding tasks and hands you three numbers: the token bill, whether it hit its target, and whether your code still works.

caveman vs verbose · haiku · $0 on your subscription
  output tokens   762 → 842   (+11% — the "saving" reversed)
  correctness     1.0 → 1.0   (the fact survived)

Point it at any harness change that claims a number — does a compression skill pay for itself, is a subagent worth its cost, which model is cheapest here. promptfoo and DeepEval bill per token, every run; vigiles runs on your own Claude Pro/Max subscription, so you measure on every change, not once. A committed lock file (like package-lock) keeps CI honest without re-calling the model. (Claude Code today; Codex landing.) Measure a skill →

Quick start

1. See what's broken — read-only, no setup:

npx vigiles audit

2. Set it up when you like what you see. Paste into Claude Code or Codex:

Set up vigiles in this repo: run `npx vigiles init` and accept the defaults. If I
already have a CLAUDE.md or AGENTS.md, audit it and show me which references are
stale and which of my rules aren't enforced. Then write + run one harness test for
a hook or skill of mine. Don't run a real-model eval without asking me first.

Or run it yourself:

npx vigiles init   # sets up the typed spec for structural rules (non-destructive — eject reverses), adds CI,
                   # installs vigiles's skills + hooks as a Claude Code plugin (in
                   # ~/.claude/, not your repo). On Codex, skills install globally too.

Interactive in a terminal, non-interactive for agents/CI (or --yes). Works with Claude Code and Codex — vigiles verifies CLAUDE.md and AGENTS.md the same way. Codex setup →

Adoption is smooth: one command, then your agent does the rest. init installs the skills and hooks, so a plain-English ask does the work — no specs to hand-write, no hooks to wire:

• "test my skills" → scaffolds and runs a trigger/behaviour test, then commits its result so CI can check it (test-harness)
• "harden my rules" → upgrades prose guidance into enforced linter rules (strengthen)
• "add a rule to my CLAUDE.md or AGENTS.md" → edits the source and recompiles (edit-spec)

The hooks keep it honest in-loop — nudging the agent to tag a linter-rule mention so vigiles can verify it, or to re-run a test whose result just went stale — so there are no chores to remember.

• Both lint and test by default; scope with --lint / --test.
• Already have a CLAUDE.md / AGENTS.md, skills, or subagents? audit and lint read them as-is — nothing is moved or rewritten. For the structural rules that want a typed spec, init sets one up non-destructively (eject undoes it).
• Adds vigiles to devDependencies; installs the Claude Code plugin (skills + hooks) via the marketplace — globally, never vendored.
• Wires CI as a zernie/vigiles@v1 workflow (needs only read + PR-comment permissions) that posts a sticky PR comment + a valid output.

Targets Claude Code and Codex out of the box, or your own harness. Prefer to write tests yourself? JS or TS (*.harness.{mjs,ts}) — run with npx vigiles test.

FAQ

• Is this a framework I have to build around? No. It's a tool you run — like ESLint, Lighthouse, or npm audit. One command, a report, an optional CI gate. There's a library API for automation, but you never touch it to get value.
• Isn't this just a markdown linter? No — it checks whether your instruction file is true (every path/script/symbol/rule exists and is enabled), then tests and measures your harness. A style linter can't do any of that.
• Do I have to write TypeScript? No — plain markdown works with zero new files, and rules run in your own linter config. The typed spec is opt-in, only for the structural checks a linter can't do — like TS's strict (why?).
• Is it stable enough to adopt? The CLI you run is small and rarely changes; the library API still moves between releases. The high version number is release automation (a new major per breaking change), not age — see Stability.
• Non-JS repo? npx vigiles lint verifies your CLAUDE.md or AGENTS.md with no install (Ruff/Clippy/Pylint/… too).

Full FAQ →

Not for you if you want a model/capability benchmark or runtime guardrails in the request path — vigiles is build-/CI-time.

Docs

The docs index is the full map, grouped by what you're doing:

• Guides — verify instruction files · test your harness · measure a skill · ship a plugin · Codex & other harnesses
• Reference — CLI · rules matrix · testing API · full API
• Explanation — what it catches · how it compares · FAQ

Project — Stability · Related tools

License

MIT

[^name]: vigiles — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. Quis custodiet ipsos custodes? — "who watches the watchmen?" (Juvenal, Satire VI).

[^audit]: audit reads only by default. Two deeper checks — live MCP connections and skill-firing — are opt-in and ask before they run.