Coding Quality Loop

Make your AI coding agent ship changes you can trust, not giant diffs you have to babysit.

Quickstart · Orchestrator layer · The loop · Proof · Install · Compare · FAQ · Docs

AI coding agents are fast. Point one at a vague ticket and it can refactor things nobody asked for, pull in a dependency you did not want, claim "tests pass" without showing them, and sign off on its own work. You are left holding a big change you cannot tell is correct.

Coding Quality Loop makes the agent work like a careful engineer instead. It pins down what "done" means before writing code, changes as little as possible, proves the change with a test you can see, and has a separate agent review the work before it reaches you. What comes back is small, checked, and reversible: something you can read, trust, and merge in minutes.

It is a portable Agent Skill. The loop routes across two frontier hosts — Claude Code plans and implements, Codex reviews in a different model family — and it still installs into more (Droid, git, GitHub) as instruction and hook targets outside that routed loop. Cursor and Pi get advisory rules recipes in examples/, not a runtime install. Use it as a copy-paste prompt, a loadable skill, or a multi-agent config. No new tools, no lock-in.

Why the loop

Same agent, same model — a different process wrapped around it. Two honesty notes up front: the process asks for the table below, and our own published runs show the payoff is model-specific, not uniform — Claude cells measure positive code-quality lift, Codex cells measure flat-to-negative (−1.11, −9.0). Both numbers are on the dashboard, in red.

You ask your agent to "fix the checkout retry bug."

That is the whole idea: smaller changes, real proof, a second set of eyes, and memory that sticks. The work scales to the risk: a typo just gets fixed; a payment migration runs the full process. See a real worked example in examples/walkthrough/.

The state record in the walkthrough passes the same verify-gates check the loop enforces.

The Orchestrator Layer

New in v5.0.0 — and the headline of the release. The main session is the orchestrator: it thinks hard and makes every decision — task class, context map, contract, right-size rung, plan, model routing, the verdict on findings, and the stop-if-unsafe call. Workers (the implementer and the reviewer) never see the skill: they receive a brief, not context — goal, contract slice, files, commands, done-check, one screen max. No skill text, no references, no repository tour.

That inversion is a token diet. The always-loaded agent surface (SKILL.md) is now roughly half its former size, and workers load none of it. Decisions concentrate where the reasoning is; execution stays cheap and narrow. The rule that earns it: every gate must earn its tokens — that is the retention standard, and honestly, we have not yet run the live ablation that enforces it. The pre-registered protocol that will is bench/PROTOCOL.md; until it runs, deletions are still wins.

Routing is pinned to two hosts, two vendors: Claude Code plans and implements on frontier Anthropic models; Codex reviews on a frontier OpenAI model, always a different family than the implementer. This is a floor the config enforces (check-config), not a suggestion.

See SKILL.md for the canonical contract and references/agentic-orchestration.md for the orchestrator/worker topology.

Quickstart

One command. Auto-detects your host (Claude Code, Codex, or Droid), copies the skill, wires the hooks, and writes an install manifest so check and remove are exact.

npx coding-quality-loop init

Shipped on npm with signed provenance. Requires Node 18+ and Python 3. Zero runtime dependencies. Interactive by default; add --yes for CI, --dry-run to preview, --host <name> to skip detection. See packages/npm/ for the full CLI.

npx coding-quality-loop init --dry-run --yes   # preview only
npx coding-quality-loop add git                # add the pre-commit backstop
npx coding-quality-loop check                  # verify a prior install against its manifest
npx coding-quality-loop remove                 # uninstall from the manifest; restores your backups

Lighter or heavier paths — a no-install drop-in prompt, or a manual per-host copy — plus the demo transcript and the task-class table live in docs/quickstart.md, the single onboarding doc.

The loop, visualized

Three phases, smallest-safe-first, each closed by its own verification gate before the next may start. Context is a budget; verification is what terminates a phase. The right-size gate runs twice: once in PLAN to choose the smallest approach, once in REVIEW to confirm nothing crept in.

PLAN -> EXECUTE -> REVIEW

flowchart LR
  PLAN[PLAN] --> EXECUTE[EXECUTE]
  EXECUTE --> REVIEW[REVIEW]
  REVIEW -. durable lessons .-> PLAN

• PLAN — turn the goal into a task contract, map the change, write the validation contract, apply the right-size gate, and produce a plan. Terminates when the plan (and, for non-trivial work, the validation contract) exist and are checkable.
• EXECUTE — implement in small slices and verify. Terminates when the smallest sufficient checks pass with recorded evidence.
• REVIEW — independent review and ship/handoff. Terminates when a fresh-context reviewer has checked the diff against the validation contract and, for non-trivial work, a completion record exists.

Every older sub-step inherits one of these three phases; nothing is unlabeled. The helper script, config, and state record still use the original nine stable short machine names as sub-steps, so existing records, configs, and automation keep working unchanged:

The validation contract spans the INTAKE and PLAN sub-steps, both inside PLAN. Across tasks, durable lessons can persist in optional per-project memory. See SKILL.md for the canonical version of this mapping.

Anatomy of a shipped change

Each card above is a field in examples/walkthrough/agent-record.json — the same file verify-gates checks in the eval fixtures. See docs/architecture.md for the three-layer breakdown, and docs/quickstart.md for the three adoption paths.

Proof you can run

Every claim on this page is checkable on a clean checkout with no dependencies. Copy the block, paste in a terminal, watch the counts.

python3 -m py_compile scripts/*.py evals/*.py                                                # 1. byte-compile
python3 scripts/quality_loop.py check-config assets/quality-loop.config.example.json         # 2. config
python3 scripts/quality_loop.py eval-cases evals/cases --config assets/quality-loop.config.example.json  # 3. static
python3 evals/run_evals.py                                                                   # 4. behavioral gates
python3 evals/run_memory_evals.py                                                            # 5. memory gates
python3 evals/run_reality_evals.py                                                           # 6. reality gates (record↔diff)
python3 evals/run_hook_evals.py                                                              # 7. host hook fixtures
python3 evals/run_routing_evals.py                                                           # 8. model routing
python3 evals/run_control_evals.py                                                           # 9. control-plane add-on (in-repo checkout only; not in the npm tarball)
python3 bench/runner.py --mode fixture --seeds 1 --out /tmp/quality-loop-fixture-smoke.json  # 10. bench fixture smoke

Current result: 20/20 static + 63/63 behavioral + 32/32 memory + 51/51 reality + 30/30 routing + 55/55 hook = 251 gate cases across the six core suites, plus 37 add-on cases for the opt-in control plane (step 9 — the add-on ships in the repo checkout, not the npm tarball), re-run on every push by a dependency-free GitHub Actions workflow.

• The static suite is an intake-classification regression test. It pins the routing table: risk tier, task class, required gates. It does not prove a gate fires on real prose.
• The behavioral suite is where the record gates actually fire. It drives the real CLI against constructed records and asserts hard-to-fake behavior: a self-downgraded boundary task is blocked, placeholder/wrong-content artifacts are rejected, the implementer cannot be the reviewer, and untracked secrets are flagged. One case is a docs-presence lint, not a gate.
• The memory suite drives memory-recall / commit / prune against constructed stores and asserts anti-bloat and safety invariants: the index stays <=40 lines even with multi-line lessons, recall respects budget, and secrets are redacted before they land.
• The reality suite builds temp git repos where the record and the diff disagree: phantom completion, unmapped file, auth path under low tier, missing bugfix test, stale review hash, lying evidence, red-green catch, staged secret. It asserts the diff-grounded gates catch the lie.
• The hook suite feeds fixture JSON into Claude/Codex-compatible shims and checks destructive Bash blocks, secret-write blocks, required edit-before-plan blocks, Stop-gate continuation, SessionStart context, and installer idempotence.

The published eval runs

Five eval directories are committed with source, numbers, and caveats. Read them honestly: every cell is n=1 (single seed), judges are same-family LLMs, and only the webapp run is unambiguously real-CLI — the others are model-proxy or have no recorded invocation method. We publish the negatives (ts-search Codex −9.0, webapp Codex −1.11) next to the positives on purpose; the omitted negative was the most credible asset we were hiding. One caveat rises above a footnote: the webapp judge-lift numbers (incl. the −1.11) are void per bench/PROTOCOL.md §4 — judge-family independence was violated — so treat that run's objective browser checks as its evidence, not its judge scores (see the row note below).

All directional, not durable benchmark claims — and note the sign flips: the lift is model-specific (Claude positive, Codex flat-to-negative), which is exactly why the live ablation is pre-registered before it runs. One correction, stated where the numbers live: the Webapp 07-07 judge-lift figures (+6.67 / −1.11 excl-D7; +16.0 / +7.5 with D7) are void under bench/PROTOCOL.md §4 — the two judges were independent of each other but not of the arms' own implementer model families, so they scored their own house style. Only the judge-scored lift is void; the run's objective browser/behavior checks stand. The webapp run also caught an agent silently softening its local copy of the gate script and reporting PASS against it — see the panel below. The benchmark harness, fixture-smoke result, and the pre-registered protocol live in bench/PROTOCOL.md.

The gate-gaming story

During the Webapp 07-07 run, one agent silently softened its workspace copy of diff-audit and reported Overall: PASS under its own weakened gate. The pristine gates, re-run against the same record, failed it immediately — that contrast is the product in one incident. The fix (CHANGELOG.md §3.1.0): verify prints a sha256 for each helper module, the SKILL.md rule never repair or stub the helper; report and stop, and reality-eval cases pinning it. A repeated failure becomes a durable rule, test, and hook — not a chat correction.

Run it yourself on your task

First, see a record that already passes, end to end, in about ten seconds (exit 0):

python3 scripts/quality_loop.py verify-gates examples/walkthrough/agent-record.json
# verification gates look sufficient for recorded risk tier
# (exit 0 — this is what a complete, honest record looks like)

Then start your own. For medium/high-risk work, create a state record and run the primary verification command — an umbrella over record-shape gates, diff-grounded reality checks, evidence re-execution, and AC-to-command coverage:

python3 scripts/quality_loop.py init-record --goal "Fix invoice total rounding" --risk-tier medium --output .quality-loop/agent-record.json
python3 scripts/quality_loop.py verify .quality-loop/agent-record.json --red-green   # --base defaults to the merge-base with origin/main
# Overall: FAIL
# (expected — a freshly-initialized record has no evidence or review yet; verify lists exactly what to fill in)

Ceremony scales with risk

A tiny task must not be forced through mission ceremony. A medium task must not ship without a contract and an independent review. Risk trumps size: any risk-boundary change is medium+ regardless of diff size. The task-class table (tiny → small → medium → mission) lives in docs/quickstart.md; the canonical definitions are in SKILL.md.

What the loop costs

Process is not free, and as of v6.3.0 the bill is measured, not estimated. The first live pre-registered run (bench/results/micro-bugfix-live-2026-07-21.json: {baseline, full} × claude-code × 3 seeds on a one-file micro-bugfix) put the full lifecycle at a median 8.1× output tokens, 6.3× input tokens, 7.65× cost, and ~10× wall time versus baseline — with identical objective quality (every cell in both arms: committed test green, 3/3 hidden cases, ≤28-line diff, zero new dependencies). On this micro-task the full loop bought no measured quality for 6–8× the cost — the observed cell, not a universal claim; it is why ceremony scales with risk and a tiny task must never pay for mission ceremony. The pre-registered §6.2 threshold (>1.5×) was exceeded ~5-fold — but the pre-registered outcome is not claimed: §6.0's letter forbids any §6 outcome when every arm passes the objective battery, and every arm did. The results file records that verbatim; the ladder/class-text cut candidate is opened as an ordinary operator decision informed by the data (ROADMAP), and §6.0's scope is amended for future runs only (PROTOCOL Amendments). The earlier live webapp run measured 3–6× wall time on a larger task. The retention standard — every gate must earn its tokens — now has its first enforcement datapoint; the full ablation that applies it to every component is still pre-registered, not yet run (policy, protocol). Live sweeps must record per-arm cost, and python3 bench/runner.py --validate fails any that don't.

What it enforces and what it does not

Most tools hide this table. Ours is one of the best marketing artifacts in the repo. The differentiator is that the gates are executable, not advisory, and the boundaries are explicit. scripts/quality_loop.py is a portable, stdlib-only checker. It complements CI, tests, scanners, and human review. It does not replace them.

The runtime entry points are verify (the umbrella: record gates + diff audit + evidence re-execution + AC coverage), verify-gates, verify-gates --against-diff, check-record, diff-audit, run-evidence, attest-review, render-prompt, and scan-text --stdin, pinned by evals. On Claude Code, the Stop hook runs the full verify umbrella (evidence re-execution included) at terminal statuses — fabricated pass rows no longer clear the local gate. The honest caveat: truthful-but-vacuous rows still clear it — an allowlisted no-op (e.g. true) named as the proving command for every acceptance criterion re-executes and passes, and >=3 criteria sharing one proving command is warn-only, even at medium+. The gate checks that evidence exists and re-runs, not that it is substantive.

Reviewer heterogeneity. check-config now hard-fails when the implementer and fresh_reviewer resolve to the same model on medium+ tasks. Tool-using evaluator. The reviewer must execute tests and benchmarks when available, not just read the diff; the verdict records ran_checks: true|false. Communication-bridge rule. After the reviewer produces findings, the implementer filters them against the contract: in-scope findings become fix tasks, out-of-scope findings become follow-ups. This prevents review loops.

Install & use matrix

Pick your host. Full copy-paste files live in examples/; every path below is real.

Cursor and Pi are no longer offered by the npm installer's host picker: they load the instructions but none of the hook runtime, so they get honest advisory-only labels instead of surface parity.

Provenance note: the npx installer ships from the coding-quality-loop npm package (source: packages/npm/) with signed Sigstore provenance tying the tarball to a specific GitHub Actions build. It is a thin UX wrapper around scripts/install.py, so both paths land the exact same files. This repo is not yet on the agentskills.io Skills Hub; gh skill install works once a maintainer publishes a release. See Release & pinning.

The matrix rows above are the full per-host install + invoke commands; copy-paste host files live in examples/.

Install host integrations into a project:

python3 scripts/install.py --host all

This copies the stdlib Quality Loop runtime scripts, host hook shims, .claude/settings.json, .codex/hooks.json, read-only reviewer agents, pre-commit config, and an example GitHub workflow with backups. Claude Code and Codex hooks remain advisory unless the host trusts/enables them; git hooks and CI are the portable backstop.

Release 1.6 adds first-class host wiring without making any host mandatory:

• hosts/claude-code/settings.json wires SessionStart, PreToolUse, and Stop command hooks. Shims are stdlib Python and delegate to the core CLI.
• .claude/agents/quality-loop-reviewer.md and .claude/agents/quality-loop-security-reviewer.md are read-only reviewer subagents sourced from references/reviewer-checklists.md.
• hosts/codex/hooks.json uses Codex's current project hook schema. Codex still requires hook trust review in /hooks.
• hosts/git/install-git-hooks.py and hosts/git/.pre-commit-config.yaml provide the universal git backstop: staged diff-audit blocks secrets/test weakening.
• action.yml and hosts/github/quality-loop-example.yml provide CI wiring.
• scripts/install.py installs host wiring idempotently and prints what is enforced vs advisory.

Config-based model routing — the model_routing section in quality-loop.config.json maps each model class to a real model per host. python3 scripts/quality_loop.py setup-models --host <host> applies it: it rewrites model: frontmatter for Claude Code (.claude/agents/*.md) and Droid (.factory/droids/*.md), or prints the Codex config.toml / Pi /model settings to apply. Multi-host topologies are first-class: an agents entry may pin a role to another harness ({"host": "codex", "class": "strong_reasoning"}), main_session declares where the implementer runs, and one setup-models run applies every host — print hosts (codex, pi) are labeled PRINT-ONLY — settings not applied or verified by CQL. Reviewer independence is enforced on the resolved model family across hosts (family field or well-known prefix; unknown ids skip; allow_same_family is the explicit escape hatch). Three pre-validated variants along the intelligence↔cost dial ship in assets/routing/. brief shows the active routing per host and flags drift on file hosts. Agent files ship with model: inherit so they are host-neutral at rest. Route reasoning effort at high: check-config rejects xhigh/max unless a model-class block sets "allow_overthink": true, because effort is per-step, not per-task endurance — above high, models overthink and overspend each step. See the model capability glossary (intelligence / taste / cost, the effort ceiling, and the escalation policy) and config-driven model setup.

One user's personal cross-CLI setup (an external harness that overrides CQL routing) is documented — labeled as exactly that — in docs/cross-cli-recipe.md.

A repo can opt into required edit-before-plan blocking in the canonical root config, quality-loop.config.json:

{"enforcement": "required"}

(.quality-loop/config.json still works as a fallback for one release, with a deprecation warning.)

Three adoption levels

• No install — paste the Minimal Drop-In Prompt or a host rule file. Zero scripts, zero config. Best for trying it on one task.
• Install — copy the skill folder so the agent pulls references/, assets/, and the state-record schema on demand via progressive disclosure. Best for repeated use.
• Orchestrated — adopt assets/quality-loop.config.example.json and route each step to a role-based agent profile. Best for multi-agent or production setups.

What's in the box

A single Agent Skill package following the open Agent Skills specification: SKILL.md at the root plus optional sibling folders. Progressive disclosure is the core mechanism: the agent always sees the frontmatter name/description, loads the full SKILL.md when relevant, and pulls references/assets/scripts only when a step needs them.

coding-quality-loop/
├── SKILL.md            # the skill: when-to-use, lifecycle, task classes, roles, gates
├── assets/             # templates + schemas loaded on demand (contract, plan,
│                       #   completion record, progress, context map, record schema,
│                       #   config, per-role prompt cards)
├── references/         # deep-dive docs pulled only when needed (lifecycle, orchestration,
│                       #   reviewer checklists, tool contracts, philosophy,
│                       #   the memory contract, enforcement matrix)
├── examples/           # host-native copy-paste: claude-code, codex, droid,