claim-check

Checks whether the test count in your commit message is actually true. pytest, vitest and jest.

If a commit message says "22 passed" or "all tests pass", claim-check runs your suite and compares the numbers. If they don't match, the commit is blocked. If the message doesn't mention tests at all, nothing happens.

As a Claude Code plugin it also checks claims made in conversation, by comparing them against test runs the agent actually made. That part reports and never blocks.

Works as a pre-commit hook, a Claude Code plugin, or a plain CLI. No runtime dependencies.

Install

Pick whichever fits how you work. They all use the same checking logic.

As a Claude Code plugin

claude plugin marketplace add bhumik154/claim-check
claude plugin install claim-check@claim-check

Then restart claude. Hooks only load when a session starts, so until you restart it's installed but doing nothing, with no warning either way.

You don't need pip install for this one. The plugin ships its own source and runs it with any Python 3.9+ it can find. If it can't find one, the hook exits quietly and your tool calls carry on as normal. See PLUGIN.md for the details.

As a pre-commit hook

repos:
  - repo: https://github.com/bhumik154/claim-check
    rev: v0.5.0
    hooks:
      - id: claim-check

Two things that will otherwise bite you:

The commit-msg stage isn't part of pre-commit's default install. Run pre-commit install --hook-type commit-msg, or put default_install_hook_types: [pre-commit, commit-msg] in your config. Without that the hook installs and then silently never runs.

The hook uses language: system rather than language: python, on purpose. A python hook runs inside pre-commit's own isolated virtualenv, which has none of your test dependencies and often not even pytest. That doesn't error; it just prints "could not verify... allowing commit" forever, because a runner crash fails open by design. So claim-check and your test dependencies need to be installed in the same environment you actually run git commit from.

As a CLI

Not on PyPI yet, so install from the repo:

pip install git+https://github.com/bhumik154/[email protected]

claim-check verify-tests path/to/COMMIT_EDITMSG

# or check a message directly
claim-check verify-tests "22 passed"

# reuse output you already have instead of running the suite again
claim-check verify-tests path/to/COMMIT_EDITMSG --pytest-output pytest_output.txt

# claim-check's own flags can go before or after the message.
# anything else gets passed through to pytest.
claim-check verify-tests "22 passed" --cwd backend/ -k "not slow"

Wiring the Claude Code hook by hand

If you'd rather not install a plugin, put this in .claude/settings.json (also in examples/claude-code-settings-snippet.json):

{
  "hooks": {
    "PreToolUse": [
      { "matcher": "Bash", "hooks": [{ "type": "command", "command": "claim-check-claude-hook", "if": "Bash(git commit *)" }] }
    ]
  }
}

This needs claim-check on your PATH, so it does require installing the package.

Scope: what it can and cannot know

claim-check compares your claim against whatever your runner collected when it ran. A claim is about a scope; the tool sees one invocation. Those can differ.

The awkward part is that they are indistinguishable from the numbers alone. "You wrote the wrong number" and "I ran a narrower scope" produce byte-identical output: a claim of 150 against a run collecting 40 looks the same either way. So scope cannot be recovered by arithmetic, and the tool does not try.

Instead it looks at the invocation it actually made. When the run was narrowed, by a -k or -m filter, an explicit path, -x, --lf, --shard, a vitest -t, a jest --onlyChanged, it says so and declines to judge rather than guessing:

claim-check: WARNING - could not verify "All 22 tests pass": this run saw 1
where the claim is about 22, and this run was narrowed by 'sub/', so it
covered less than the whole suite

That cuts both ways, and the second direction matters more than it looks. A narrowed run cannot confirm a whole-suite claim either. pytest -k test_a on a file whose test_b fails reports 1 passed, 1 deselected, so "all tests pass" is perfectly true of what ran while the suite is red. Confirming there would launder a false claim into a verified one, so it abstains.

It also reads deselected straight out of pytest's own output, which catches filtering configured in pytest.ini rather than passed on the command line. That signal is pytest-only; vitest and jest never report it.

None of this weakens the ordinary case. A run that was not narrowed is judged exactly as strictly as before, so "all 22 tests pass" against a real 21 is still caught.

What it still cannot see

• Directory-level narrowing. A --cwd into one package of a monorepo covers less than the repo, but a hook legitimately passes a cwd on every run, so treating that as narrowing would abstain always. Undetected on purpose.
• testpaths and --ignore in your config, when nothing is deselected. Those files are never collected, so there is no signal in the output at all.
• A bare "all tests pass" with no number, combined with either of the above.

If your setup hits these, run claim-check against the same scope your claim is about, and check its cwd and your project's own test-discovery config.

Configuration

When claim-check's environment isn't your project's

By default claim-check runs <the Python it's installed under> -m pytest. If claim-check lives somewhere else, say a pipx-style global install, or a Claude Code hook running under a different interpreter than your poetry or hatch project, that command can land on a Python with no pytest, or a pytest without your dependencies.

When that happens it fails open silently: "could not verify... allowing commit", on every commit, looking exactly like a tool that isn't installed.

Every entry point takes --command for this. Point it at whatever actually runs your tests:

claim-check verify-tests COMMIT_EDITMSG --command "poetry run pytest"
claim-check-precommit COMMIT_EDITMSG --command "hatch run test"
claim-check-claude-hook --command "poetry run pytest"

Not sure if this affects you? Run claim-check verify-tests "22 passed" from the same shell you commit from. If it prints WARNING - could not verify (...) mentioning a missing module or command, that's this, and --command fixes it.

Timeouts

Every pytest run is killed after 120 seconds by default and fails open with a reason. That's mostly for the commit-msg hook, which runs synchronously on every commit including every one replayed during an interactive rebase. A hung suite there doesn't block one commit, it blocks the whole rebase with no visible cause.

120 seconds is deliberately generous. A large but perfectly normal integration suite can take minutes, and a timeout set too short just means the tool quietly verifies nothing. Lower it if your suite is fast and you'd rather fail loudly:

claim-check-precommit COMMIT_EDITMSG --timeout 30

--timeout has to be greater than zero. Zero and negative values are rejected outright, which is a deliberate exception to the fail-open rule everywhere else. A non-positive timeout kills every run the instant it starts, then fails open, so you get "could not verify... allowing commit" forever while everything looks fine. Verifying nothing silently is worse than a loud error about a typo.

What it's tested against

tests/ has 370 cases covering the claim parser, the pytest and JavaScript output parsers, the shell-command parser, the comparison policy, the subprocess runner, the evidence store, all five entry points, and the plugin manifest.

One example, from the comparison core:

def test_all_tests_pass_claim_with_zero_collected_tests_is_flagged_not_silently_matched():
    # Vacuously true if you only check failed == 0; almost certainly not
    # what was meant by "all tests pass".
    verdict = compare_claims([_all_pass()], _counts(passed=0, failed=0))
    assert verdict.status == "mismatch"

The cases worth knowing about:

What this is not

It checks truthfulness, not whether your tests pass. If you write "14/15 passing" and the suite really is 14 passed and 1 failed, that's accurate, so the commit goes through with a test failing. Most people assume a hook that runs pytest blocks red builds; this one only blocks claims that don't match reality. If you want red builds blocked, use a pre-push hook or CI alongside this, not instead of it.

It still doesn't know what "the full suite" means to you. It knows when its own run was narrowed and abstains there, but directory-level narrowing and testpaths restrictions leave no trace in the output, so those go unnoticed. See the scope section above.

It isn't NLP. It can't resolve tense or discourse. "Was at 15 passed, now 22/22" is genuinely ambiguous to a regex. Only one claim per message gets checked: the most specific kind, then the last one among ties. That covers a stale count corrected later, but a later sentence describing an earlier state can still slip through.

pytest only, for now. vitest, jest and cargo test are a real gap, documented rather than silently unsupported.

The Claude Code hook only sees commits made through the Bash tool, since that's what carries a parseable command string. A commit through some other tool goes unchecked. In practice this hasn't been a gap, because the heredoc pattern for multi-line commit messages is bash syntax anyway.

Only git commit in the form git [global options] commit. A leading wrapper like sudo git commit or env FOO=1 git commit isn't recognised and goes unchecked.

The pre-commit hook is client-side and --no-verify skips it. Pair it with the CLI in CI if you want actual enforcement rather than a local nudge.

A test that forges a whole pytest session can still fool it. Output printed by a test gets replayed in the failure report, which always comes before that session's real summary, so the last-line-per-session rule handles the realistic case. The gap that remains depends on ordering. A forged summary line printed before a forged test session starts header is believed, because the forged header closes the segment right after the forged line and its count gets added to the real one. The reverse order is safe: both summaries land in the same segment and the real one still comes last. Both orderings are pinned by tests. Anything that can make a test print output in that exact order can already edit the test suite directly.

The end-of-turn check reports, it does not block. It also only speaks when it has fresh, whole-suite, same-session evidence from a run the agent itself made, and stays silent otherwise. Roughly half of claims have no such evidence, so about half go unchecked. That is deliberate: unknown is never reported as wrong.

An honest claim that mentions an error still gets flagged. "22 passed, 1 error" against a real 22-passed-1-error run is reported as a mismatch, because an error means some test never ran and the true denominator is unknown. That strictness is what makes a bare count mean anything.

Where this came from

The same habit used building recurring-free-slots and spiral-galaxy-ic: check a claim computationally before shipping it, rather than trusting it because it sounds right.

Building those two, the test count drifted constantly as fixes landed. More than once a stated number was simply wrong, a "22 passed" that was really 21, or a golden reference value computed by hand that turned out to be off, and it stayed wrong until someone re-ran the check. That happened to careful, deliberate work. It happens far more easily in a commit message written at the end of a long change, by a person or by an AI agent, with nothing actually re-running the suite.

License

MIT