falsegreen-skill

LLM-based semantic analysis for false-positive test detection. Companion to falsegreen, the Python static scanner.

For Python, this skill applies the complete falsegreen catalog directly - all structural and semantic patterns - via LLM analysis, without requiring the static scanner to run first. For TypeScript, JavaScript, and Robot Framework it is the primary detection tool. It is a superset of the three static scanners (falsegreen, falsegreen-js, robotframework-falsegreen) plus semantic patterns only an LLM can detect.

The falsegreen family (install the one for your stack):

Tool	Stack	Install	Package
falsegreen	Python / pytest	`pip install falsegreen`	PyPI
falsegreen-js	JS / TS	`npm i -D falsegreen-js` (`npx falsegreen-js`)	npm
robotframework-falsegreen	Robot Framework	`pip install robotframework-falsegreen`	PyPI
falsegreen-skill	semantic LLM pass	`npx falsegreen-skill analyze <path>`	npm

New here? Start here

This is an LLM skill that reads your tests and flags the false-green ones: tests that stay green even when the code they cover is wrong. It catches the semantic cases the static scanners cannot, because it reads the test as text and works out what the test was meant to prove.

A test like this passes forever, no matter what the code does:

# before - false-green: asserts the mock back to itself
def test_discount(mock_rate):
    mock_rate.return_value = 0.1
    result = apply_discount(100, mock_rate)
    assert result == mock_rate.return_value   # passes for ANY result, even a wrong one

The fix is an independent expected value, one the code did not produce:

# after - the test can now fail when apply_discount is wrong
def test_discount():
    result = apply_discount(100, rate=0.1)
    assert result == 90                       # 10% off 100, computed by hand from the spec

The skill reads the first version and reports it as a J2 finding (the expected value is borrowed from the code, not from an independent source) with the line, the reason, and a fix hint. Three steps to try it:

Install or enable it for your host - see Installation below.
Point it at a test file - npx falsegreen-skill analyze tests/test_discount.py, or, in an editor host, just ask it to "analyze this test for false-positive smells".
Read the finding - each one names the catalog code, the failed judgment (J1-J6), why the test cannot fail, and how to fix it. See Quick example.

For a step-by-step walkthrough of all three modes with runnable examples and flow diagrams, read the user guide. The full catalog, the judgments, and the per-language reference live in the docs site and in reference.md. For a visual architecture overview, host and language routing plus the Mode A/B/C flow diagrams, see docs/architecture.md.

Why this exists

A test suite with 100% green tests is not a proof of correctness. It is a proof that no test failed - which is a different thing. Tests can pass permanently not because the code is right, but because the test never checks anything meaningful.

Static analysis tools catch some of these cases. Linters like ruff or flake8-pytest-style catch syntax-level patterns: a bare assert True, a missing assert call, an unreachable block. Mutation testing tools like mutmut probe whether tests actually fail when the code changes. Both approaches have limits: linters cannot reason about test intent, and mutation testing requires the code to run.

This skill fills the gap between linters and mutation testing. It reads the test as text, reconstructs the intent, and asks six structural questions about whether the test can actually fail. The questions are derived from the taxonomy of false-positive test patterns collected in CREDITS.md.

The core insight: a test is useful if and only if there exists some incorrect implementation that would cause it to fail. If no such implementation exists, because the assertion is unreachable, tautological, or verifies the mock instead of the code, the test is structurally green regardless of whether the production code is correct.

The methodology

One rule underlies every judgment: a test is useful only if it can fail when the code breaks.

The six-judgment framework (J1-J6) makes this rule concrete:

#	Question	Catches
J1	Does the assertion run?	Dead assertions, vacuous loops, swallowed failures
J2	Is the expected value from an independent oracle?	Echo mocks, formula re-implementation, spec contradictions
J3	Is the real unit under test, not a mock of it?	Mock-the-SUT, self-confirming literals
J4	Does the assertion verify enough?	Truthiness-only, len > 0, repr coupling, broad raises
J5	Is the test coupled to implementation internals?	Positional mock args, private method testing
J6	Does the test pass in isolation, without ordering?	Shared mutable state, test-order dependency

A test is flagged HIGH only when the first failed judgment has no plausible legitimate interpretation. A test is flagged LOW when the smell is likely but has plausible intent. Everything else is PASS.

Precision over recall. One wrong flag on a legitimate test costs more goodwill than a missed smell. Exemptions are explicit:

Semantic case 18 requires a cited independent oracle (spec, docstring, API contract). Without a citation, do not report case 18.
Characterization tests - intentionally freezing current behavior - are not false positives.
Boolean predicates (isinstance, .exists(), .is_dir()) are not weak assertions.
In HTTP/UI layer tests, a truthiness check on a response object means "the request succeeded" and is meaningful.

Full protocol: SKILL.md.

What it detects

Python - structural patterns (complete falsegreen catalog)

Family A - The test never checks anything

Code	Pattern	Example
C1	Assert inside `if`/`for` that may not run	`if items: assert items[0].valid` when `items` can be `[]`
C2	No assertion at all	test body contains only setup calls
C2b	Calls SUT but discards result	`result = process(x)`, result never asserted
C3	Assert inside `try` whose `except` swallows it	`except Exception: pass` catches `AssertionError`
C4	Test function nested inside another function	pytest does not collect inner defs
C4b	Test class with `__init__`	pytest skips classes that have `__init__`
C20	Assertion after unconditional `return`/`raise`	dead code, never runs
C21	Every assert is conditional, none runs unconditionally	all asserts inside `if/else` branches
CC	Commented-out assertion	`# assert result == 42`

Family B - The check is weak or always true

Code	Pattern	Example
C5	Always-true check	`assert True`, `assert (a, b)` (non-empty tuple)
C6	Truthiness / `len > 0` / substring in `str()`	`assert result`, `assert len(x) > 0`
C6b	Positional mock arg via computed index	`call_args.args[expected_args.index("target")]`
C7	Self-comparison	`assert name == name`
C8	Exact float equality	`assert ratio == 3.14159`
C9	`pytest.raises` too broad or no `match=`	`with pytest.raises(Exception)`
C11a	Self-confirming literal	`product.price = 100; assert product.price == 100`
C13	Mock assertion uncalled or misspelled	`mock.assert_called_once` (no parens)
C13b	`@patch` without `autospec=True`	typos in kwargs pass silently
C14	Golden file written from actual output	first run records any output as truth
C16	Depends on wall clock, random, or `sleep`	`datetime.now()` unfrozen, `time.sleep()`
C18	`str()`/`repr()` comparison	`assert str(user) == "User(Alice, 30)"`
C25	`@pytest.mark.xfail` without `strict=True`	XPASS silently accepted
C34	Suboptimal assertion form	`== True`, `== None`, `not x in y`, `len == 0`

Family C - The test checks its own setup

Code	Pattern	Example
C19	`pytest.raises` wraps multiple calls	setup call inside raises block may be the one that raises
C28	`pytest.raises` binding variable never read	`as exc:` but `exc` never asserted
C29	`os.environ` mutated directly	`os.environ["KEY"] = "x"` without `monkeypatch`

Family D - Green depends on outside factors

Code	Pattern	Example
C17	`pytest.skip()` inside broad `except`	assertion failure silently becomes a skip
C23	Hard-coded absolute or home-relative path	`/home/user/data.csv`
C24	Module-level mutable state shared between tests	`_cache = {}` at module scope
C27	`try/except/pass` instead of `pytest.raises`	both raise and no-raise leave test green
C30	`responses.add()` without activating interceptor	real HTTP goes through
C31	`capsys.readouterr()` result discarded	captured output never asserted
C32	`@pytest.mark.skip` without `reason=`	forgotten skip
C35	`@pytest.mark.flaky` / retry decorator	masks non-determinism

Family E - The test checks the wrong thing

Code	Pattern	Example
C33	sklearn/ML metric computed but not asserted	`accuracy_score(y, y_hat)` result discarded
C36	`pytest.fail()` without reason	CI shows only `FAILED`, no context
C37	Duplicate case in `@pytest.mark.parametrize`	same `(a, b, expected)` tuple appears twice

Semantic patterns (all three languages)

Semantic patterns require LLM judgment - no static rule can detect them.

Case	Pattern
10	Patches the unit under test (not a dependency)
11	Asserts the value fed to the mock (echo)
12	Re-implements the production formula as the expected value
15	Passes only when another test has already run
18	Expected value contradicts the spec (freezes a bug as correct)

The tables above are the most common Python and semantic patterns, not the whole catalog. The full set is 125 codes: the Python C* series (up to C59), the TS/JS JS* series (up to JS31), the Robot R* series (up to R8b), the project-layer PL* and diagnostic D*/M* codes, and the semantic S1-S21 series. Each carries a bad-pattern example and a clean look-alike in reference.md, the canonical catalog.

Diagnostic and coupling codes (opt-in)

These codes do not create false positives, but they reduce observability and make failures harder to diagnose. They are OFF by default and can be enabled per code in .falsegreen.toml:

[tool.falsegreen]
severity = { D1 = "info", D3 = "info", D4 = "info", D5 = "info", D6 = "info", M2 = "info" }

Code	Pattern	Why it matters
D1	Assertion Roulette: 2+ asserts without messages	CI output says only the line number, hard to triage
D3	Duplicate Assert: exact same assertion written twice	second assertion adds nothing
D4	Unnamed Parametrize: 3+ cases, no `ids=`	CI shows `test[0]`, `test[1]`, unreadable failure reports
D5	Inline Setup Excess: 5+ setup statements before first assert	test should be split or setup moved to a fixture
D6	Debug Print: `print()` or `pprint()` in test body	suppressed by default, often a forgotten debug statement
M2	Long Test Method: test body over 50 lines	trying to verify too many concerns at once

What we don't flag (and why)

Measured against the Open Catalog of Test Smells (517 documented smells), only the false-green slice is in scope. The skill is the broadest of the family - it reads intent - but it is still false-green only. These stay out, on purpose:

Brittleness / false-red (a test that breaks without a real bug): sensitive equality, brittle or fragile assertions. The opposite axis.
Hygiene / maintainability: assertion roulette, magic numbers, long tests. Linter territory (ruff/ESLint/Robocop); a few are surfaced here as opt-in diagnostics.
Slow, design, naming, duplication, runtime/culture: none are about whether the test protects.

The skill carries every structural proxy of the scanners (C16 for uncontrolled time, C23 for hard-coded paths, C24/C15 for shared state) plus the semantic patterns no parser sees: negative-only security assertions (S11), patching the unit under test (S12), and order-dependence across files (S13). See CREDITS.md for the full cross-walk against the literature.

How it compares

vs. ruff / flake8-pytest-style

Ruff and flake8-pytest-style catch syntax-level patterns: assert True, pytest.raises with no type, magic values in assertions. They are fast and precise for the patterns they cover - about 8-10 of the 37+ cases in the falsegreen catalog.

This skill covers all 37+ structural codes and the 5 semantic cases that require reading the test as a whole - echo mocks, formula re-implementation, spec contradictions. The two tools are complementary: run the linter for instant feedback on simple cases, run the skill for semantic judgment on the rest.

vs. PyNose / pytest-smell

PyNose and pytest-smell are the closest research counterparts. Both apply the classic Palomba 2018 test-smell taxonomy (Assertion Roulette, Duplicate Assert, General Fixture, etc.). The falsegreen taxonomy is narrower: it focuses only on patterns that create false-positive green tests, not on maintainability smells in general.

Where there is overlap (Assertion Roulette = D1, Duplicate Assert = D3), falsegreen flags them as diagnostic codes - informational, not blocking. The structural codes unique to falsegreen (C1-C45, C48) cover patterns that Palomba's taxonomy does not address because they were derived specifically from studying how green tests hide broken code in CI.

vs. mutmut / cosmic-ray

Mutation testing answers the question definitively: change the code, does the test fail? That is the ground truth. Mutmut and cosmic-ray are accurate for the programs they can run, but they require an executable environment, a full test suite, and minutes to hours per run.

This skill is a static pre-flight check. It cannot prove that a test fails when the code changes - that is mutation testing's job. It can identify, in seconds, tests that are structurally unable to fail: assertions that never execute, checks that are always true by construction, mocks that intercept the function being tested. Think of the skill as a fast filter before the mutation testing pass.

How to use

Installation

Platform	How
Claude Code	`/plugin marketplace add vinicq/falsegreen-skill` then `/plugin install falsegreen-skill@falsegreen`
Claude.ai / Anthropic API Skills	`npm run build:targets`, then package `dist/claude-agent-skill/` as the standalone skill
OpenAI Codex CLI	No single-command install: clone the repo and run `codex` for the full protocol (scoped to the clone), or copy `AGENTS.md` into your own project / `~/.codex/AGENTS.md` for compact review (see `contexts/codex.md`)
Antigravity CLI (`agy`)	Install as a plugin: `agy plugin install https://github.com/vinicq/falsegreen-skill` (subpath `.antigravity-plugin/`), or open the repo and `agy` discovers the workspace skill at `.agents/skills/falsegreen-skill/SKILL.md`. From Gemini CLI: `agy plugin import gemini`
Gemini Agent Skill	workspace skill at `.agents/skills/falsegreen-skill/SKILL.md`, or `npm run build:targets` for `dist/gemini-skill/`
Cursor	Copy contents of `contexts/cursor.md` to `.cursor/rules/falsegreen-skill.mdc`
CLI	`npx falsegreen-skill analyze tests/test_example.py`, see docs/cli.md
API	Use the defined provider guides in `contexts/claude.md`, `contexts/codex.md`, and `contexts/gemini.md`

Skill or static scanner?

Use the static scanner (falsegreen for Python, falsegreen-js, robotframework-falsegreen) for fast, deterministic checks in CI and pre-commit: it proves what a parser can see and never needs an API key. Use this skill for the semantic cases a parser cannot reach - a mock standing in for the unit under test, an expected value copied from the code, a value that contradicts the spec. The skill is a superset: it carries every structural code of the three scanners plus the AI-only semantic patterns. A common setup runs the scanner on every commit and the skill on the files the scanner cannot fully judge.

Quick example

Given this test that echoes the mock back to itself:

# tests/test_tax.py
def test_calculate_tax(mock_calc):
    mock_calc.return_value = 0.15
    result = calculate_tax(100, mock_calc)
    assert result == mock_calc.return_value  # J2: asserting the mock, not behavior

export ANTHROPIC_API_KEY=sk-ant-...
npx falsegreen-skill analyze tests/test_tax.py

Output:

CASE 11 (J2) - HIGH - Python - spec

Test: test_calculate_tax (line 3-6)
Finding: The assertion checks mock_calc.return_value - the same value the
mock was configured to return. This passes for any return value, including
wrong ones.
Evidence:
  mock_calc.return_value = 0.15
  assert result == mock_calc.return_value
Fix hint: Assert against an independently computed expected value, e.g.
assert result == 15.0 for a 15% tax on 100.

SUMMARY
Tests reviewed: 1
Findings: 1 (1 high, 0 low)
Clean: 0

Try it on your test suite

Point the CLI at any test file or directory:

# single file
npx falsegreen-skill analyze tests/test_orders.py

# multiple files
npx falsegreen-skill analyze tests/test_orders.py tests/test_payments.py

# JSON report for CI, exits 2 if any HIGH finding is present
npx falsegreen-skill analyze tests/test_orders.py --json --fail-on-high

# deep analysis with a stronger model
npx falsegreen-skill analyze tests/test_orders.py --model claude-opus-4-8

# lower temperature for more deterministic output (default is already 0.2)
npx falsegreen-skill analyze tests/test_orders.py --temperature 0.0

The skill identifies the language from the file extension. TypeScript and JavaScript work the same way - no extra flags needed.

Full flag reference: docs/cli.md.

Propose a fix and prove it (AI-fix mode, V1)

analyze finds a false-green; fix proposes a stronger test and proves it before you trust it. It is opt-in, Python/pytest only, and propose-only: it prints a test-file patch but never applies it and never edits your production code.

# propose a patch for a C2b finding and run the gate against the real SUT
npx falsegreen-skill fix tests/test_discount.py --case C2b --line 14 --sut src/discount.py

# parse + preserve only, no mutation gate (no runnable SUT or a quick pass)
npx falsegreen-skill fix tests/test_discount.py --case C5 --line 9 --cheap

# machine-readable gate verdict (schema/fix-validation.json)
npx falsegreen-skill fix tests/test_discount.py --case C20 --line 22 --sut src/discount.py --json

The gate runs three checks on a clean replica: the patch parses, it passes pytest against the real code, and it fails on a line-scoped mutation of the SUT (a built-in operator on the SUT line; full mutmut integration is deferred to a later version). A patch is accepted only when it both passes on correct code and goes red on the mutant, which is what proves the new assertion catches a bug instead of being a fresh tautology. Without --sut it degrades to propose-only and says the fix is unvalidated. The honest limit: the gate proves the fix catches the targeted mutant, not every possible bug. JS/TS/Robot and the deep semantic cases are v2.

Author a test from a spec (authoring mode, Mode B)

The same protocol runs in reverse. Given a spec and the code under test, the skill proposes a test with an independent oracle: an expected value derived from the spec, not read back from the code. Before writing it, an architect/QA gate (A0) runs the review judgments and the precision rules over the design, so the generated test is one the review pass would accept rather than a fresh false-green.

Two ways to drive it. In a host (Claude Code, Codex, Gemini, Cursor, plain LLM) ask the skill to "write a test for this function against this spec" and it elicits the level, language, and oracle interactively. Or on the CLI, write the answers into a test-spec file (schema/test-spec.json) and render one stack, self-checked:

# render the shipped example spec to a Python test, then run Mode A on the result
falsegreen-skill generate examples/authoring/apply-discount.spec.yaml --lang python
# other stacks from the same spec: --lang typescript | javascript | robot

The CLI does not elicit - a spec with no oracle is refused, because a test generated from the code's current output only freezes the bug. See SKILL.md, docs/cli.md, and docs/decisions/0004-authoring-mode.md.

Claude Code (primary path)

Add the marketplace and install the plugin:

/plugin marketplace add vinicq/falsegreen-skill
/plugin install falsegreen-skill@falsegreen

Then invoke the skill with /falsegreen-skill:falsegreen-skill, or just attach a test file and ask for false-positive analysis - the skill triggers on intent. The skill identifies the language and framework, classifies the test intent, applies the six-judgment protocol, and reports findings with case numbers, confidence levels, and fix hints.

For Python, the skill applies the full pattern catalog directly. Optionally, run the static scanner first to speed up batch analysis:

pip install falsegreen
falsegreen tests/

If you provide the scanner output, the skill uses it as the structural pass and applies semantic judgment on top. Without it, the skill runs everything.

Defined API providers

This skill is not tied to Claude. The maintained provider paths are Anthropic, OpenAI/Codex, Google Gemini, and the configured CLI providers listed in providers.md.

See providers.md for per-provider invocation code and Cursor setup.

Cursor

Add .cursor/rules/falsegreen-skill.mdc to your project (template in providers.md). Open a test file, ask Cursor to analyze it for false-positive smells, and the J1-J6 protocol runs automatically.

Supported languages and frameworks

Language	Frameworks
Python	pytest, unittest
TypeScript	Jest, Vitest, Mocha + Chai, React Testing Library, Vue Test Utils, Angular TestBed
JavaScript	Jest, Vitest, Mocha + Chai, Jasmine, React Testing Library
Robot Framework	BuiltIn, Collections, SeleniumLibrary, Browser, RequestsLibrary, RESTinstance, DatabaseLibrary, AppiumLibrary

Gherkin/BDD (Cucumber.js, behave, pytest-bdd) and Tavern are covered as secondary semantic passes in reference.md.

Frontend component tests - React, Vue, Angular, Svelte - use the same J1-J6 framework as backend tests. The structural failures are identical: a J4 weak assertion on a rendered component is the same smell as a J4 on a service method. See the family-based examples under examples/typescript/ (for instance family_a_never_checks.ts, which carries the Testing Library patterns) for annotated cases.

Test levels (the pyramid)

The skill detects the test level and reads the oracle in light of it, the step the static scanners cannot fully do. The level changes what counts as a valid check:

Unit: a function or component with its boundaries doubled. A real assertion on the return value is the oracle.
Integration (API and database): API tests (supertest, httpx, a framework TestClient, Tavern) and database tests against a real datastore. The response or the row IS the verification at this level, so the skill does not flag it as a weak check.
E2E: Cypress, Playwright, Selenium, Robot Browser. The presence of a rendered element or a page state is a real check here.

The level itself is part of the judgment: a real API or database call inside a test that claims to be a unit test is a smell (over-mocking inverted, mystery guest), and the skill says so rather than accepting the level at face value.

Patterns by test level and scope

The same false-green shape is classified by the level the test runs at: the level is a per-finding axis (J3), read as unit, integration, or E2E. As the superset of the three scanners, the skill covers every level and every language, plus the semantic S1-S21 patterns no parser sees. The clusters at each level:

Unit: always-true and self-compare (C5/C7/JS30), no oracle (C2/C2b), asserts its own double (JS8/JS27/C13b/S8), and the semantic S1/S5 (intent mismatch, tests the framework).
Integration: request oracle off (C9b), captured log never asserted (C50), patching the edge wrong (S12/S18).
E2E: sleep as synchronization (C16), forced green in Robot (R1/R2/R4/R6), and the semantic S1/S2 (intent mismatch, irrelevant oracle).

Full matrix on the docs site: patterns by test level and what we do not flag.

Project layout

falsegreen-skill/
  SKILL.md              the skill protocol (language and LLM agnostic)
  AGENTS.md             Codex + Antigravity CLI rule file (auto-parsed at workspace root)
  GEMINI.md             Antigravity CLI rule file (auto-parsed) + legacy Gemini CLI contextFileName / import source
  llm.md                self-contained prompt context used by CLI/API examples
  reference.md          per-language case catalog and framework cues
  providers.md          multi-LLM invocation guide (API snippets)
  CREDITS.md            the research this skill builds on
  gemini-extension.json legacy Gemini CLI manifest (source for `agy plugin import gemini`)
  .antigravity-plugin/  Antigravity CLI (`agy`) plugin manifest + skill (`agy plugin install`)
  .agents/skills/       Antigravity CLI (`agy`) workspace Agent Skill entry point
  .gemini/              legacy Gemini Agent Skill entry point
  .claude-plugin/       Claude Code plugin manifest + marketplace catalog
  .codex-plugin/        Codex CLI plugin manifest
  .agents/plugins/      Codex CLI marketplace catalog
  skills/
    falsegreen-skill/   shared skill entry point (Claude Code + Codex plugins)
  bin/
    falsegreen-llm.js   zero-dependency CLI (npx falsegreen-skill)
  scripts/
    validate-package.mjs validate manifests, frontmatter, and schema naming
    build-targets.mjs    generate standalone Claude/Gemini skill packages
  docs/
    architecture.md     architecture overview + Mermaid flow diagrams
    cli.md              CLI usage guide
    packaging.md        target packaging and release checklist
  schema/
    finding.json        JSON Schema for a single finding
    report.json         JSON Schema for a full report
  contexts/             ready-to-use context files per platform
    claude.md           Claude Code CLI, Claude.ai, Anthropic API
    codex.md            ChatGPT, OpenAI API, structured output, batch
    gemini.md           Google AI Studio, Gemini API, long context
    cursor.md           Cursor IDE, full .cursor/rules/ MDC template
  examples/
    python/
      family_a_never_checks.py       C1, C2, C2b, C3, C4, C4b, C20, C21, CC
      family_b_weak_always_true.py   C5, C6, C6b, C7, C8, C9, C11a, C13, C13b, C14, C16, C18, C25, C34
      family_c_checks_own_setup.py   C19, C28, C29
      family_d_external_state.py     C17, C23, C24, C27, C30, C31, C32, C35
      family_e_wrong_thing.py        C33, C36, C37
      semantic_cases.py              cases 10, 11, 12, 15, 18 (LLM-only)
      diagnostic_codes.py            D1, D3, D4, D5, D6, M2 (opt-in)
    typescript/
    javascript/

Setup and usage reference

Everything above is the tour. This section is the complete reference: every install mode, every CLI flag, every provider, and every host, with copy-paste blocks. Commands and flags here are taken from bin/falsegreen-llm.js --help and the host manifests, so they match the shipped 0.6.x line.

1. Install the CLI

The CLI is a zero-dependency Node script. It needs Node 18 or newer (the engines floor in package.json; it relies on the built-in fetch).

# run once, no install
npx falsegreen-skill analyze tests/test_payment.py

# install globally, then call `falsegreen-skill` anywhere
npm install -g falsegreen-skill
falsegreen-skill analyze tests/test_payment.py

# pin it as a dev dependency in a repo
npm install -D falsegreen-skill
npx falsegreen-skill analyze tests/test_payment.py

falsegreen-skill --version prints the installed version; falsegreen-skill --help prints the full command and flag list.

2. `analyze` - review test files

falsegreen-skill analyze <file...> [options]

Each file is sent to the provider in its own request. Plain-text output is printed under a === {filename} === header per file. With --json, each response is validated against schema/report.json and the CLI emits one aggregate JSON report.

Full flag reference (from --help):

Flag	Meaning	Default
`--provider <name>`	`anthropic`, `openai`, `gemini`, or `openai-compatible`	`anthropic`
`--model <model>`	Override the provider default. Required for `openai-compatible`	per provider (below)
`--base-url <url>`	API base URL. Required for `openai-compatible`	none
`--json`	Validate and output JSON conforming to `schema/report.json`	off
`--conventions <file>`	Conventions YAML/text injected per SKILL.md Step 0	none
`--temperature <n>`	Sampling temperature 0.0-1.0. Omitted automatically for OpenAI o-series	`0.2`
`--max-tokens <n>`	Max output tokens per request	`4096`
`--fail-on-high`	Exit 2 when any HIGH finding is present. Requires `--json`	off

Default models per provider: anthropic -> claude-sonnet-5, openai -> gpt-5, gemini -> gemini-2.5-flash. The openai-compatible provider has no default model, so --model and --base-url are both required.

Exit codes:

Code	Meaning
0	Analysis completed (findings may still exist; analyze is not a gate by itself)
1	Error: missing file, missing API key, bad flag, invalid JSON, schema mismatch, non-2xx API response
2	`--fail-on-high` was set and the JSON report contains at least one HIGH finding

Environment variables (one per provider, read from the environment):

Variable	Used by
`ANTHROPIC_API_KEY`	`--provider anthropic`
`OPENAI_API_KEY`	`--provider openai`, and fallback for `openai-compatible`
`GEMINI_API_KEY`	`--provider gemini`
`FALSEGREEN_API_KEY`	`--provider openai-compatible` (takes precedence over `OPENAI_API_KEY`)

One example per provider

# Anthropic (default provider)
export ANTHROPIC_API_KEY=sk-ant-...
falsegreen-skill analyze tests/test_payment.py
falsegreen-skill analyze tests/test_payment.py --model claude-opus-4-8   # deep case 18

# OpenAI
export OPENAI_API_KEY=sk-...
falsegreen-skill analyze tests/test_payment.py --provider openai
falsegreen-skill analyze tests/test_payment.py --provider openai --model o3   # reasoning, temperature auto-omitted

# Google Gemini
export GEMINI_API_KEY=...
falsegreen-skill analyze tests/test_payment.py --provider gemini

openai-compatible: any provider that speaks the OpenAI Chat API

Set FALSEGREEN_API_KEY to the provider key, point --base-url at the /v1 root, and pass the provider's model id. The CLI appends /chat/completions for you.

# Groq
export FALSEGREEN_API_KEY=gsk_...
falsegreen-skill analyze tests/test_payment.py \
  --provider openai-compatible \
  --base-url https://api.groq.com/openai/v1 \
  --model llama-3.3-70b-versatile

# Nvidia NIM (OpenAI-compatible endpoint)
export FALSEGREEN_API_KEY=nvapi-...
falsegreen-skill analyze tests/test_payment.py --json \
  --provider openai-compatible \
  --base-url https://integrate.api.nvidia.com/v1 \
  --model qwen/qwen3.5-397b-a17b \
  --max-tokens 8192

# Fireworks
export FALSEGREEN_API_KEY=fw_...
falsegreen-skill analyze tests/test_payment.py --json \
  --provider openai-compatible \
  --base-url https://api.fireworks.ai/inference/v1 \
  --model accounts/fireworks/routers/kimi-k2p6-turbo \
  --max-tokens 8192

# Ollama (local - no key needed, any placeholder works)
export FALSEGREEN_API_KEY=ollama
falsegreen-skill analyze tests/test_payment.py \
  --provider openai-compatible \
  --base-url http://localhost:11434/v1 \
  --model qwen2.5-coder:32b

# Alibaba Qwen (DashScope, OpenAI-compatible mode)
export FALSEGREEN_API_KEY=sk-...
falsegreen-skill analyze tests/test_payment.py \
  --provider openai-compatible \
  --base-url https://dashscope-intl.aliyuncs.com/compatible-mode/v1 \
  --model qwen-plus

Prompt size vs. free tiers. The system prompt carries the full catalog (llm.md + reference.md), ~33k tokens. Providers with a small per-minute token cap on their free tier (e.g. Groq's 12k TPM) reject the request with HTTP 413/429. Use a provider whose free or paid tier allows a ~35k-token request (Ollama locally, or a large-context model on OpenRouter/NVIDIA/etc.), and raise --max-tokens if a reasoning model gets cut off mid-report.

Set --model to the id your account actually exposes; the CLI passes the string through unchanged. Reasoning models work with --json as of 0.5.2: the CLI requests native JSON output, strips <think>/<reasoning> blocks, and recovers a slashed-key form (/findings) some schema-guided decoders emit. Verbose reasoners spend their output budget on chain-of-thought and can get cut off mid-JSON; if that happens the CLI says so and points at --max-tokens - raise it (8192 or higher) and retry.

3. `fix` - propose a stronger test and prove it (mutation gate)

falsegreen-skill fix <test-file> --case <code> --line <n> [options]

analyze finds a false-green; fix proposes a stronger test and runs a local gate to prove it before you trust it. It is opt-in, Python/pytest only, and propose-only: it prints a test-file patch but never applies it and never edits production code.

fix flags (in addition to the provider flags above):

Flag	Meaning
`--case <code>`	Catalog code of the finding to fix. V1 fixable set: `C2b`, `C20`, `C21`, `C5`, `C7`
`--line <n>`	Line of the finding in the test file (1-indexed)
`--sut <file>`	Production file the test protects. Required for a validated fix
`--sut-line <n>`	Line in the SUT to mutate. Defaults to `--line`
`--cheap`	Validation tier: parse + preserve only, no mutation gate

# propose a patch for a C2b finding and run the full gate against the real SUT
falsegreen-skill fix tests/test_discount.py --case C2b --line 14 \
  --sut src/discount.py --sut-line 12

# parse + preserve only, no mutation gate (no runnable SUT, or a quick pass)
falsegreen-skill fix tests/test_discount.py --case C5 --line 9 --cheap

# machine-readable gate verdict (schema/fix-validation.json)
falsegreen-skill fix tests/test_discount.py --case C20 --line 22 \
  --sut src/discount.py --json

What the gate proves. On a clean replica it runs three checks: the patch parses (py_compile), it passes pytest against the real code (preserve), and it fails on a line-scoped mutation of the SUT (a built-in operator flipped on the SUT line). A patch is accepted only when it passes on correct code AND goes red on the mutant, which is what shows the new assertion catches a bug instead of being a fresh tautology. The exit code is 0 on accept, 1 on reject/unvalidated, so CI can branch on it.

The honest limit. Without --sut (or with --cheap) the gate degrades to propose-only and labels the fix unvalidated. Even with the gate, it proves the fix catches the targeted mutant, not every possible bug; full mutmut integration and the deep semantic cases (10/11/12/18) and JS/TS/Robot fix paths are deferred to a later version.

4. `generate` - author a test from a spec (Mode B)

falsegreen-skill generate <spec-file> [--lang <language>] [options]

Renders a language-neutral test-spec into a real test, then runs analyze (Mode A) on the result, so a false-green test cannot pass the self-check undetected. It catches false-green shapes; it does not, and cannot on its own, tell a hand-written wrong oracle from a right one (see the honest limit below). The spec is a schema/test-spec.json file (YAML or JSON); see examples/authoring/ for one spec rendered into all four stacks. If --lang is omitted, the spec's first declared languages entry is used, else Python.

generate flags (in addition to the provider flags above):

Flag	Meaning
`--lang <language>`	Target stack: `python`, `typescript`, `javascript`, `tsx`, `jsx`, `robot`. `tsx`/`jsx` cover the React side of the JS/TS family (same JS* catalog). One language per run

# render the example spec to a Python test and self-check it
falsegreen-skill generate examples/authoring/apply-discount.spec.yaml --lang python

# same spec, TypeScript; the spec is the single source, re-run per stack
falsegreen-skill generate examples/authoring/apply-discount.spec.yaml --lang typescript

# machine-readable: { language, test, self_check, self_c

Falsegreen Skill

README

falsegreen-skill

New here? Start here

Why this exists

The methodology

What it detects

Python - structural patterns (complete falsegreen catalog)

Semantic patterns (all three languages)

Diagnostic and coupling codes (opt-in)

What we don't flag (and why)

How it compares

How to use

Installation

Skill or static scanner?

Quick example

Try it on your test suite

Propose a fix and prove it (AI-fix mode, V1)

Author a test from a spec (authoring mode, Mode B)

Claude Code (primary path)

Defined API providers

Cursor

Supported languages and frameworks

Test levels (the pyramid)

Patterns by test level and scope

Project layout

Setup and usage reference

1. Install the CLI

2. `analyze` - review test files

One example per provider

openai-compatible: any provider that speaks the OpenAI Chat API

3. `fix` - propose a stronger test and prove it (mutation gate)

4. `generate` - author a test from a spec (Mode B)

Comments (0)

Falsegreen Skill

README

falsegreen-skill

New here? Start here

Why this exists

The methodology

What it detects

Python - structural patterns (complete falsegreen catalog)

Semantic patterns (all three languages)

Diagnostic and coupling codes (opt-in)

What we don't flag (and why)

How it compares

How to use

Installation

Skill or static scanner?

Quick example

Try it on your test suite

Propose a fix and prove it (AI-fix mode, V1)

Author a test from a spec (authoring mode, Mode B)

Claude Code (primary path)

Defined API providers

Cursor

Supported languages and frameworks

Test levels (the pyramid)

Patterns by test level and scope

Project layout

Setup and usage reference

1. Install the CLI

2. analyze - review test files

One example per provider

openai-compatible: any provider that speaks the OpenAI Chat API

3. fix - propose a stronger test and prove it (mutation gate)

4. generate - author a test from a spec (Mode B)

Comments (0)

2. `analyze` - review test files

3. `fix` - propose a stronger test and prove it (mutation gate)

4. `generate` - author a test from a spec (Mode B)