SkillCI

Regression testing for Claude Skills. When a model update silently changes how your skill behaves, SkillCI catches it in CI — and turns the failure into a permanent test case, automatically.

A skill fails against a model it's never been tested on → SkillCI doesn't just report red, it writes the missing test case for you (evals/_generated/...) so skillci accept turns it into permanent coverage. That loop — catch once, covered forever — is the whole point.

Why · What's inside · How this compares · Install · Quick start · GitHub Actions · Dashboard · VS Code · Compliance reports · For AI agents · MCP server · Commands · Status

Why

You write a Claude Skill. It works today. Six months from now, Anthropic ships a new model, and nobody tested your skill against it first — because until now, no tool did that automatically. It might stop triggering, ignore instructions it used to follow, or blow past a token budget you never knew it had. You find out by accident, not by CI.

Every other kind of software has a safety net for this — tests that fail the moment behavior changes. Skills have had none. SkillCI is that safety net: write down what a skill should do once, and get an automated answer to "did it survive the last model release" instead of finding out the hard way. Full rationale below.

What's inside

How this compares

Every other tool in this space does one slice of what skillci does — none combine OWASP-mapped skill security scanning, cross-model regression testing, git-native culprit-commit bisection, and a self-growing eval loop in one tool. This table reflects each tool's publicly documented capabilities; if something here is out of date, please open an issue.

Where a competitor is genuinely ahead — promptfoo ships 50+ red-team plugins to skillci's 16, and years of production hardening skillci's redteam assertions haven't had. Multi-turn crescendo attacks, one of the two gaps this table used to name specifically, are now covered (crescendo-jailbreak); plugin count is still smaller by design — docs/adding-a-redteam-plugin.md plus skillci scaffold redteam-plugin cover how new ones get added. Where skillci is structurally ahead — no cloud roundtrip to generate attacks, and a successful attack becomes a permanent CI regression test instead of a one-time report — that's a narrower, verifiable claim, not "better overall."

Install

Homebrew (macOS, Linux):

brew tap kabirnarang39/skillci
brew trust kabirnarang39/skillci  # newer Homebrew requires explicitly trusting third-party taps
brew install --cask skillci

Scoop (Windows):

scoop bucket add skillci https://github.com/kabirnarang39/skillci
scoop install skillci/skillci

Prebuilt binaries (Linux/macOS/Windows, amd64/arm64): download from the latest release — Linux/macOS ship as .tar.gz, Windows as .zip, with a checksums.txt alongside to verify against.

Supply chain: every archive ships with an SPDX SBOM (<archive>.sbom.json), and checksums.txt is signed keylessly via cosign using the release workflow's GitHub Actions OIDC identity — no private key exists to leak. Verify with:

cosign verify-blob --bundle checksums.txt.bundle \
  --certificate-identity-regexp "https://github.com/kabirnarang39/skillci/.*" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  checksums.txt

Go install (any platform with Go 1.25+):

go install github.com/kabirnarang39/skillci/cmd/[email protected]

An ANTHROPIC_API_KEY is needed for eval/regress (not for check, which is local-only and free).

Shell completion: skillci is built on cobra, so bash, zsh, fish, and powershell completion scripts are generated for free — no extra install.

# bash (current shell)
source <(skillci completion bash)
# bash (persist across shells)
skillci completion bash > /etc/bash_completion.d/skillci

# zsh (persist across shells — run once)
skillci completion zsh > "${fpath[1]}/_skillci"

# fish (persist across shells)
skillci completion fish > ~/.config/fish/completions/skillci.fish

Run skillci completion --help for the full per-shell instructions (including oh-my-zsh setups).

Quick start

# Scaffold config + an example eval case inside your skill's folder
skillci init path/to/your-skill

# Lint SKILL.md — no API calls, catches malformed frontmatter, missing
# references, description-length issues, committed secrets, a
# first-layer static security scan (OWASP Agentic Skills Top 10:
# malicious payloads, unpinned dependencies, over-privileged access,
# insecure metadata parsing, untrusted external instructions,
# cross-platform format issues), and basic "skill bloat" warnings
# (oversized body, duplicate instructions, too many/too-large referenced
# files)
skillci check path/to/your-skill

skillci check's security rules are mapped directly to 6 of OWASP's Agentic Skills Top 10 categories — AST01 (malicious skills), AST02 (supply chain: an install/pull command or Dockerfile FROM pinned to a floating latest tag instead of an immutable version/hash — OWASP's own AST02 mitigation list names this exact practice), AST03 (over-privileged skills), AST04 (insecure metadata), AST05 (untrusted external instructions: a skill body that fetches instructions/config from a URL and directs the agent to treat it as authoritative), and AST10 (cross-platform reuse). This is a first-layer static scan, not a malware scanner — obfuscated or natural-language-only attacks can bypass pattern matching, a limitation OWASP itself documents (AST08). The other 3 categories — AST06 (weak isolation), AST07 (update drift), AST09 (no governance) — were checked against OWASP's own published mitigation lists for each: every mitigation (container sandboxing, signature verification on registry updates, SOC/CMDB inventory) is a property of the runtime, registry, or organization, never of a single SKILL.md file's text — so there's no honest static check to add for these three, not an omission.

One narrow, opt-in exception, since it's directly checkable without a registry: a skill author can declare pinned_sources in frontmatter — the exact expected SHA-256 of an external URL this skill's instructions depend on — and skillci check --verify-pinned-sources fetches it and confirms the content hasn't silently changed since it was pinned. This is the one flag in this entire command that makes a network call; everything else, including the check above, stays local-only. Off by default, and never runs unless you pass it by name:

---
name: my-skill
description: Fetches shared guidelines from a central doc.
pinned_sources:
  - url: https://example.com/shared-guidelines.md
    sha256: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
---

skillci check --verify-pinned-sources path/to/your-skill

A hash mismatch means the pinned content changed since someone last reviewed and pinned it — implementing AST02's "pin dependencies to immutable hashes" mitigation, and catching AST07-style update drift, for the one case that's actually checkable without a registry: content the skill's own author already opted to pin.

skillci check also flags basic skill bloat: an oversized SKILL.md body (over 8000 characters — every extra instruction is loaded on every invocation), exact-duplicate instruction lines (copy-paste bloat), and skills that reference too many files or too much referenced-file content (over 10 files or 100KB combined). These are fixed thresholds, not user-configurable, and — like the security rules — purely local pattern matching, not a judgment call about whether a skill is good, just whether it's carrying more than it needs to.

Add --format json for machine-parseable output — a stable array of {file, line, rule, msg, severity} objects on stdout ([] for a clean skill, never null), with the pass/fail exit code as the only other signal a caller needs. This is what the VS Code extension below runs under the hood; useful for any other editor integration or script too.

Every issue is block severity by default — the long-standing behavior, a lint issue fails the command. For rolling check out to a team gradually instead of an all-or-nothing cutover, --mode warn reports every issue but never fails the command:

skillci check --mode warn path/to/your-skill

Or make it permanent (and promote specific rules back to blocking) via .skillci.yaml:

lint:
  mode: warn                        # default severity for any rule not named below
  rules:
    ast01-pipe-to-shell: block      # promote just this one while piloting the rest

--mode on the command line always wins over .skillci.yaml — including over a per-rule block override — for a quick one-off check without touching the config file.

# Run the eval suite against one model
skillci eval path/to/your-skill --model claude-sonnet-5

# Run the full regression matrix (every model in .skillci.yaml), diffed
# against the last known-good run — fails CI only on a *new* regression
skillci regress path/to/your-skill

--verbose additionally prints, for any judge criterion that used judge_samples > 1, every individual sample's verdict — and flags a non-unanimous vote as ⚠ SPLIT VOTE even when it passed. LLM judges are known to disagree with themselves at close-to-coin-flip rates on borderline calls; a 2/3 pass and a 3/3 pass are not equally trustworthy, and this makes that difference visible instead of collapsing both into the same bare "PASS."

What actually fails CI is controlled by fail_on in .skillci.yaml — three values:

• regression (the default) — fail only on a case+model that used to pass and now doesn't. A case with no prior recorded run never fails CI on its own; it gets proposed as a generated case instead (see below).
• any_fail — fail on any failing case+model, regardless of history. Stricter; useful once a skill's suite is mature enough that every failure should be treated as a real problem.
• triggered_only — fail only when a case's triggered assertion doesn't match (ignores contains/budget/snapshot/fuzz/etc. failures entirely). Useful early on, when trigger accuracy is the only thing you're confident enough in to gate on.

A strict_dimensions match (see below) always fails CI regardless of which fail_on policy is set.

Heads up if you're new here: under the default regression policy, a brand-new eval case that fails on its very first run will NOT fail CI — there's no prior passing run to regress from, so it's proposed as a generated case instead (see below) rather than blocking the build. If you just added a case and expected red CI, check for a proposed new eval case line in the output before assuming something's broken. Switch to any_fail if you want every failure, including a brand-new case's, to gate CI from day one.

An eval case (evals/*.yaml) looks like this:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  contains: ["autumn"]
  max_tokens_loaded: 3000

When regress finds a failure with zero prior coverage for that case+model, it writes a proposed case to evals/_generated/ instead of just failing:

[FAIL] unrelated-request-should-not-trigger (claude-sonnet-5)
    triggered = true, want false
proposed new eval case: evals/_generated/unrelated-request-should-not-trigger-generated-claude-sonnet-5.yaml

skillci accept unrelated-request-should-not-trigger-generated-claude-sonnet-5

promotes it into evals/, so it's a tracked regression guard from then on. Pass --open-pr to skip the manual git add/commit step too: skillci commits the generated case(s) onto a new branch, pushes it, and opens a pull request against the branch the run started from, so review happens in your normal PR flow instead of a file someone has to notice in the workspace:

skillci regress --open-pr

Requires GITHUB_REPOSITORY and GITHUB_TOKEN (a token with contents: write and pull-requests: write on the repo — secrets.GITHUB_TOKEN in a GitHub Actions workflow already has both if the job's permissions: block grants them). A failure here (missing token, push rejected, API error) is reported as a warning, never as a CI failure — the generated case file is already written either way.

The generated file also carries the failure context that produced it — model, detection time, and the model's actual response — as a YAML comment header, so a reviewer deciding whether to accept it doesn't have to go dig that back out of CI logs:

# generated by skillci's self-growing eval loop — informational, not part of the case spec
# model: claude-sonnet-5
# detected_at: 2026-07-24T21:30:00Z
# actual_response:
#   I can't help with that request.
name: unrelated-request-should-not-trigger-generated-claude-sonnet-5
prompt: unrelated request
assert:
  triggered: false

That detected_at also backs a staleness check: once a case+model's first failure is recorded, regress never re-proposes or re-flags the same combination again on later runs (see fail_on: regression above) — so a generated case nobody ever ran accept on can otherwise sit under evals/_generated/ forever with CI staying green the whole time. regress warns instead of staying silent once one's been sitting there over 14 days:

warning: 1 unaddressed generated eval case(s) older than 14 days — run `skillci accept <name>` or delete them:
  evals/_generated/unrelated-request-should-not-trigger-generated-claude-sonnet-5.yaml (detected 2026-07-10)

For cases where you want to know if a skill's actual response content drifts between runs — not just whether it triggered or hit the right substrings — add snapshot: true:

name: "haiku-tone-check"
prompt: "Write a haiku about the ocean."
skill_under_test: "haiku-writer"
assert:
  triggered: true
  snapshot: true

The first run captures the response as a per-model golden baseline (evals/<case>.<model>.golden.txt). Every later run word-diffs the new response against it and shows what changed — without failing CI, unless you also set snapshot_strict: true. This is deliberately informational by default: it tells you that something changed, the same way any snapshot-testing tool (Jest, ApprovalTests) does — not whether the change is good or bad. You decide, then run:

skillci diff my-case --path path/to/your-skill --model claude-sonnet-5   # inspect
skillci accept my-case --model claude-sonnet-5                    # promote

To check whether a skill's trigger behavior is robust to rewording — not just whether the exact eval prompt fires it — add fuzz: true to a case that also asserts triggered:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  fuzz: true

Every run generates deterministic paraphrases of the prompt — synonym swaps, negation insertion, sentence reordering, and unrelated leading context — and checks whether the skill still triggers (or doesn't) the way triggered expects. No LLM writes the paraphrases; the mutations are fixed, non-random string transformations, so a fuzz run costs nothing beyond the extra model calls it makes. Note that regress fuzzes every model in your configured matrix, so total API calls scale as models × fuzz-enabled cases × (1 + up to 31 mutations) — the worst case is a 3-sentence prompt that hits all eight operators (1 synonym-swap + 2 negation + 5 non-identity 3-sentence reorderings + 3 context-prefix + 5 each from the four character-level operators below = 31 mutations), for 32 calls per model per case including the primary run. Like snapshot, this is informational by default:

[FUZZ] 2/26 mutations flipped trigger behavior
  negation: "Can you don't write me a haiku about autumn leaves?" -> triggered=false (want true)

Add fuzz_strict: true to fail CI on a flip. Run it standalone with:

skillci fuzz path/to/your-skill --model claude-sonnet-5

or let it run automatically as part of skillci regress for any case that sets fuzz: true — no separate invocation needed for full coverage.

Four more operators cover character-level perturbation specifically — typo-perturbation (a single deterministic transposed-letter edit), case-mutation (alternating case), whitespace-obfuscation (a zero-width space inserted mid-word), and unicode-homoglyph (Cyrillic look-alike substitution, the same technique ascii-homoglyph-smuggling uses for security testing, applied here to robustness instead). Together with the original 4 operators, this maps onto all 4 levels of PromptBench's own published LLM-robustness taxonomy (character/word/sentence/semantic) — see microsoft/promptbench, whose paper is titled PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (arXiv:2306.04528).

A mutation that flips trigger behavior on a fuzz_strict: true case's first-ever run doesn't just print once and disappear — it self-grows into a permanent evals/_generated/*.yaml case pinned to that exact wording, through the same self-growing eval loop redteam: uses. skillci accept promotes it into a normal regression test that fails until the skill genuinely stops mis-handling that phrasing. Capped at 5 self-grown cases per case+model per run — the console [FUZZ] line states the cap explicitly when hit, so nothing is silently dropped from view even though only 5 become files.

The deterministic operators above test a different failure mode than the one that actually burns teams: a fixed synonym dictionary can't anticipate how a real user would reword a request. Add fuzz_llm: true alongside fuzz: true to additionally have the model itself generate 3 realistic paraphrases:

assert:
  triggered: true
  fuzz: true
  fuzz_llm: true

Unlike the deterministic operators, this costs a real API call — but only once per unique prompt, ever. Generated paraphrases are cached to .skillci/fuzz-llm-cache.json (git-committed like history.json), so every subsequent run against the same prompt — including on a different machine or CI runner — reuses the cached set instead of asking the model again. Pay the generation cost once, not per run.

Model responses aren't fully deterministic — even at low temperature, sampling variance can make a triggered/contains/not_contains check fail on an otherwise-healthy skill. For cases where that matters more than the extra API cost, flake_retries reruns a failing case's trigger checks and takes a majority verdict instead of trusting a single sample:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  flake_retries: 2

By default, this only fires when the FIRST attempt's trigger checks fail — a passing case never pays the extra cost, but that has a real blind spot: a case that's genuinely flaky but happens to pass on attempt 1 is recorded as a clean pass with no vote at all, so it catches false positives (one unlucky roll) but not false negatives (a real regression that got lucky). Add flake_always_sample: true to vote every time, regardless of the first attempt's result, closing that gap at the cost of running the full vote on every case, not just failing ones — the same symmetric approach judge_samples already uses for the judge layer (see below). Up to 1 + flake_retries total attempts are made either way, stopping early once a majority is mathematically decided (e.g. 2 failing attempts out of 3 possible stops before the 3rd call). Budget assertions (max_tokens_loaded, max_output_tokens, max_latency_ms, max_cost_usd) are never retried — they're checked once, same as always, since rerunning can't change a token-count-derived cost or latency reading into something more "correct."

An odd flake_retries value can tie (e.g. flake_retries: 1 → 2 total attempts, 1-1). A tie is informational only by default:

[RETRY] triggered check unstable after 2 attempts — 1/2 passed (tie), informational only unless flake_strict is set

Add flake_strict: true to fail CI on an unresolved tie instead.

Deterministic assertions can't check response quality — tone, empathy, whether an explanation is actually clear. For that, judge sends the response to a separate model with a rubric of named criteria and takes its verdict, informational by default:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
judge:
  - name: tone
    criterion: "Is the response warm and encouraging, not clinical?"
  - name: imagery
    criterion: "Does the haiku use at least one concrete visual image?"

Requires judge_model in .skillci.yaml — deliberately a separate model from the ones under test, never the model judging itself, since a model can't reliably judge its own drift:

judge_model: claude-opus-4-8

All criteria must pass for the judge step to pass. By default, every criterion is evaluated together in a single API call (the batched mode). Judging only runs once every other assertion has already passed — it's the last check, not a substitute for triggered/contains. Add judge_strict: true to fail CI on a failing criterion; without it, a failure just prints:

[JUDGE] 1/2 criteria failed
  tone: FAIL — reads as clinical rather than warm

To trade cost for full per-criterion reasoning isolation, set judge_mode: isolated in .skillci.yaml (the global default) or override it per-case with assert.judge_mode. In isolated mode, each criterion gets its own API call and independent reasoning, eliminating cross-criterion bias; in batched mode (the default), all criteria share one call. Judge model's own sampling variance can make verdicts flaky on borderline cases. For cases where that matters, judge_samples draws N independent verdict samples per criterion group and votes unconditionally every time — the same symmetric approach flake_always_sample opts the case model into (see above), just always-on here rather than a flag:

assert:
  judge:
    - name: tone
      criterion: "Is the response warm and encouraging, not clinical?"
  judge_samples: 3

The [JUDGE] output shows sample tallies when judge_samples is set above 1 (e.g. "(2/3 samples passed)"). Judge verdicts are cached to .skillci/judge-cache.json so a repeat call against an unchanged response never re-pays for the API call, matching how .skillci/fuzz-llm-cache.json avoids regenerating paraphrases.

This is deliberately the most opt-in, most secondary assertion type in skillci: the whole premise of this tool is catching model drift deterministically, and an LLM judge is the one technique that can't judge itself reliably when the judge model is also something that might drift. Reach for the deterministic assertions first; add judge only for what genuinely can't be checked any other way.

Deterministic assertions and judge criteria both test whether a skill behaves correctly on intended input. redteam tests whether it holds up against adversarial input, across 16 plugins:

name: "haiku-request-redteam"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  redteam:
    - plugin: prompt-injection-canary
    - plugin: instruction-leakage
    - plugin: jailbreak-direct-override
    - plugin: harmful-content-elicitation
    - plugin: system-prompt-leak-semantic
    - plugin: excessive-agency
    - plugin: ssrf-bait
    - plugin: ascii-homoglyph-smuggling
    - plugin: pii-exfiltration
    - plugin: crescendo-jailbreak
  redteam_strict: true

The deterministic plugins in the table above are fully local — a pure string/token check against the attack response, zero judge calls, zero network calls beyond the one attack request. Every judge-graded plugin reuses the same judge model judge: criteria use, and requires judge_model configured the same way. "Calls" in the table above is the cost of running that one plugin for one case: most are a single extra model call (plus one judge call, for judge-graded plugins), but a multi-turn plugin sends one call per turn before its judge call — budget accordingly if you enable a multi-turn plugin broadly. A successful attack is informational only unless redteam_strict: true — and, same as any other case, a first-time-failing redteam case self-grows into a permanent regression under evals/_generated/ via the existing self-growing eval loop, with zero special-casing: a caught jailbreak becomes a test that runs forever, not a report you read once.

When a case that used to pass starts failing, skillci bisect finds which commit in your skill's own git history broke it — the same binary-search idea as git bisect, aimed at your skill instead of your code, holding the current model fixed throughout:

skillci bisect my-case --path path/to/your-skill --model claude-sonnet-5

With no --good/--bad flags, it looks up the last recorded passing run for that case in .skillci/history.json and the most recent recorded run, and searches the commits between them that touched the skill's files — checking out each candidate into a disposable git worktree (your actual working tree is never touched) and re-running the case against it:

verifying good/bad endpoints...
  9f8e7d6 — fail
  a1b2c3d — pass
good: a1b2c3d (2026-06-01) — passes
bad:  9f8e7d6 (2026-07-20) — fails
7 candidate commits, up to 3 more API calls
bisecting...
  4f3a2b1 — pass
  8c7d6e5 — fail

culprit: 6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3
author:  Kabir Narang <[email protected]>
date:    2026-07-10
message: tighten haiku-writer's tone guidance

--- SKILL.md (6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3^)
+++ SKILL.md (6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3)
@@ ...
- Write a haiku about the requested topic.
+ Write a haiku about the requested topic, staying strictly formal in tone.

skillci regress also prints a skillci bisect ... suggestion inline whenever it detects a new regression, so you don't need to remember the command yourself. Pass --auto-bisect to have it run bisect immediately instead of just printing the suggestion — useful in CI, where the run's own logs then already contain the culprit without a second manual invocation:

skillci regress --auto-bisect

Merge commits in the range are detected automatically — bisect falls back from binary search to a full linear scan of every candidate, which correctly finds the actual commit that introduced the regression instead of (incorrectly) landing on the merge commit itself. If the history genuinely has more than one commit where behavior transitions from passing to failing, that's reported as a warning rather than silently picked.

Every verified (case, model, commit) result is persisted to .skillci/bisect-cache.json, so re-running bisect on the same case — after an interruption, or while investigating a related case whose range overlaps commits already tested — never re-checks-out or re-runs a commit it already has an answer for; those show up as (cached) in the output instead of a fresh API call.

For cost and latency budgets, three more assertions are available:

name: "cost-budget-case"
prompt: "Write a haiku about autumn."
skill_under_test: "haiku-writer"
assert:
  triggered: true
  max_output_tokens: 500
  max_latency_ms: 3000
  max_cost_usd: 0.01

max_output_tokens and max_cost_usd are hard caps — like max_tokens_loaded, exceeding either fails the case immediately. max_cost_usd needs a pricing entry in .skillci.yaml — skillci never hardcodes or guesses prices, since Anthropic can reprice without notice:

pricing:
  claude-sonnet-5:
    input_per_million: 3.0
    output_per_million: 15.0

A case asserting max_cost_usd for a model with no pricing entry fails loudly, naming the missing model, rather than silently skipping the check.

max_latency_ms is the one exception to the hard-cap rule: latency reflects network and inference variance, not what the skill actually did, so an exceeded cap is informational only — printed, not failed — unless you also set latency_strict: true.

Eval cases can also carry free-form dimensions for slicing results — e.g. a case representing your enterprise-tier traffic or a specific language variant:

name: enterprise-billing-question
prompt: "..."
assert:
  triggered: true
dimensions:
  segment: enterprise
  language: es

By default this only affects reporting — skillci regress groups its output by dimension (--- by dimension ---) so a cratering segment is visible at a glance instead of buried in a flat case list. To make a specific slice's failures always fail CI regardless of the global fail_on policy, name it in .skillci.yaml:

fail_on: triggered_only
strict_dimensions:
  segment: [enterprise]

Any case tagged segment: enterprise now fails CI on any failure, even though the rest of the suite is gated more loosely.

GitHub Actions

- uses: kabirnarang39/skillci/.github/actions/[email protected]
  with:
    path: path/to/your-skill
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

Pin the action itself to a release tag (@v0.4.1 above), not @main — @main floats onto whatever the action definition looks like next, silently changing your CI's behavior with no version control. The version input (defaults to v0.4.1) pins which skillci CLI binary gets installed, separately from the action reference; override it once a newer tag ships, or set it to latest if you deliberately want to float (not recommended for production — every consumer's CI would silently pick up whatever ships next, with no build reproducibility).

Gates CI on new regressions only — a flaky non-deterministic miss won't fail your build every time. Commits both the status badge (passing / partial / regressed) and .skillci/history.json back into the checkout on every run, including runs where a regression is caught — that history is what lets the self-growing eval loop avoid re-proposing the same generated case every run and lets skillci bisect auto-detect its good/bad commits, so it needs to actually reach your repo. The action only commits locally within the checkout; add a push step in your own workflow (or git-auto-commit-action) to land it.

Not on GitHub Actions? skillci is a plain Go binary with no GitHub-specific dependency — see docs/ci/gitlab.md, docs/ci/azure-devops.md, and docs/ci/jenkins.md for copy-paste snippets. No dedicated plugin needed for any of them; a shell step calling the binary is all it takes.

On a pull request, set pr-comment: true to get results posted directly on the PR instead of only in the Action's own log — the same comment gets updated on every subsequent push to that PR rather than piling up a new one each time:

- uses: kabirnarang39/skillci/.github/actions/[email protected]
  with:
    path: path/to/your-skill
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
    pr-comment: true
permissions:
  pull-requests: write

Requires pull-requests: write on the workflow's GITHUB_TOKEN (the default token from a pull_request-triggered workflow already has this if the job's permissions: block grants it — see above). A failure posting the comment (missing permission, API hiccup) is reported as a warning in the Action's log, never as a build failure — pr-comment is a convenience on top of the exit code, not a replacement for it.

Optional: hosted dashboard

cmd/skillci-server is a small Postgres-backed HTTP server that turns skillci regress --upload results into a public, per-skill compatibility history and leaderboard — the "does my skill still pass on the model shipped this week" trust signal, shareable the way a codecov badge is. Entirely opt-in; the CLI works standalone forever without it.

Each skill's page renders one