Behavioral regression tests for Agent Skills

Catch skills that miss the right prompts or activate on the wrong ones. Run the tests locally, commit the expected behavior, and gate every skill change in CI.

Quick start · GitHub Action · How it works · Commands · Security · Browser lint

Why Tripwire exists

An agent decides whether to load a skill from its name and description. That makes the description routing code, not ordinary documentation.

Two costly failures are silent:

• A user asks for the intended outcome, but the skill never activates.
• An unrelated prompt activates the skill and injects the wrong instructions.

A schema linter cannot observe either failure. Tripwire runs real agent sessions, records the activation event, and compares the result with an explicit behavioral contract.

Tripwire is useful when other people depend on your skill: a public skill, a shared company library, or a repository where skill changes require review. A personal, low-stakes skill usually does not need a behavioral test suite.

Quick start

Tripwire requires Node.js 20 or newer.

npm install -g tripwire-skills

Start with the free static check:

tripwire lint ./skills/code-review

Run one real activation check with your existing agent CLI login:

tripwire test ./skills/code-review \
  --prompt "review this pull request for security problems" \
  --expect activate

This path does not need a separate prompt-generation API key. Once it works, generate and run a broader activation matrix:

export ANTHROPIC_API_KEY=...
tripwire analyze ./skills/code-review

analyze writes tripwire-scenarios.yaml beside the skill. Review it like a test file:

skillName: code-review
scenarios:
  - prompt: review this pull request for security problems
    zone: core
    expectedActivation: true

  - prompt: write release notes for this pull request
    zone: negative
    expectedActivation: false

Commit that file, then replay the same contract:

tripwire test ./skills/code-review

A missed activation, false trigger, agent timeout, or authentication failure returns a non-zero exit code. Infrastructure failures are reported separately from behavioral regressions.

The production workflow

1. Run tripwire lint while authoring a skill.
2. Run tripwire analyze to discover core, edge, negative, and paraphrased cases.
3. Review and commit tripwire-scenarios.yaml.
4. Run tripwire test in CI whenever the skill or its scenarios change.

This turns "the description looks right" into a reviewable contract with an executable gate.

GitHub Action

The Action always lints changed skills. Set probe: true to replay committed activation scenarios:

# .github/workflows/tripwire.yml
name: Tripwire
on: pull_request

jobs:
  skills:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v5
        with:
          fetch-depth: 0
      - uses: bharath31/tripwire@v1
        with:
          probe: true
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

The Action annotates the diff, posts or updates one PR summary, and returns a failing check for lint errors or probe failures. On forked pull requests without a key, it reports that the paid probe was skipped and still runs static lint.

How it works

For each scenario, Tripwire:

1. Copies only the skill under test into a disposable agent workspace.
2. Runs the selected agent with a read-only or plan-mode permission boundary.
3. Reads the structured transcript for the skill activation signal.
4. Compares observed activation with expectedActivation.
5. Deletes the temporary workspace.

The Claude adapter watches the structured Skill tool-use event. This is stronger than asking another model whether a description "looks likely" to activate.

Agent support is intentionally explicit:

Use --agent claude, --agent gemini, or --agent codex. The default can also be set in tripwire.yaml. Tripwire searches from the skill directory toward the repository root, so one file can govern a whole skill library:

agent: claude
model: claude-sonnet-4-6
judge_model: claude-haiku-4-5-20251001
concurrency: 3
probe_count:
  core: 8
  adjacent: 8
  negative: 8
  variants: 5

analyze currently uses Anthropic to generate the prompt matrix and judge activated sessions, even when a different runtime adapter is selected. The selected agent CLI must also be installed and authenticated. test replays committed prompts and does not regenerate them.

Security model

• Probe sessions run in a disposable workspace containing only the SKILL.md under test.
• Claude and Gemini run in plan mode. Codex runs with a read-only sandbox.
• Tripwire does not host your repository or proxy agent sessions through a Tripwire backend.
• analyze sends the skill content to Anthropic to generate probe prompts and judge sessions.
• The GitHub Action uses credentials from your runner and does not print the API key.
• Adapter failures are not converted into false behavioral results.

Tripwire sends one anonymous event after a behavioral run so project DAU and reliability can be measured. It contains the command, adapter, outcome, source, version, and a random hashed install or repository ID. It never contains prompts, skill names, paths, repository names, outputs, usernames, or credentials. Disable it with TRIPWIRE_TELEMETRY=0, TRIPWIRE_TELEMETRY_DISABLED=1, or DO_NOT_TRACK=1. The full event contract is in docs/analytics.md.

Please report vulnerabilities privately using the process in SECURITY.md.

Command reference

Run tripwire <command> --help for all options.

What static lint checks

The built-in rules cover frontmatter shape, kebab-case names, trigger-focused descriptions, maximum description length, placeholder text, body length, and the presence of an example. Rules can be changed or extended with tripwire.yaml:

Opinionated authoring style, including the exact Use when opening, is advisory by default. Teams that enforce that convention can promote it to an error:

extends: tripwire:recommended
rules:
  no-code-example: off
  description-use-when: error
plugins:
  - ./org-rules.mjs

Static lint is also available at tripwire.bharath.sh. It runs in the browser and is useful for a quick authoring check. Behavioral activation testing requires the CLI because it needs a real agent session.

Additional checks

• tripwire conflicts flags duplicate skill names and likely description overlap.
• tripwire test-all reruns every committed contract to expose model drift.
• tripwire eval asserts response content after a skill activates.
• The coverage badge lints a public GitHub-hosted skill at request time:

![tripwire](https://tripwire.bharath.sh/api/badge?repo=owner/repo&path=path/to/SKILL.md&branch=main)

These are secondary to the core workflow: define expected activation, run it against a real agent, and fail the change when behavior diverges.

Current maturity

Tripwire is pre-1.0. The static linter and Claude activation path have automated coverage and live verification. Gemini and Codex activation checks remain experimental until their live canary suites are published. Pin Tripwire and your agent CLI version in production workflows so upgrades are intentional.

Development

npm install
npm run typecheck
npm run typecheck:web
npm test
npm run test:web
npm run build
npm run build:action
npm run build:web

The test suites cover the CLI, Action, adapters, lint engine, evals, drift checks, browser playground, setup command generation, and public API contracts. Pull requests run them on Node.js 20 and 22, build the Next.js application, audit production dependencies, verify the published package contents, and check the committed Action bundle.

The Next.js application in web/ deploys to tripwire.bharath.sh through Vercel. The legacy Cloudflare Pages configuration is retained temporarily as a manual rollback path during the cutover window.

License

MIT