SOTA Engineering Skills

Make your AI coding assistant build and audit like your most senior engineer.

Your assistant is brilliant — it just doesn't know your standards, and it forgets the ones it does know as the task grows long. SOTA-skills fixes both, and the fix is measured: from a bare "build X" prompt, best-practice coverage climbs from ~59% to ~98% (+0.39) — the model stops silently dropping tests, rate limiting, structured logging, and TLS (see every number →).

It works by being a loop, not a prompt dump: route in only the rules a task needs, re-state them every turn, and re-check them last before shipping — so the guidance survives a long context instead of fading into it. That's why it beats a bigger prompt instead of becoming one. Native on Claude Code; works with Gemini CLI, Codex, and any agent that reads AGENTS.md.

Under the hood: 41 skills (303 files, ~66k lines) of state-of-the-art 2026 practice, each instruction file under 500 lines so only the matching rules load — the cap applies to skills/** alone, never to README/CHANGELOG/docs/ — every fast-moving claim web-verified against a primary source.

Two commands to install:

/plugin marketplace add martinholovsky/SOTA-skills
/plugin install sota-skills@sota-skills

Or clone + link (best if you want a local checkout to read, hack on, or pin). Skills are discovered from .claude/skills/ (per project) or ~/.claude/skills/ (personal, all projects). Clone the repo, then run the installer — it symlinks every skill (and your profile, if you have one):

git clone https://github.com/martinholovsky/SOTA-skills && cd SOTA-skills
./scripts/install.sh                 # personal: ~/.claude/skills (all projects)
./scripts/install.sh --project DIR   # one project: DIR/.claude/skills
./scripts/install.sh --copy          # copy instead of symlink (pin a snapshot)

The installer colour-codes what it did (✓ done · ↻ changed or act on this · · no-op) and drops to plain ASCII when the output is not a terminal, on a non-UTF-8 locale, on TERM=dumb, or with NO_COLOR set — --color=always|never|auto (or --no-color) overrides the detection either way.

Then describe the task in plain language — routing loads the right skills; the stack comes from your profile or the skills' defaults (naming one is optional):

Design a multi-tenant invoicing service.

Run a full audit of this repo — severity, effort, and fix on every finding.

More install options: Installation · more prompts: Using it.

Contents

• Standards & practices baked in · What the audit hunts that a scanner can't · How the numbers are kept honest
• Skills · Coverage & non-goals
• Installation · Always-on routing · Updating
• Using it
• Optional setup & integrations — badge, gates, other agents, status line, plugin extras
• Structure · How it works · Conventions
• Found a gap? Tell us · Contributing · License

Deeper docs: Find it fast (docs index) · Does it work? (measured results) · Why it works · Why some lifts expire and others don't · Keeping rules applied as context fills · Roadmap

Standards & practices baked in

Findings name the control they violate — not just "this looks wrong":

• Security — OWASP Top 10 (2025), ASVS, API & LLM Top 10; findings cite CWE IDs
• Languages — all 9 language skills (Rust → Ruby, below) get the same rigor; formal standards where they exist: SEI CERT (C, C++, Java), MISRA C/C++, ANSSI Rust
• Supply chain — SLSA, Sigstore, in-toto, SBOM (CycloneDX/SPDX), NIST SSDF
• Cloud & identity — CIS Benchmarks, NIST 800-207 zero trust, NIST 800-63-4, OAuth 2.1, FAPI 2.0, passkeys, SPIFFE
• Privacy & compliance — GDPR, CCPA/CPRA, HIPAA, PCI DSS 4.x, SOC 2, ISO 27001, EU AI Act, NIS2, DORA
• Government & regulated — NIST CSF 2.0, 800-53, 800-171/CMMC, FedRAMP, EU Cyber Resilience Act, IEC 62443
• Threats, detection & AI/ML — STRIDE, LINDDUN, MITRE ATT&CK & ATLAS, NIST 800-61, NIST AI RMF
• Frontend, mobile & testing — WCAG 2.2 AA, Core Web Vitals, OWASP MASVS & WSTG

Named standards are the floor. Most of the library is the practice layer no regulation writes down: cancellation & backpressure, retries with jitter, circuit breakers, outbox/saga, double-entry ledgers and the reconciliation that proves an integration is complete rather than merely correct, zero-downtime migrations, measure-first performance, API evolvability, per-language idioms, SLOs, test-suite health.

Measured, not asserted — library vs. an unguided model (same model, no library); clean, blind-judged, stable across samples (results & method →). Every number below names the model it was measured on, because a lift can be overtaken by model progress and then reads as current when it is historical — that happened to the defect-avoidance row on 2026-08-21 and is stated there rather than quietly left standing:

• Completeness +0.39 — and it did not expire. Re-measured 2026-08-21 on the current flagship claude-sonnet-5: 0.62 → 1.00, +0.38 — unchanged. From a bare "build X" prompt, best-practice coverage goes ~59% → ~98% (7 tasks): the model stops silently dropping tests, rate limiting, structured logging, and TLS. Web search likely can't recover this (an agent won't search "should I add rate limiting"). Not model-specific, and not stale: a different-family model (openai/gpt-5.1) shows +0.44 on the same tasks (cross-model →), and claude-sonnet-5 — four months newer than the model this was established on — still shows +0.38 (2026-08-21 →). Why this one lasted while defect-avoidance did not: newer models stopped writing known-bad patterns, but sonnet-5 unguided still omits tests in 7 of 7 tasks, transport in 5, rate limiting in 5. Knowledge gaps close with model progress; salience gaps do not.
• Freshness +0.53 → +0.30 — it erodes, and we measured that rather than assuming either way. Current-2026 facts (RFCs, CVEs, EOLs) 0.44 → 0.97 on claude-sonnet-4.6, where an unguided model is confidently wrong. Re-measured 2026-08-25 on claude-sonnet-5: 0.69 → 0.99, +0.30. The pre-registered prediction was that a training cutoff is a gap model progress cannot close — refuted: the newer model answers 8 of these facts unaided that its predecessor could not, because its cutoff advanced into a fixed case set. Ten of 32 remain outside it, and that is the surviving lift. So freshness is a third shape — neither expired like defect-avoidance nor flat like completeness. And the same day we proved the cause was the instrument, not the library: a fresh 10-case freshness set built from recent facts, each verified against its primary source and chosen by a rule fixed before any model ran, reads 0.33 → 1.00, +0.67 on the same model an hour later — higher than the original +0.53. The guidance did not improve between two runs; the questions got newer. A freshness lift is therefore quoted with the date its questions were written, the way every other number here is quoted with its model (2026-08-25 →).
• Routing +0.10 — and it did not saturate. The right skills load for the task (0.90 → 1.00 on claude-sonnet-4.6), even ones a keyword read misses. Re-measured 2026-08-25 on claude-sonnet-5: 0.87 → 0.99, +0.13 — unchanged within the set's one-case resolution (20 cases, 0.05/case). The unguided arm still misses the same rule-driven routes a model generation later — testing, sandboxing, code-security, web-frameworks (2026-08-25 →).
• Defects avoided +0.19 — a different axis, and baseline-dependent. Every lift above measures what the model puts into code. This measures what it doesn't: given a spec that states operational pressure ("cache it", "must never 5xx", "keep the guard cheap") and never names a defect, does the model still write SQL injection, IDOR, an inert control? On claude-sonnet-4.6: unguided 0.81 → with the library 1.00 across 7 defect classes; on the stricter measure that also demands positive evidence of the safe path, 0.29 → 0.62. It is baseline-dependent, and we tested that rather than assuming it: on openai/gpt-5.1 the unguided arm already scores 1.00, so there is no gap to close and the lift is +0.00 — while the stricter measure still moves (0.43 → 0.52). Same law the breadth test found for completeness: the lead tracks the unguided baseline, not the domain — and here, not the model. Honest headline: closes a defect-avoidance gap where one exists. Small pilot, 3×, two models, one task — method and limits →.
• Prompt independence +0.51 — the newest result, and the one measured where it matters. Every number above is measured under a neutral prompt. This one measures the same tasks under a competing prompt — the user's own words arguing against the rule: "internal MVP and we demo tomorrow, skip the extras", "just the function please, no tests", "put the requirement in the system prompt where it's easy to tweak". Unguided 0.491 → 1.000 with the library, +0.509 (6 tasks × 3 samples, temp 0.7 on claude-sonnet-4.6; the with-library arm was perfect in 18 of 18 runs). The lift grows with the pressure against it — +0.083 supportive, +0.236 neutral, +0.509 competing — so a rule that only survives a neutral prompt is absent exactly when it is needed. Told to skip the extras on a login endpoint, the unguided arm dropped both rate limiting and the password hash; the guided arm kept all four criteria. Includes a null that was opened and withdrawn the same day when its own confirmation run refused to reproduce it (method, both runs, six limits →).

How the numbers are kept honest — the same discipline the library teaches. Every prediction is committed before the run, with the result that would falsify it, so a refuted one is published rather than quietly re-framed (2026-08-25: a freshness prediction was refuted and the write-up leads with it). And case sets are built by a rule fixed before any model runs — never from the cases a model got wrong, which is selection bias and guarantees whatever gap you go on to report. That rule is now part of the library itself (sota-llm-engineering rules/01 §8), because the temptation is real: when an ageing set stops discriminating, the still-failing cases are sitting right there.

And not just vs. an unguided model — head-to-head against the most popular guidance libraries on backend build tasks, SOTA-skills leads on completeness (content-only, blind-judged; wins or ties all 21 cases, loses none):

A five-domain breadth test shows when this edge holds: SOTA-skills leads the field wherever a base model ships incomplete code — production backend in any language and complex/security-sensitive frontend (~+10 pts) — and ties where the base model is already near-complete (simple UI, templated infra). The lead tracks task difficulty, not the domain — we measure it and say so. Full breadth result, consolidated table, method & honest limits →

What the audit hunts that a scanner can't

Eleven classes of defect survive every linter, SAST rule, and CVE scanner, because in each one the code isn't wrong. The library hunts them as explicit passes:

Finding them is the cheap half — and we can prove it. Across nine instruments, a frontier model recognises these classes unaided: audit lift +0.00, published below rather than buried. The expensive half is not writing them in the first place. On a model that does write them, the library stops it: 0.81 → 1.00. On one that already doesn't (gpt-5.1, unguided 1.00) there is nothing to close — we ran that second model and report the +0.00 too (2026-08-21).

• Controls that are inert — a safeguard whose success and whose total failure look identical from outside: a swallowed enforcement exception, a ruleset that loads zero rules, presence decided by exists() rather than a loaded artifact, a CI gate whose every run is skipped (and on GitHub a skipped job reports Success to branch protection), a policy engine left in Audit/warn/report-only since the day it shipped, a report whose word "verified" traces to no line that can fail, silent truncation on the way out of a generator — an unset token cap returns a fragment that a swallowed parse error publishes as a valid empty result — a test that still passes when the control's body is replaced with a no-op, a write-back controller logging updated=1 errors=0 every cycle for fifteen minutes while pushing no commit at all — its log describes the update it decided on, not the write landing. The same shape one layer down: a count that is computed and still false, because it was derived from an intermediate the function discards and emitted before the return that dropped it. No function can attest to its own return value — every emission site has a suffix that can drop the result after the line is written — so the claim belongs in the consumer, derived from the value received. Where a job runs unattended, its log is the only witness, so the probe is to change what the function returns and read the output, not to re-run the tests. (rules/10, rules/14, kubernetes rules/04 §7)
• Controls that block everything — the mirror image, and the one every other pass here looks past. An enforcement control (cap, quota, filter, allowlist, sandbox policy) can be tightened until it refuses the legitimate case too, and it passes the same tests: a security suite asserts refusal, refusal is exactly what an over-tight control produces, and the mutation probe above only ever installs the permissive no-op, so it looks in one direction. The fix is an allow arm — a representative legitimate case that must complete through the control, never against the bare environment, because proving the machine can do the work says nothing about whether your control permits it. Worked instance: a memory budget set with RLIMIT_AS refuses the runaway allocation exactly as intended, and also kills every Go or JVM process at startup, since those runtimes reserve gigabytes of address space they never touch (measured: 1.17 GiB reserved against 2.3 MiB resident). (rules/12 §1a, sandboxing rules/02)
• Layers that were never layers — the same test applied to an architecture diagram rather than a function. A second-opinion classifier drawn from the same model family as the system it guards shares its blind spots by construction (common-cause failure), and one that only ever sees what the primary already flagged uncertain cannot, even in principle, catch what the primary got confidently wrong. A TEE bought to fix records that were never emitted is answering a liveness question no hardware guarantee covers. Both are counted in a threat model and neither reaches the case it is counted for. (rules/08 §1, rules/04 §8)
• Stages that report success while doing nothing — found by the cheap signals rather than by reading every line: a step returning "nothing found" far faster than its claimed work allows, a gate that never prints how many items it examined (0 checked, 0 failed, exit 0), a size-gated branch no fixture crosses, a cache key narrower than the behaviour it gates, a control written as an assert that -O/NDEBUG/a missing -ea deletes in production. When a tool's correct output cannot be stated at all — the test oracle problem, which is why "it found 0" goes unchallenged — a metamorphic check pins how the output must change: a fixture with N known items, and an assertion that the count moves when the input does. (rules/11, rules/13)
• A field that is always empty, and the seam nobody could have diffed. A defaulted read — .get(key, default) on a dict you did not construct — turns a key-name disagreement into a plausible constant, and when the producer is a model choosing field names from a distribution there is no change event to test against: the prompt and the reader can both be correct while the response uses a synonym. The only sound detector is at runtime — diff the keys the model returned against the keys anything consumed, and log the ones nothing read. (rules/13)
• A search that never ran, reported as a clean tree. In zsh — macOS's interactive shell, and the one these checklists get pasted into — an unquoted glob in a flag value (grep --include=*.md) trips the default NOMATCH and aborts the command, where bash would pass the word through and run it. Add the 2>/dev/null everyone uses to hide Permission denied noise and the broken probe is byte-identical to a real no-match: empty output, exit 1. The sweep you read as "nothing is stale" may never have executed. (rules/01 §3a)
• A watcher that cannot say "the thing I was watching is gone". Done/not-done cannot express "I could not tell", and once you add that you find you also need GONE — terminal and knowable, not unknown. Collapse it and the watch either invents a success or never ends. It is the row people delete while fixing the other bug: one rewrite replaced an explicit "no longer exists" branch with a blindness counter, which then reported "cannot read for 20min" about a job that had simply been garbage-collected. (rules/12 §2.2a)
• A measurement whose rows came from somewhere else. When tests and production write to one shared sink, every aggregate over it merges two populations — and the danger is not the false positive but the destroyed true finding, because the contaminated number carries the larger n and so reads as the more rigorous one. (rules/11, rules/05)
• Your own scorer, gate or benchmark doing none of the above — a whole file turns the lens around: anything whose output decides whether something is OK is a control too. A broken feature produces a complaint; a broken instrument produces a number, and numbers get quoted. So give it a known-bad input it must fail and a known-good one it must pass, bound what it reads, and never trust a number from an instrument you haven't watched produce a wrong answer on purpose. The sharpest case is a guard that is an instance of what it guards — a coverage test whose scope is narrower than the population and whose predicate the defect satisfies, so it passes on exactly what it exists to catch. The remedy every mature discipline reached independently — the proof test, the clinical positive control, aviation built-in test, adversary emulation — is one move: a negative control, a committed known-bad the gate must reject on every run, verified per target rather than once, because a tripwire that fires for 2 of 20 targets looks identical to full coverage. Worth knowing that no mainstream framework asks for this: NIST SSDF and the EU CRA require a record that a scan ran, OpenSSF Scorecard's SAST check detects only that a tool is configured, and SLSA will sign provenance for a scanner set to scan zero files. A passing compliance check is evidence of process, not of protection. And a negative control answers only whether the gate can fail — never whether it still covers you: a refactor into a nested module, a second manifest or a sidecar image moves code out of a gate's scope with no diff to the workflow file, so watch the number of units each gate enumerated and treat a drop as a failure. (rules/12, devsecops rules/05)
• Absence encoded as a value — the in-band sentinel: a number whose domain includes an "absent/unknown/error" marker (-1, 0, "", 9999-12-31). It type-checks and no linter flags it: -1 is truthy, so the if x: presence check everyone reaches for admits it, and because the sentinel carries an ordering it loses every < and wins every > — one missing operand makes a guard skip silently in one direction and fire spuriously in the other, from a single input. The tell is not the constant but the asymmetric guard: one operand filtered against the sentinel and the other, in the same comparison, not — because that filtering is applied per site, so it lands only where the author happened to be thinking about it. (architecture rules/02 §8a, with a per-language row measured on each toolchain)
• Dependencies declared but never reached — packages, modules, and plugins wired in and inert. Proven by deleting them in a scratch copy and running the real build, lint, and full suite, with exit codes and before/after transitive counts reported — a grep is not proof. (rules/03 §3.9)
• Decisions that stopped being right — the datastore picked for scale that never arrived, the rewrite justified by a benchmark that no longer reproduces. Every expensive-to-reverse decision is classified JUSTIFIED / STALE / UNJUSTIFIED / UNVERIFIABLE, and any number one rests on is re-measured this session.
• Findings that don't survive contact — every Critical/High gets an independent pass prompted to kill it, working from the code rather than the write-up and defaulting to REFUTED when the evidence is ambiguous. Refutations are recorded, so the next auditor doesn't re-raise them — and swept before they're dropped, because the refuted pattern routinely closes somewhere the audit's scope didn't cover. A severity also has to name its chain (reach → primitive → boundary crossing → channel), and on a diff it is rated against the code the change replaced: filing a High against a hardening fix teaches authors that hardening attracts findings. The refuter also gets less than the finder had — no execution, no writes — with only the artifact crossing over (the PoC, the failing command, the file:line), never your write-up; and it returns a number against a threshold fixed before the findings were seen, because a paragraph of hedging is a judgement you then re-judge. (findings §1, §4, §4a)
• A result you saw once — where the evidence is a behaviour (a crash, a race, a timing bypass, anything an agent or a sampled model produced), it is reproduced N of N and both numbers are reported: 1/1 and 3/3 are typeset identically and mean different things. On a repeat audit the yield curve is read too — count falling while difficulty rises. A flat count wave after wave is a statement about the audit, not the code. (findings §2, methodology §4)
• A control that works, on some of the sites it's credited with — not inert, not missing: a real containment check guarding one channel of five, a disable_tools=True passed at 2 call sites of 6, a path guard on 9 target walks of 61. Every signal is a working control's signal, because on the guarded subset it is one. The audit move is a census — enumerate the protected operation, mark each call site, report the ratio — and the API fix is that a parameter selecting a trust boundary defaults to the closed side. (code-security rules/14 §6)
• Security prose the code no longer keeps — "containment is applied at every target walk" (it was 9 of 61); "no function-calling, no shell — this RCE mechanism is NOT APPLICABLE" (tools were on by default). Each had been true when written. Most such claims are universally quantified and therefore falsifiable by counting, and a wrong NOT APPLICABLE is the worst case: it cancels the next reader's own investigation. (code-security rules/14 §7)
• Your own tool, ingesting somebody else's repository — scanners, SAST wrappers, review bots and agentic analysers run on a maintainer's machine with that identity's credentials, and the target is the attacker. Four legs, usually owned by four people: staging that dereferences symlinks out of the tree, "static" analysis that evaluates target-controlled build metadata, an LLM step spawned with tools live and the parent's cwd, and egress left on by default. (sandboxing rules/05 §7)
• Absence claims — "no hardcoded secrets remain" is the one finding nobody can falsify: a narrow search and a true absence produce identical output. Any absence claim needs a widened search plus a second independent method, with the search stated. (findings §2, §4)
• Controls keyed to a neighbouring setting — the predicate reads a proxy that agrees with the real dependency right up until someone configures the two apart, three files away. A coupling defect: the control's own site never changes, so neither per-file review nor a per-gate probe can see it — the signature is a control that silently stops at the moment it starts mattering. Field-reported 2026-08-26 with two shell shapes that fail the same way: a process substitution whose producer failed (its exit status is unreachable, so zero lines reads as success) and an "append" to a keyed store that is really an upsert, silently deleting the link target of a hash chain on the second run. The audit for this class is the proxy question, which the usual falsification question cannot answer: is this the thing I actually depend on, or something that currently agrees with it — and who can change one without the other, and would I find out? A control that is correctly enforcing the wrong predicate is not inert, so "would anything observable differ?" answers yes while the control is still wrong.

Where this is not backed by a number: the measured lift is in BUILD (completeness, freshness). Nine audit instruments across four designs all sit at +0.00 — recognition (snippets, cross-file repo, precision), procedure (does the model actually mutate the control and re-run the build), question-set (an unscoped "audit this repository", with defect classes outside the standard repertoire), and, since 2026-08-14, a real repository at a real vulnerable commit — the one design a synthetic fixture provably could not stand in for, because a planted defect is a deviation from its filler and agents find deviations without security reasoning. That last one is the strongest form of the test and it closed the question: across 16 real BOLA sites in Harbor v2.5.1 both arms recalled 15/16, and across 59 blinded findings both scored precision 1.00. A frontier model handed the code is already at ceiling — on synthetic code, on real code, and when you stop telling it what to look for. The audit half is justified by gap analysis and by real defects it found in this repo — not by a measured lift, and it is reported that way rather than implied. There will be no tenth accuracy instrument: only a different dependent variable (time-to-find, report usability, reach for a non-expert) is still untested. Every null, the retraction, and the pre-registered predictions that were wrong →

How the numbers are kept honest

The measurement discipline is the part that is hard to copy, so it is worth stating plainly. Every item below is in the repo, not a claim about it:

• Nulls are published, not buried. Nine +0.00 rows sit on the scoreboard next to the +0.39 — including the ones that say, in our own words, that the audit half of this library adds nothing a good model doesn't already do. That null is why the defect-avoidance result above matters: we went looking for value where our own measurements said there wasn't any, and reported both. (This line read "seven" until 2026-08-21 — it was understating the count.)
• A lift was retracted. An early +0.07 on inert-control detection did not reproduce when the sample grew from 15 to 49 cases. It was withdrawn and the retraction is documented rather than quietly dropped.
• The gates are proven able to fail. scripts/check-negative-controls.sh runs in CI as its own job: it injects a known-bad per invariant into a disposable git worktree and requires the intended check to be the one that complains — a non-zero exit for any other reason is reported as a false pass, not a catch. A passing gate proves the tree is clean; only this proves the gate still works.
• Predictions are pre-registered. Before the 2026-07-30 audit experiments ran, the expected numbers and ranges were committed and pushed (PRE-REGISTRATION.md). Both predictions turned out wrong — one by three times its own lower bound — and are reported as wrong.
• The scorers are themselves tested. A mutation probe once replaced a scoring function with return 1.0 and nothing noticed; the golden tests that now run in CI were watched to fail against that exact mutation first.
• The eval runners carry a duration baseline. Each run records to a local ledger and the next run of the same runner prints the delta — [run-completeness elapsed 12.3s over 7 cases | previous 380.0s — 30.9x faster] — because "finished far faster than the work allows" is a comparison, and a duration without its denominator says nothing. A swing over 5× is flagged for a human; nothing is gated on time. That ledger was itself a shared sink until 2026-09-02 — --selftest runs and aborts landed in it in the same shape as measurements (46 of 60 rows), which had quietly disarmed the comparison for the most-run runner. Runs now mark completion, and only a completed run can be a baseline.
• The library is applied to itself, and it finds things. Its own dead-path rules caught this repo's CI gates passing over zero files — green, exit 0, examining nothing — and the fix was verified by re-running the mutation. The same lens later caught this repo's secret scan: gitleaks prints 179 commits scanned, and nobody read it. In a shallow clone it scans 1 of 179, prints no leaks found, and exits 0 — a green scan over 0.5% of history, with one CI setting the only thing preventing it. CI now asserts the scope. Two more, both 2026-09-04. Two releases had shipped and were never tagged — a CHANGELOG entry, a version bump, a merge, and no tag: unreachable by git tag, and invisible because the one version gate checked only that a tag is never ahead of VERSION. And an eval runner had been a dead runner for six days — its ablation target moved to another file in a split, so its guard refused every run. The guard was right; what failed is that the smoke harness rendered it as a bland ok, the same string it prints for a runner that merely wants arguments.

The point is not that every number is flattering. It is that you can tell which ones are load-bearing, because the ones that aren't are labelled.

Skills