Claude Skills · Code Review & Testing
Eval Design Forensics
wanshuiyin/Anti-AutoresearchAudit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing...
At a glance
This skill is for Code Review & Testing and helps you audit evaluation design validity, detect train-test leakage in ml papers, and identify measurement validity problems.
git clone --depth 1 https://github.com/wanshuiyin/Anti-Autoresearch
cp -r Anti-Autoresearch/skills/eval-design-forensics ~/.claude/skills/eval-design-forensics
Setup, runtime and requirements describe wanshuiyin/Anti-Autoresearch, the repo this skill ships in.
Ml EvaluationExperimental DesignPaper ReviewLeakage DetectionValidityReproducibility
Also in wanshuiyin/Anti-Autoresearch
View the repoSynthesize the single strongest EVIDENCE-BOUND reviewer case to reject a paper, built ONLY from the evidence ledger (claims.json) + the othe...
Transparent, itemized impressions of AI-generated WRITING STYLE — the repo's ONLY non-integrity track. Two passes: a deterministic defensive...
End-to-end substantive-integrity forensic sweep of a research paper (especially autoresearch / AI-Scientist-style output). Orchestrates the...
Audit whether a paper's baseline comparisons are COMPLETE, FAIR, and SIGNIFICANT: a required recent SOTA baseline is missing while 'best/SOT...
Citation-integrity forensics: is every reference real, correctly attributed, and used in a context the cited work actually supports? Catches...
Flagship intra-paper self-consistency forensics: does the paper contradict ITSELF across abstract/intro/tables/body/appendix, and does the m...
Build the deterministic evidence ledger (artifact_manifest.json + claims.json) that every other Anti-Autoresearch auditor reads. One pass in...
Audit experiment integrity against the evidence ledger. At L2 (repo + result files present) a fresh cross-model reviewer reads the eval code...
MEMO-ONLY prior-work overlap advisory: surfaces the two ADVISORY taxonomy signals neither a tool nor a model can decide from the paper alone...
Checkable-ish surface presentation signals a reviewer notices first — duplicate/near-identical tables, leftover pipeline/template strings, t...
Family-G proof & derivation integrity forensics: does a THIRD PARTY's written proof/derivation actually establish its theorem, or does it sk...
Other Code Review & Testing skills
A relentless interview to sharpen a plan or design, which also creates docs (ADR's and glossary) as we go.
Use when you need to resolve an in-progress git merge/rebase conflict.
Create exercise directory structures with sections, problems, solutions, and explainers that pass linting. Use when user wants to scaffold e...
Migrate test files from `as` type assertions to @total-typescript/shoehorn. Use when user mentions shoehorn, wants to replace `as` in tests,...
Move issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.
A relentless interview that asks every frontier question at once, round by round.