proofrag

Point your agent at your docs and your RAG app. Get a golden test set, an LLM-as-judge + retrieval scorecard, and a CI gate — in one command.

Evaluation is the #1 unmet pain in production RAG/LLM work, and the hardest part is building a good test set in the first place. proofrag generates one from your own corpus, judges your system on it, and emits a shareable HTML scorecard. It's an Agent Skill (works in Claude Code, Codex, Cursor) and a plain Python CLI — wrapping the eval loop, not reinventing the metrics.

pipx install "proofrag[anthropic]"        # or: pip install / uv tool install / uvx
proofrag demo --out scorecard.html && open scorecard.html

Use [openai] instead of [anthropic] for an OpenAI-compatible or local (Ollama) backend. No install? Run it ad-hoc: uvx "proofrag[anthropic]" demo.

Install as an Agent Skill

proofrag is a skill (the agentskills.io open standard) backed by a real CLI — so any agent can run "evaluate my RAG" and get a reproducible scorecard.

Claude Code (plugin):

/plugin marketplace add unshDee/proofrag
/plugin install proofrag@proofrag

Then ask "evaluate my RAG" (auto-triggered) or type /proofrag.

Claude Code (manual) — cp -r skills/proofrag ~/.claude/skills/ Codex / other agents — cp -r skills/proofrag .agents/skills/

The skill drives the proofrag CLI; install it with uv tool install "proofrag[anthropic]" (or pipx install, or run ad-hoc via uvx). See AGENTS.md for details.

Why this exists

"Running evals aren't the problem — the problem is acquiring or building a high-quality, non-contaminated dataset."

Most RAG systems reach production with no evals because writing a balanced golden set by hand is tedious. So teams ship prompt and model changes blind. This closes that loop: change something → re-run → see if quality moved → gate the merge.

The loop

# 1. Generate a golden set from YOUR docs (questions + gold answers + gold contexts)
proofrag generate --corpus ./docs --out goldenset.jsonl --n 20

# 2. Validate it before committing it
proofrag validate --goldenset goldenset.jsonl --corpus ./docs --out validation.json

# 3. Run your RAG over each question -> predictions.jsonl
proofrag run --goldenset goldenset.jsonl --endpoint http://localhost:8000/ask --out predictions.jsonl
# or: proofrag run --goldenset goldenset.jsonl --callable myapp.rag:answer --out predictions.jsonl

# 4. Judge: groundedness, correctness, completeness, context attribution + retrieval metrics
proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl --out results.json

# 5. Shareable HTML scorecard
proofrag report --results results.json --out scorecard.html

# Optional: Markdown summary for CI logs / job summaries
proofrag summary --results results.json

Run the whole thing end-to-end against the bundled example:

uv sync --extra anthropic && export ANTHROPIC_API_KEY=...
uv run proofrag generate --corpus examples/docs-rag/corpus --out goldenset.jsonl --n 8
uv run proofrag validate --goldenset goldenset.jsonl --corpus examples/docs-rag/corpus
uv run python examples/docs-rag/naive_rag.py --goldenset goldenset.jsonl --corpus examples/docs-rag/corpus --out predictions.jsonl
uv run proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl --out results.json
uv run proofrag report --results results.json --out scorecard.html

Corpus loading

Before generating a golden set, inspect what proofrag will actually read:

proofrag corpus ./docs
proofrag corpus ./docs --include "**/*.md" --exclude "drafts/**"

Corpus loading skips noisy directories by default (.git, .venv, node_modules, dist, build, caches) and honors .gitignore patterns. Use --no-gitignore to disable .gitignore filtering. The same --include, --exclude, --no-gitignore, and --chunk-chars flags work on proofrag generate.

Supported inputs include Markdown, plain text, reStructuredText, MDX, common code files, and HTML. PDF loading is optional:

pip install "proofrag[pdf]"
proofrag corpus ./docs

Generated golden sets include context_metadata for each gold context, preserving source path, chunk id, chunk index, character count, and extension.

Golden set validation

Generated eval sets should be reviewed before they become a committed baseline. proofrag validate checks the JSONL schema, duplicate ids/questions, answerable cases without gold contexts, unanswerable cases that still cite context, difficulty tiers, source coverage, and a stable file fingerprint:

proofrag validate --goldenset goldenset.jsonl --corpus ./docs --out validation.json

It exits non-zero on hard errors. Add --strict to fail on warnings too when you want CI to enforce review hygiene.

Generated unanswerable questions are candidates, not proof of absence: search the full corpus before accepting them. For multi_doc, confirm both distinct sources are actually required. The validator catches structural problems; human review owns meaning.

Prediction adapters

The only app-specific step is producing predictions.jsonl. You can still write your own driver, but most projects can start with proofrag run:

# HTTP: proofrag POSTs {"id": "...", "question": "..."}
proofrag run --goldenset goldenset.jsonl \
  --endpoint http://localhost:8000/ask \
  --header "Authorization: Bearer $TOKEN" \
  --out predictions.jsonl

# Python: calls myapp.rag.answer(question)
proofrag run --goldenset goldenset.jsonl \
  --callable myapp.rag:answer \
  --out predictions.jsonl

# Python record mode: calls myapp.rag.answer(full_golden_record)
proofrag run --goldenset goldenset.jsonl \
  --callable myapp.rag:answer --call-style record \
  --out predictions.jsonl

Adapters may return an answer string, a tuple like (answer, contexts), or a dict like {"answer": "...", "retrieved_contexts": ["...", "..."]}. The endpoint form accepts the same JSON response shape. See examples/docs-rag/naive_rag.py for a fully custom driver.

CI gate

Two kinds of gate. An absolute floor:

proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl \
  --out results.json --fail-under 0.7      # non-zero exit if overall score drops below 0.7

…and a regression gate against a committed baseline (a known-good results.json):

proofrag diff --baseline baseline.json --candidate results.json --tolerance 0.02
# prints a per-metric delta table; exits 1 if any metric dropped > tolerance.
# Refuses incompatible datasets, backends, k values, matchers, or metric schemas.
# Different judge models also require an explicit --allow-judge-mismatch override.

GitHub Action

Drop proofrag into any repo's CI in a few lines — it installs the CLI, evaluates, writes the scorecard, adds a GitHub Actions job summary, uploads the scorecard and results as an artifact, and gates on both the floor and the baseline:

- uses: unshDee/[email protected]
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  with:
    goldenset: eval/goldenset.jsonl
    predictions: predictions.jsonl     # produced by your RAG earlier in the job
    baseline: eval/baseline.json        # optional regression gate
    fail-under: "0.7"                   # optional absolute gate

Full runnable workflow: examples/ci/proofrag-eval.yml.

The artifact and job summary are on by default. Disable them with upload-artifact: "false" or summary: "false" if your workflow handles those separately.

A/B: compare two RAG variants

Vector vs GraphRAG? Two prompts? Two models? Run both over the same golden set, then let the same judge pick the better answer per question — blind (answers shown in randomized order, so position bias is shuffled out):

proofrag compare --goldenset goldenset.jsonl \
  --a vector_preds.jsonl  --a-name vector \
  --b graphrag_preds.jsonl --b-name graphrag \
  --out comparison.json --html comparison.html

Deterministic retrieval metrics for each variant sit beside the verdict, so you can tell whether a win came from better retrieval or better generation.

Case studies

Three reproducible studies show how Proofrag separates retrieval changes from answer quality. Each uses a hash-checked official corpus, a reviewed golden set, retained raw artifacts, blind A/B judging, and an explicit limitations section.

In the Python study, FTS5 won 13 blind comparisons, token overlap won 6, and 11 tied. The largest retrieval difference appeared on multi-document questions—not the overall average. In the RFC study, section metadata raised exact NDCG@5 by only 0.018—below the predeclared 0.05 threshold—despite winning 4 of 5 decided comparisons. In the OWASP study, top-6 raised exact Recall@6 by 0.024 but reduced judged answer quality; top-3 won the blind comparison 7–5, with 12 ties. The negative results are retained rather than presented as universal optimization claims.

What makes it different

• Golden set from your corpus — the wedge. Difficulty tiers: single-doc, multi-doc, and unanswerable (so you catch hallucination-instead-of-refusal).
• Golden set validation — schema checks, duplicate detection, source coverage, and a stable fingerprint help teams review generated evals before committing them.
• Retriever vs generator split — rank-aware retrieval metrics (Recall@k, Precision@k, NDCG@k, MRR) separate "the context never arrived / ranked too low" from "the model fluffed it." Jaccard overlap is the default; use --exact when predictions return original chunks, or --semantic for embedding match.
• Pinned, fingerprinted evaluation — scorecards record judge, prompt version, matcher, cutoff, and golden-set fingerprint; diff rejects incompatible runs.
• Cheap & portable — defaults to a small model; Anthropic, OpenAI, or local/Ollama (OPENAI_BASE_URL). Self-contained HTML, zero JS, zero external assets.
• Prediction adapters — proofrag run can call an HTTP endpoint or Python callable so teams do not need to hand-write predictions.jsonl glue on day one.
• CI-native output — the GitHub Action writes a markdown job summary and uploads the HTML scorecard/results artifact automatically, including when a gate fails.
• Agent-native — drop it in as a skill and say "evaluate my RAG"; the agent wires your pipeline to the kit.
• Pluggable scoring backends — swap proofrag's own judge for DeepEval or Ragas without changing the workflow, scorecard, CI gate, or A/B flow.

Scoring backends

By default proofrag judges generation with its own pinned LLM-as-judge. You can swap in an external library instead — the retrieval metrics, scorecard, diff, and compare all stay the same; only the generation metrics change.

pip install "proofrag[deepeval]"
proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl \
  --backend deepeval --out results.json
# generation metrics become: faithfulness, answer_relevancy, correctness (GEval)

pip install "proofrag[ragas]"
proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl \
  --backend ragas --out results.json
# generation metrics become: faithfulness, factual_correctness
# plus answer_relevancy when OpenAI-compatible embeddings are configured

The DeepEval judge uses the same model config as proofrag (ANTHROPIC_API_KEY → AnthropicModel, OPENAI_API_KEY → GPTModel). Verified against deepeval 4.0.6. Metric reasons are preserved in the scorecard's weakest-case notes when DeepEval provides them.

The Ragas backend is verified against ragas 0.4.3. It uses proofrag's configured LLM provider for faithfulness and factual correctness. Ragas answer relevancy needs embeddings, so it is enabled when OPENAI_API_KEY or OPENAI_BASE_URL is set.

Providers

proofrag is provider-agnostic. Set one of these and everything — generate, judge, compare, and the DeepEval/Ragas backends — uses it:

If both API keys are present, Anthropic wins auto-detection. Set PROOFRAG_PROVIDER=openai to choose OpenAI explicitly. A local .env is not loaded automatically; run set -a && source .env && set +a first.

--semantic retrieval matching uses embeddings, which only exist on the OpenAI-compatible path (Anthropic has no embeddings API), so it needs OPENAI_API_KEY or OPENAI_BASE_URL even when your judge is Anthropic.

Environment

Contributing

Issues and PRs welcome — see CONTRIBUTING.md. MIT licensed.