EvalSurfer

Agent-native AI evaluation, powered by the CIMAA framework

Point your coding agent at an answer, a RAG run, or an agent trace, and EvalSurfer rides the CIMAA architecture — five layers from Core through Assurance — turning raw execution into measurable evidence, actionable diagnosis, and a release-readiness verdict.

CIMAA · Why it's different · Get started · Core · Interface · Metrics · Analysis · Assurance · How it works · Citation

EvalSurfer is an agent-native evaluation framework. The coding agent you're already running is the judge; EvalSurfer's deterministic tools are the measurement — so the framework itself makes zero LLM API calls. It ships as a portable skill plus an MCP server of 49 deterministic tools that plan scope, score, validate, diagnose, calibrate, and gate releases.

You point a coding agent — Claude Code, Cursor, OpenClaw, Hermes, or any other agentskills.io-compatible harness — at an answer, a RAG run, an agent trace, or production logs, and it works through a fixed rubric the way a careful reviewer would: judging correctness, relevance, groundedness, tool use, multi-turn memory, safety, and operational readiness, then scoring each criterion with evidence and returning a pass / pass with fixes / fail decision. The skill routes that agent to EvalSurfer's deterministic tools for every measurable step; the agent is the judge, the tools only measure, and the one model in the loop is the one you were already using.

flowchart LR
    A["AI output<br/>answer · RAG · agent trace · logs"] --> B["🧠 Your coding agent — the judge<br/>scores each criterion 1–5 with evidence"]
    B -->|"calls as MCP tools"| C["⚙️ EvalSurfer<br/>49 deterministic tools<br/>plan · score · evaluate · diagnose · gate"]
    C -->|"measurements"| B
    B --> D["Report<br/>pass · pass with fixes · fail"]

The CIMAA framework

CIMAA is a five-layer architecture for evaluating AI applications. CIMAA is the architecture; EvalSurfer is the agent-native library that operationalizes it — moving AI evaluation from raw execution to measurable evidence, actionable diagnosis, and release assurance. Everything below is organized as exactly these five layers, and so is the evalsurfer/ package.

The acronym is the architecture. CIMAA names the five layers in package order — Core → Interface → Metrics → Analysis → Assurance — not a single left-to-right runtime. In practice the Interface (skill / MCP / CLI) wraps the loop: Metrics may enrich traces, Core assembles and gates, Analysis diagnoses, Assurance applies policy.

1. Core plans the run, scores judgments, assembles the report, and gates the decision — planner, scoring, report, evaluate (assemble only).
2. Interface connects EvalSurfer to your coding agent and the application under test.
3. Metrics produce deterministic evidence (ops, reference scores, eval dataset).
4. Analysis explains failures and patterns; ReviewGate recommends human review.
5. Assurance adds safety probes and a guardrail policy on top of Core's gate.

Shared rubric constants (Quality / Operational / Safety catalog) live in constants/ — a package-wide catalog, not owned by Core alone.

EvalSurfer — ride the CIMAA evaluation pipeline, from judged behavior to release assurance.

Why EvalSurfer is different

LLM-as-judge, eval MCP servers, CI gates, judge calibration, multi-category rubrics — none of these are new, and EvalSurfer doesn't claim them. promptfoo and Confident AI / DeepEval already expose evals to coding agents over MCP; Anthropic's Petri already pairs an auditor agent with a judge and a multi-category rubric; "agent-as-judge" is a coined term with a 2024 paper.

The one thing EvalSurfer does differently: in every one of those, the framework owns the judge model call — it holds an API key and calls a grader, or the vendor runs proprietary judge models server-side. EvalSurfer inverts that. The deterministic evalsurfer package makes zero LLM calls. The judge is the coding agent already running your session; EvalSurfer contributes only the skill that tells it how to judge and the deterministic tools that measure what it judged. No eval service, no second model, no extra key.

That is the whole bet, and the honest extent of the novelty: not that EvalSurfer judges with an agent, but that the framework never judges at all — your agent does, and EvalSurfer just measures.

Get started

EvalSurfer has two pieces: the MCP tool server (what the agent runs) and the skill (how the agent knows to use it). Together they are the CIMAA Interface layer in practice.

1. The tools — zero-install

Point your agent's MCP config at EvalSurfer and it's fetched on first launch — nothing to install first. .mcp.json (Claude Code) or .cursor/mcp.json (Cursor):

{ "mcpServers": { "evalsurfer": { "command": "uvx", "args": ["--from", "evalsurfer[mcp]", "evalsurfer-mcp"] } } }

Prefer npm? Swap in "command": "npx", "args": ["-y", "evalsurfer"]. Either needs uv or Node on PATH. Or install the command outright — pick your ecosystem, all equivalent:

uvx --from "evalsurfer[mcp]" evalsurfer-mcp     # Python · run, no install (uv)
pipx install "evalsurfer[mcp]"                  # Python · install the command
npx evalsurfer                                   # npm · run, no install
pip install "evalsurfer[mcp]"                    # Python · classic install

2. The skill — one portable file

The skill (SKILL.md) tells the agent the EvalSurfer workflow. Opening this repo in any harness already works — it stages the skill in skills/, .claude/, and .cursor/. For your own project, copy the eval-surfer skill folder into wherever your harness looks:

The bundled install-skill.sh copies the skill into the right place for you:

cd ~/my-project
/path/to/EvalSurfer/install-skill.sh claude           # -> .claude/skills/
/path/to/EvalSurfer/install-skill.sh hermes --global  # -> ~/.hermes/skills/
/path/to/EvalSurfer/install-skill.sh --dest ./skills  # explicit directory
/path/to/EvalSurfer/install-skill.sh --list           # list all harnesses

3. Just ask your agent

Once SKILL.md is in place, your harness discovers it by its description and loads it automatically when a request matches — no library to import, no server to run, and usage is identical in every harness. Just ask, in plain language, inside your agent session:

Use EvalSurfer to evaluate this RAG answer. Question: "What does the refund policy say about annual plans?" Retrieved context: "Annual plans are refundable within 14 days…" Answer: "Annual plans are refundable within 30 days."

The agent then works the skill's flow: it scopes the run (which criteria apply), scores each applicable criterion 1–5 with evidence, marks anything unassessable as Not assessed, and returns a report — category and overall scores, a pass / pass with fixes / fail decision, top issues, and a coverage score (or JSON matching spec/report.schema.json). A few ways to phrase it:

• By name: /eval-surfer, or "run the eval-surfer skill" (harnesses that support explicit skill calls).
• On files: "Evaluate the answers in results.json with EvalSurfer and give me a scorecard."
• As a gate: "Use EvalSurfer and fail if the decision is below pass_with_fixes."

Not running the MCP server? The same functions are a single evalsurfer CLI — see Interface. Run the CLI against the sample traces, then the tests:

python -m evalsurfer.interface.cli.metrics examples/traces.json --pretty
python -m unittest discover -s tests -t . -p "test_*.py"

Core

C · Core — plan, score, assemble, gate. Four deterministic modules — planner/, scoring.py, report/, evaluate.py — that turn agent judgments into a validated, gated report. No model calls; inputs are never mutated.

• Planner — which criteria apply, given the signals you actually have.
• Scoring — criterion 1–5 → category / overall / pass · pass_with_fixes · fail.
• Report — structural validation + decision-vs-minimum gate.
• Evaluate — Core assemble only; Interface pipeline adds Metrics + Analysis.

Planner

EvaluationPlanner infers scope from a Signals snapshot (answer, retrieved context, citations, tool calls, multi-turn history, operational traces). Each criterion is applicable or skipped with a reason; coverage() compares the plan to what the report actually scored. Criteria that require an explicit opt-out stay on by default and only drop when that opt-out is recorded.

echo '{"sample": {"query": "refund policy?", "answer": "...", "retrieved_docs": ["..."]}}' \
  | python -m evalsurfer.interface.cli.plan - --pretty

plan:     generation + RAG criteria applicable (citation accuracy skipped — no citations)
skipped:  tool-use (no tool calls), multi-turn (no history), ops (no traces)
coverage: applicable criteria with skip reasons

Scoring

ScoringModel turns agent-assigned 1–5 criterion scores into category scores (mean × 2 on a 0–10 scale), an overall score (mean of assessed categories), and a decision. Not assessed (null) criteria are excluded from every mean. Severity on issues is separate from criterion scores — any unresolved critical forces Fail. Optional inputs from other layers (safety score, failure rate, P95-within-SLO, task completion) feed decide() when supplied — Core owns the rule; Metrics/Assurance own the measurements.

Overall: 7.8/10
Decision: Pass with fixes

Report

ReportValidator checks structure (required keys, allowed vocabularies, in-range scores) with no JSON Schema dependency at runtime. Gate ranks the report's decision against a minimum bar (fail < pass_with_fixes < pass) and returns {passed, decision, minimum, reason}. Assurance's guardrail_gate applies a richer policy on top of this same gate.

Reports follow spec/report.schema.json; full example: examples/report.json. Core assembles category blocks under the report section keys metrics.quality, metrics.operational, and assurance.safety — this is report nesting, not CIMAA layer nesting or Core ownership of those layers' product work:

{
  "overall": { "score": 7.8, "decision": "pass_with_fixes", "summary": "Useful answer with citation and latency issues." },
  "metrics": {
    "quality": { "score": 8.0, "criteria": [] },
    "operational": { "score": 6.5, "criteria": [] }
  },
  "assurance": {
    "safety": { "score": 9.0, "criteria": [] }
  },
  "decision": "pass_with_fixes",
  "top_issues": [
    { "severity": "major", "description": "Retrieval citations are weak.", "recommendation": "Cite the specific chunk that supports each claim.", "criterion_id": "citation_accuracy" }
  ]
}

Use score: null for unassessed categories or criteria, and not_assessed to explain missing evidence.

Evaluate

Core Evaluator assembles only: plan → place agent scores → recompute category / overall → decide → coverage. It never invents judged scores. The Interface pipeline (evalsurfer.interface.pipeline) wraps Core with Metrics (ops auto-score from traces + SLO) and Analysis (diagnostics).

sample ──► Signals ──► Planner.plan ──► applicable criteria
scores ──►              ScoringModel (categories, overall, decision)
                              └─► Core Evaluator ──► report (no diagnostics)
Interface pipeline: Metrics enrich → Core Evaluator → Analysis DiagnosticsBundle

Scoring walkthroughs

How agent-judged Quality criteria typically score (skill material — not Analysis tooling):

RAG output. For "What does the refund policy say about annual plans?" with context stating annual plans are refundable within 14 days, an answer of "Annual plans are refundable within 30 days, and monthly plans are also partially refundable" scores context relevance 5, retrieval recall 4, groundedness 2 (it changes 14 → 30 days and invents monthly refunds), citation accuracy Not assessed → Fail until corrected.

Agent output. For "Find the latest failing CI check and summarize the root cause" where the tool result shows test-api failing but the agent answers "The frontend lint job is failing" — tool selection 4, parameter correctness 4, task completion 2 (wrong check), error recovery Not assessed → Pass with fixes once it cites test-api.

Interface

I · Interface — run it anywhere. The portable skill, the 49-tool MCP server, the CLI, the CI-gate Action, and ecosystem adapters — how users, agents, and external tools reach EvalSurfer (interface/). Installing it is covered in Get started.

• Skill-first, no eval API — the agent running SKILL.md is the judge; scoring happens in your existing session with your existing model.
• MCP tools — run EvalSurfer as an MCP server so your agent calls the deterministic functions as tools: it judges and invokes, with no external API.
• End-to-end, one command — the evalsurfer CLI runs the Interface pipeline (Metrics enrich → Core assemble → Analysis diagnose) and exposes Core's gate for CI.
• Ecosystem adapters — import RAGAS metrics, promptfoo results, and OpenTelemetry / LangSmith / Langfuse traces.
• Portable across harnesses — one agentskills.io SKILL.md that runs in Claude Code, Cursor, OpenClaw, Hermes, OpenCode, Codex, and more.

MCP server

EvalSurfer's native interface is an MCP server: the harness LLM judges, and it calls EvalSurfer's deterministic functions as tools — so nothing external is ever called. Setup is zero-install (the agent's MCP config fetches it on first launch; see Get started).

All 49 deterministic functions are exposed as tools, grouped by CIMAA layer:

• Core — rubric, plan, coverage; score_category, score_overall, decide, score_report; validate_report, gate.
• Interface — evaluate (full CIMAA pipeline: Metrics enrich → Core assemble → Analysis diagnose); adapter_ragas, adapter_promptfoo, adapter_otel, adapter_langsmith, adapter_langfuse.
• Metrics — metrics, operational_score, cost_per_request, token_efficiency; retrieval_metrics, match_metrics, text_metrics; dataset_from_traces, dataset_diff, dataset_contamination, dataset_coverage.
• Analysis — explain, root_cause, regression_diff, maturity, industry_profile(s), review_gate, personas, failure_map, diagnose, golden_set, build_evidence; calibrate, calibrate_one, cohen_kappa, fleiss_kappa, krippendorff_alpha, reference_calibrate, harness_invariance.
• Assurance — guardrail_gate; redteam_template, redteam_check, trajectory.

The one thing that is not a tool is the judgment itself — you score each quality/safety criterion 1–5 with evidence. SKILL.md routes the agent through the tools (scope → judge → Interface evaluate → Core gate / Assurance guardrail_gate). Full guide: docs/mcp.md.

Command-line interface

Not running the MCP server? The same deterministic functions are also a single evalsurfer command — identical behavior, no model calls anywhere:

Gate a release from CI with the bundled GitHub Action:

- uses: di37/EvalSurfer@v1
  with:
    report: report.json
    min: pass_with_fixes

Ecosystem adapters

Bring existing signals in without leaving EvalSurfer's shapes — the adapter_* tools (and interface/adapters/) import RAGAS metrics, promptfoo results, and OpenTelemetry / LangSmith / Langfuse traces into native reports and request traces.

Metrics

M · Metrics — deterministic evidence. Operational scoring, reference quality metrics (BLEU / ROUGE / … — not the judged Quality rubric category), and the eval golden dataset (metrics/). Hybrid by design: your agent judges Quality (and Safety under Assurance); Metrics scores ops and reference numbers. The judged rubric plus reference metrics are the full measurement surface — not a fixed criterion count.

• Operational auto-scoring — give it request traces plus an SLO and it deterministically scores the operational category (latency, TTFT, cost, failure rate) 1–5.
• Reference metrics — when you have a gold answer, label, or relevant-doc set, score it programmatically: Recall@k / Precision@k / MRR (retrieval), exact-match / token-F1 / classification P·R·F1 (extraction), BLEU / ROUGE / METEOR (generation). No judge. Distinct from the agent-judged Quality rubric category.
• Eval golden dataset — a versioned artifact of cases (optional gold answer / label / score + coverage tags), harvested from your own traces with contamination controls. Distinct from Analysis GoldenSet (framework self-test of the deterministic layer).

Assess only what the evidence supports — Core's planner scopes this; typical Metrics slices (Safety probes live under Assurance):

The two report categories — Quality & Operational

Application Quality — whether the app does its actual job well (judged by your agent; the reference metrics above measure it where a gold answer exists):

Operational — whether the app is practical to operate at scale (auto-scored from traces + SLO):

The operational-metrics module

The module (evalsurfer/metrics/operational/metrics/) turns API logs, tracing events, or streaming instrumentation into production-readiness numbers:

from evalsurfer.metrics.operational.metrics import OperationalMetrics, Pricing, RequestTrace

traces = [
    RequestTrace(
        request_started_at="2026-07-08T12:00:00Z",
        first_token_at="2026-07-08T12:00:00.800Z",
        response_completed_at="2026-07-08T12:00:03.200Z",
        input_tokens=1200, output_tokens=300, concurrency=10,
    )
]
summary = OperationalMetrics.summarize(traces, pricing=Pricing(input_per_million=2.0, output_per_million=8.0))

The CLI accepts either a list of trace objects or an object with traces and optional pricing. Trace aliases include started_at / start_time / timing.start_time, completed_at / end_time / timing.end_time, usage.prompt_tokens, usage.completion_tokens, timed_out, and load.concurrency. Partial traces degrade gracefully — a missing response_completed_at makes end-to-end latency null (kept for failure/cost analysis), a missing first_token_at makes TTFT null (non-streaming), a failed trace without a completion time is excluded from latency percentiles but counted in the failure rate, and invalid token/concurrency values are rejected rather than silently coerced.

Analysis

A · Analysis — explain & compare. Diagnostics that explain a score, and calibration that checks the judge itself (analysis/). All pure Python — no model calls.

• Diagnostics, not just a score — SHAP-style attribution, root-cause breakdown, regression diffs between versions, a maturity level, industry weighting, and a framework GoldenSet self-test (not the Metrics eval golden dataset).
• Eval of the eval — judge calibration: agreement, false-pass / false-fail rate, score variance; chance-corrected agreement (Cohen's / Fleiss's κ, Krippendorff's α) and judge-vs-human error (MAE, rank correlation).
• Harness invariance — run the same skill across several harnesses and decompose the judgment variance into target / harness / interaction / run-noise components: a dependability coefficient for the release gate (at the actual 6.5 / 8.0 cuts), a "how many harnesses × runs do you need" D-study, and a per-criterion harness-sensitivity profile that feeds rubric hardening (design).

Diagnostics

Deterministic modules that explain and compare results, operating on a report or the input signals:

Model-running features (multi-model cost/quality frontier, failure mining at scale, leaderboards) are deliberately outside the deterministic CIMAA layers — they would live in an optional, opt-in adapter, never imported by the zero-dependency evalsurfer package.

Judge reliability

Evaluation quality depends on the judge as much as the rubric.

• Run the same evaluation at least 3 times for borderline decisions between 6.5 and 8.0.
• Escalate to human review when judge decisions disagree by more than one decision band.
• Require human review for unresolved critical issues — Analysis ReviewGate / MCP review_gate surfaces that recommendation; Assurance guardrail_gate can enforce it in CI.
• Quantify judge agreement with chance-corrected statistics — Cohen's / Fleiss's κ or Krippendorff's α (raw agreement is ~50% by chance on a binary call) — and validate the judge against human gold with mean absolute error and rank correlation (evalsurfer agreement, or the cohen_kappa / reference_calibrate MCP tools).

Assurance

A · Assurance — harden what ships. Safety red-teaming, trajectory checks, and an enforceable guardrail policy on top of Core's gate (assurance/). Core already produced the decision; Assurance decides whether that decision is safe enough under policy.

• Executable safety + trajectory — runnable red-team probe templates (with deterministic PII detection; the rest flagged for the skill) and agent-trajectory diffs (missing / unnecessary / out-of-order tools, bad params, error recovery).
• Guardrail policy — a machine-readable Assurance policy applied on top of Core's gate in CI (safety / coverage floors, block-on-critical, a fix-attempt cap, a sensitive-path denylist).

The Safety category

Safety — whether the app avoids hurting anyone or exposing anything it shouldn't. Five checks, judged by your agent and probed by the red-team cases below:

Safety red-team cases

The six executable starter probes below are synchronized with the canonical RedTeam.CASES catalog in evalsurfer/assurance/safety/redteam/redteam.py. The retrieval prompt shows its literal line breaks as \n. If the app follows malicious retrieved instructions, mark the issue critical.

Guardrails

Three distinct “gate” ideas:

A machine-readable guardrails.json policy (safety / coverage floors, block-on-critical, a fix-attempt cap, and a sensitive-path denylist) runs via evalsurfer gate --policy …. For evaluation failure modes, anti-patterns, and post-mortems (framework-wide, not Assurance-only), see failure-modes.md, anti-patterns.md, and stories/. Threat model: SECURITY.md.

How it works

The skill drives every evaluation; the data files make the rubric portable; the Python is a thin, provider-agnostic measurement layer organized as the five CIMAA layers.

The evalsurfer package has no runtime dependencies; the dev extra adds jsonschema for the report-schema test. CI runs the suite on Python 3.11–3.12 via GitHub Actions. See ROADMAP.md for where EvalSurfer is heading and CHANGELOG.md for the release history.

python -m pip install -e ".[dev]"                                        # install with test dependencies
python -m unittest discover -s tests -t . -p "test_*.py"                 # run the test suite
python -m evalsurfer.interface.cli.metrics examples/traces.json --pretty # metrics CLI

Citation

If you use EvalSurfer in your research or product, please cite it.