Awesome LLM Observability

A current, hand-checked list of 60+ LLM observability tools, plus 26 agent skills you install in one command.

Star counts refresh themselves. The skills get validated in CI. More on both further down.

The tooling for watching LLM apps in production is a fast-moving mess of overlapping projects, and every list I found had stars from 2023 on it. So I kept my own, verified the entries, and wired up a job to keep the numbers honest. Use it to find the right tool without re-researching the whole space.

Legend: 🟢 open-source · 🔵 open-core / hybrid · 🟠 commercial (public repo is an SDK/client only - low star counts don't reflect the product). Star counts are pulled live from the GitHub API and auto-refreshed weekly by CI (tools/refresh_stars.py), so they stay current instead of rotting.

Contents

• What is LLM Observability?
• 🧰 Skills (Batteries Included)
• Platform Comparison (At a Glance)
• Tracing & Observability Platforms
• Evaluation Frameworks
• Prompt Management & Experimentation
• LLM Gateways & Proxies
• Instrumentation & Standards
• Guardrails & Safety Monitoring
• Self-Hosted / Open-Source First
• Research & Benchmarks
• Learning Resources
• Guides & Tools (Original)
• How to Choose
• Contributing

What is LLM Observability?

Regular observability assumes your system is deterministic. LLM apps aren't. They make things up, drift as inputs change, quietly burn tokens, and fail without ever throwing an error. So you end up watching different things:

• tracing: every step of a run, with tokens, latency, and cost per span
• evals: whether the output is actually any good, both offline and on live traffic
• monitoring: alerts when quality, cost, latency, or error rate move
• prompt and dataset management: version your prompts, and pull real production cases back into your tests

People call this LLMOps observability, AI observability, or just LLM monitoring. Same idea.

🧰 Skills (Batteries Included)

Most lists stop at links. This one also ships 26 agent skills, the SKILL.md kind, so your coding agent knows how to actually do this stuff instead of just knowing where the tools live. They fire on plain requests like "add tracing" or "reduce my LLM bill".

A handful (marked ⚙️) are backed by real Python with unit tests that run in CI. The rest ship trigger cases and get checked by skillvitals, a small tool I built to measure whether a skill actually fires when it should. A free heuristic runs on every push as a regression gate.

Graded on a cheap model (llama-3.1-8b via NVIDIA NIM), the 26 skills average a trigger F1 of 0.99 (98.7% recall, 0% false-fire), and that holds even against adjacent negatives, prompts about a different LLM task the skill should refuse to fire on. It's not a clean sweep though. choose-observability-stack came in at 0.80 because the model kept confusing a "compare Langfuse vs Phoenix" prompt with the model-comparison skill. That's exactly the kind of overlap this is meant to catch. Run it yourself:

skillvitals trigger skills/reduce-llm-cost/SKILL.md \
  --cases validation/cases/reduce-llm-cost.yaml \
  --model meta/llama-3.1-8b-instruct --base-url https://integrate.api.nvidia.com/v1

Install (about 10 seconds)

As a Claude Code plugin (easiest):

/plugin marketplace add ContextJet-ai/awesome-llm-observability
/plugin install llm-observability@contextjet

Or copy the skills into any agent's skills folder:

npx degit ContextJet-ai/awesome-llm-observability/skills ~/.claude/skills

Or clone and run the installer:

git clone https://github.com/ContextJet-ai/awesome-llm-observability
./awesome-llm-observability/install.sh          # installs into ~/.claude/skills

The skills

Also original in this repo:

• 📖 Guide - LLM Observability for Finance & Regulated Industries
• 🛠️ Tool - genai_trace.py: a minimal, vendor-neutral OpenTelemetry GenAI tracer with a built-in PII-redaction hook (no-op if OTel isn't installed).

→ Browse skills/, guides/, tools/. Contributions welcome.

Platform Comparison (At a Glance)

The question everyone asks first is "Langfuse or Phoenix or LangSmith or Opik or Helicone?" Pick by the constraints you can't bend (self-hosting, license), then worry about features. Double-check specifics against the docs, this table moves.

✅ first-class · ➖ limited/not the focus. Need help deciding? Use the choose-observability-stack skill.

Tracing & Observability Platforms

End-to-end tracing + dashboards for LLM/RAG/agent apps.

Evaluation Frameworks

Test and score LLM/agent output. One thing to sort out before you pick a tool: are you doing offline/batch evals (run over a dataset in CI to optimize a prompt) or inline evals (score a response live, to gate or block it before it reaches the user)? They're pretty different jobs. Most of the frameworks below are built for the offline case. Live gating usually lands in the guardrails tools or a platform's online-scoring feature.

Prompt Management & Experimentation

Langfuse and Agenta also provide strong prompt management - listed once under their best-fit category.

LLM Gateways & Proxies

Route to many providers through one endpoint; get logging, cost tracking & caching for free.

Instrumentation & Standards (OpenTelemetry GenAI)

The observability layer is standardizing on OpenTelemetry - emit these and your traces export to any OTel backend.

Guardrails & Safety Monitoring

Self-Hosted / Open-Source First

Need data residency / on-prem? These ship a genuinely self-hostable OSS core: Langfuse (MIT) · Arize Phoenix · Comet Opik (Apache-2.0) · Helicone (Apache-2.0) · OpenLIT (Apache-2.0) · Langtrace (AGPL) · LangWatch (Apache-2.0) · MLflow (Apache-2.0) · Evidently (Apache-2.0) · SigNoz (open-core, OTel-native).

Research & Benchmarks

The tooling above is grounded in a fast-moving literature. Key anchors:

• LLM-as-a-Judge - Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685) - the basis for scalable, model-graded evaluation used by most eval tools.
• HELM - Liang et al., Holistic Evaluation of Language Models (arXiv:2211.09110) - canonical multi-metric evaluation.
• SelfCheckGPT - Manakul et al., 2023 (arXiv:2303.08896) - reference-free hallucination detection.
• Agent hallucination - MIRAGE-Bench (arXiv:2507.21017); paper list: EdinburghNLP/awesome-hallucination-detection.
• OTel GenAI semantic conventions - the observability standard (see Instrumentation & Standards).

Learning Resources

• OpenTelemetry GenAI semantic conventions - the emerging standard for LLM telemetry.
• Arize Phoenix docs & Langfuse docs - practical tracing/eval concepts.
• Hamel Husain - Your AI Product Needs Evals.
• Chip Huyen - Building LLM applications for production.
• Loop & Retry - Measuring agent failure in production - which signals actually surface a broken agent in prod, and why request-level success rates hide the failures that matter.

Guides & Tools (Original)

Original work in this repo, not just curated links:

• 🧰 Agent Skills - 26 ready-to-run workflows (instrument, evals, debug-from-traces, guardrails, PII-safe tracing, tool selection).
• 📖 LLM Observability for Finance & Regulated Industries - a guide for teams where observability is a compliance control, not a dashboard.
• 🛠️ - a minimal vendor-neutral OpenTelemetry GenAI tracer with a PII-redaction hook.

How to Choose

• Just need traces + cost, minimal code? Start with a gateway/proxy (change base URL, zero SDK) or a tracing platform with OTel auto-instrumentation.
• Need to test quality, not just watch it? Add an evaluation framework (offline eval in CI + online LLM-as-a-judge).
• Data residency / on-prem required? Pick from Self-Hosted.
• Want vendor-neutral, future-proof traces? Emit OTel GenAI semantic conventions so you can swap backends.

A common production setup pairs a gateway (cost + routing) with an evaluation tool (quality) on top of an OTel-native tracing backbone.

Contributing

Contributions welcome - see CONTRIBUTING.md. Add a tool via PR with: name, link, one-line neutral description, correct category, and the OSS/commercial marker. Keep it factual and non-promotional - a wrong entry or dead link discredits the whole list.

License

To the extent possible under law, the contributors have waived all copyright and related rights to this work.