CI Documentation Latest Release Python License Last Commit Stars

Deploy on Railway

DeltaLLM

DeltaLLM is a self-hosted LLM gateway. Point OpenAI-compatible clients at one endpoint, then manage model deployments, routing, scoped API keys, budgets, guardrails, MCP tools, and usage from one control plane.

One Endpoint For Your Apps

from openai import OpenAI

# Before: call OpenAI directly.
client = OpenAI(api_key="sk-...")

# After: send the same OpenAI-compatible requests through DeltaLLM.
client = OpenAI(
    base_url="http://localhost:4002/v1",
    api_key="sk-deltallm-key",
)

Your application keeps its OpenAI request format. DeltaLLM handles provider credentials, routing, policy checks, caching, failover, and spend tracking.

What It Handles

  • Unified API - Use one OpenAI-compatible endpoint across OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Groq, and other providers.
  • Scoped API keys - Issue virtual keys with model allowlists, rate limits, budgets, owners, and expiry.
  • Routing and failover - Route by strategy, retry failed deployments, and separate provider credentials from application code.
  • Batch API - Run embeddings and non-streaming chat completions asynchronously through OpenAI-compatible files and batches, even when upstream providers are synchronous.
  • MCP gateway - Register external MCP servers and expose approved tools through controlled gateway flows.
  • Guardrails - Detect PII and prompt injection before provider calls.
  • Spend and usage - Attribute cost by key, team, organization, model, and provider.
  • Admin UI - Manage models, credentials, route groups, teams, users, usage, audit logs, and settings in the browser.
  • Operations - Export Prometheus metrics, request logs, audit events, cache behavior, and health checks.

Admin UI

DeltaLLM model deployments

The UI is the control plane for model deployments, route groups, API keys, access, usage, guardrails, audit logs, and runtime settings.

Quick Start With Docker

Use Docker Compose for the shortest local evaluation path. It starts DeltaLLM, PostgreSQL, and Redis.

git clone https://github.com/deltawi/deltallm.git
cd deltallm
cp config.example.yaml config.yaml

For a first local request, enable one-time model bootstrap in config.yaml:

general_settings:
  model_deployment_source: db_only
  model_deployment_bootstrap_from_config: true

Generate the required secrets:

python3 -c 'import secrets; print("DELTALLM_MASTER_KEY=sk-" + secrets.token_hex(20) + "A1")'
python3 -c 'import secrets; print("DELTALLM_SALT_KEY=" + secrets.token_hex(32))'

Create .env with the generated values and a provider key:

DELTALLM_MASTER_KEY=sk-your-generated-master-key
DELTALLM_SALT_KEY=your-generated-salt-key
OPENAI_API_KEY=sk-your-openai-key
[email protected]
PLATFORM_BOOTSTRAP_ADMIN_PASSWORD=ChangeMe123!

Start the stack:

docker compose --profile single up -d --build

Check health and send a request:

curl http://localhost:4002/health/liveliness

curl http://localhost:4002/v1/chat/completions \
  -H "Authorization: Bearer $DELTALLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Hello from DeltaLLM"}
    ]
  }'

Open the Admin UI at http://localhost:4002.

Full Docker setup, environment variables, HA Compose, and Dockerfile usage are covered in the Docker guide.

Deployment Paths

Path Use it for Start here
Railway Managed evaluation with hosted PostgreSQL and Redis Railway deployment
Docker Compose Local evaluation, demos, small teams, simple self-hosting Docker guide
Kubernetes Multi-instance production, ingress, autoscaling, and managed infrastructure Kubernetes guide
Local development Backend, UI, and contributor workflow from the repository Installation guide

Documentation

Contributing

License

See LICENSE.