Features

MCP Server Aggregation: Connect multiple MCP servers (STDIO, SSE, HTTP) through a single endpoint
LLM Provider Routing: Unified interface for OpenAI, Anthropic, Google, and AWS Bedrock LLM providers with full tool calling support
Context-Aware Tool Search: Intelligent fuzzy search across all connected tools using natural language queries
Protocol Support: Supports STDIO (subprocess), SSE (Server-Sent Events), and streamable HTTP MCP servers
Flexible Configuration: TOML-based configuration with environment variable substitution
Security: Built-in CORS, CSRF protection, OAuth2, and TLS support
Rate Limiting: Multi-level rate limiting with in-memory or Redis backends
Docker Ready: Available as a container image with minimal configuration needed

Installation

Quick Install (Linux/Windows (WSL)/macOS)

curl -fsSL https://nexusrouter.com/install | bash

Docker

Pull the latest image:

docker pull ghcr.io/grafbase/nexus:latest

Or use the stable version:

docker pull ghcr.io/grafbase/nexus:stable

Or use a specific version:

docker pull ghcr.io/grafbase/nexus:X.Y.Z

Build from Source

git clone https://github.com/grafbase/nexus
cd nexus
cargo build --release

Running Nexus

Using the Binary

nexus

Using Docker

docker run -p 8000:8000 -v /path/to/config:/etc/nexus.toml ghcr.io/grafbase/nexus:latest

Docker Compose Example

services:
  nexus:
    image: ghcr.io/grafbase/nexus:latest
    ports:
      - "8000:8000"
    volumes:
      - ./nexus.toml:/etc/nexus.toml
    environment:
      - GITHUB_TOKEN=${GITHUB_TOKEN}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

Configuration

Create a nexus.toml file to configure Nexus:

# LLM Provider configuration
[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
forward_token = true

# Model configuration (at least one model required per provider)
[llm.providers.openai.models.gpt-4]
[llm.providers.openai.models.gpt-3-5-turbo]

[llm.providers.anthropic]
type = "anthropic"
api_key = "{{ env.ANTHROPIC_API_KEY }}"

[llm.providers.anthropic.models.claude-3-5-sonnet-20241022]

# MCP Server configuration
[mcp.servers.github]
url = "https://api.githubcopilot.com/mcp/"
auth.token = "{{ env.GITHUB_TOKEN }}"

[mcp.servers.filesystem]
cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/YOUR_USERNAME/Desktop"]

[mcp.servers.python_server]
cmd = ["python", "-m", "mcp_server"]
env = { PYTHONPATH = "/opt/mcp" }
cwd = "/workspace"

Configuration Options

Server Configuration

server.listen_address: The address and port Nexus will listen on (default: 127.0.0.1:8000)
server.health.enabled: Enable health endpoint (default: true)
server.health.path: Health check endpoint path (default: /health)

LLM Configuration

llm.enabled: Enable LLM functionality (default: true)
llm.protocols.openai.enabled: Enable OpenAI protocol endpoint (default: true)
llm.protocols.openai.path: OpenAI endpoint path (default: /llm/openai)
llm.protocols.anthropic.enabled: Enable Anthropic protocol endpoint (default: false)
llm.protocols.anthropic.path: Anthropic endpoint path (default: /llm/anthropic)

For detailed LLM provider configuration, see the LLM Provider Configuration section below.

MCP Configuration

mcp.enabled: Enable MCP functionality (default: true)
mcp.path: MCP endpoint path (default: /mcp)
mcp.enable_structured_content: Control MCP search tool response format (default: true)
- When true: Uses modern structuredContent field for better performance and type safety
- When false: Uses legacy content field with Content::json objects for compatibility with older MCP clients

MCP Server Types

STDIO Servers: Launch local processes that communicate via standard input/output

[mcp.servers.my_tool]
cmd = ["path/to/executable", "--arg1", "--arg2"]

# Optional: Set environment variables
env = { DEBUG = "1", API_KEY = "{{ env.MY_API_KEY }}" }

# Optional: Set working directory
cwd = "/path/to/working/directory"

# Optional: Configure stderr handling (default: "null")
stderr = "inherit"  # Show in console
# or
stderr = { file = "/var/log/mcp/server.log" }  # Log to file

Note: STDIO servers must output valid JSON-RPC messages on stdout. The cmd array must have at least one element (the executable).

SSE Servers: Connect to Server-Sent Events endpoints

[mcp.servers.my_sse_server]
protocol = "sse"
url = "http://example.com/sse"
message_url = "http://example.com/messages"  # Optional

HTTP Servers: Connect to streamable HTTP endpoints

[mcp.servers.my_http_server]
protocol = "streamable-http"
url = "https://api.example.com/mcp"

For remote MCP servers, if you omit the protocol Nexus will first try streamable HTTP and then SSE.

Authentication

Add service token authentication to any server:

[mcp.servers.my_server.auth]
token = "your-token-here"
# Or use environment variables
token = "{{ env.MY_API_TOKEN }}"

If you enable OAuth2 authentication to your server, and your downstream servers all use the same authentication server, you can configure Nexus to forward the request access token to the downstream server.

[mcp.servers.my_server.auth]
type = "forward"

Header Insertion for MCP Servers

Nexus supports inserting static headers when making requests to MCP servers. Headers can be configured globally (for all MCP servers) or per-server.

Note: MCP currently only supports header insertion with static values. Headers from incoming requests are not forwarded.

Global MCP Headers

Configure headers that apply to all MCP servers:

# Global headers for all MCP servers
[[mcp.headers]]
rule = "insert"
name = "X-Application"
value = "nexus-router"

[[mcp.headers]]
rule = "insert"
name = "X-API-Version"
value = "v1"

Server-Specific Headers

Configure headers for individual HTTP-based MCP servers:

[mcp.servers.my_server]
url = "https://api.example.com/mcp"

# Insert headers for this specific server
[[mcp.servers.my_server.headers]]
rule = "insert"
name = "X-API-Key"
value = "{{ env.MY_API_KEY }}"  # Environment variable substitution

[[mcp.servers.my_server.headers]]
rule = "insert"
name = "X-Service-Name"
value = "my-service"

MCP Header Features

Static Values Only: Headers are set at client initialization time with static values
Environment Variables: Use {{ env.VAR_NAME }} syntax for environment variable substitution
HTTP Servers Only: Headers only apply to HTTP-based MCP servers (not STDIO servers)
Insert Rule: Currently only the insert rule is supported for MCP

MCP Access Control

Restrict access to MCP servers and tools based on user groups:

# Server-level access control
[mcp.servers.premium_tools]
cmd = ["premium-server"]
allow = ["premium", "enterprise"]  # Only these groups can access
deny = ["suspended"]                # Block specific groups

# Tool-level override (more specific than server-level)
[mcp.servers.premium_tools.tools.expensive_feature]
allow = ["enterprise"]  # Only enterprise can use this tool

[mcp.servers.premium_tools.tools.deprecated_tool]
allow = []  # Empty allow list blocks all access (no client ID needed)

Access control rules:

If allow is set, only listed groups can access (requires client identification)
If deny is set, listed groups are blocked (requires client identification)
Empty allow = [] blocks all access without requiring client identification
Tool-level rules override server-level rules
Deny takes priority over allow

OAuth2 Authentication

Configure OAuth2 authentication to protect your Nexus endpoints:

[server.oauth]
url = "https://your-oauth-provider.com/.well-known/jwks.json"
poll_interval = "5m"
expected_issuer = "https://your-oauth-provider.com"
expected_audience = "your-service-audience"

[server.oauth.protected_resource]
resource = "https://your-nexus-instance.com"
authorization_servers = ["https://your-oauth-provider.com"]

OAuth2 configuration options:

url: JWKs endpoint URL for token validation
poll_interval: How often to refresh JWKs (optional, default: no polling)
expected_issuer: Expected iss claim in JWT tokens (optional)
expected_audience: Expected aud claim in JWT tokens (optional)
protected_resource.resource: URL of this protected resource
protected_resource.authorization_servers: List of authorization server URLs

When OAuth2 is enabled, all endpoints except /health and /.well-known/oauth-protected-resource require valid JWT tokens in the Authorization: Bearer <token> header.

Rate Limiting

Nexus supports rate limiting to prevent abuse and ensure fair resource usage:

# Global rate limiting configuration
[server.rate_limits]
enabled = true

# Storage backend configuration
[server.rate_limits.storage]
type = "memory"  # or "redis" for distributed rate limiting
# For Redis backend:
# url = "redis://localhost:6379"
# key_prefix = "nexus:rate_limit:"

# Global rate limit (applies to all requests)
[server.rate_limits.global]
limit = 1000
interval = "60s"

# Per-IP rate limit
[server.rate_limits.per_ip]
limit = 100
interval = "60s"

# Per-MCP server rate limits
[mcp.servers.my_server.rate_limits]
limit = 50
interval = "60s"

# Tool-specific rate limits (override server defaults)
[mcp.servers.my_server.rate_limits.tools]
expensive_tool = { limit = 10, interval = "60s" }
cheap_tool = { limit = 100, interval = "60s" }

Rate Limiting Features:

Multiple levels: Global, per-IP, per-server, and per-tool limits
Storage backends: In-memory (single instance) or Redis (distributed)
Flexible intervals: Configure time windows for each limit
Tool-specific overrides: Set different limits for expensive operations

Redis Backend Configuration:

[server.rate_limits.storage]
type = "redis"
url = "redis://localhost:6379"
key_prefix = "nexus:rate_limit:"
response_timeout = "1s"
connection_timeout = "5s"

# Connection pool settings
[server.rate_limits.storage.pool]
max_size = 16
min_idle = 0
timeout_create = "5s"
timeout_wait = "5s"
timeout_recycle = "300s"

# TLS configuration for Redis (optional)
[server.rate_limits.storage.tls]
enabled = true
ca_cert_path = "/path/to/ca.crt"
client_cert_path = "/path/to/client.crt"  # For mutual TLS
client_key_path = "/path/to/client.key"
# insecure = true  # WARNING: Only for development/testing, skips certificate validation

Note: When configuring tool-specific rate limits, Nexus will warn if you reference tools that don't exist.

LLM Token Rate Limiting

Nexus provides token-based rate limiting for LLM providers to help control costs and prevent abuse. Unlike request-based rate limits, token rate limits count an estimate of actual tokens consumed.

Prerequisites

IMPORTANT: LLM rate limiting requires client identification to be enabled:

[server.client_identification]
enabled = true

# Choose identification methods (at least one required)
client_id.jwt_claim = "sub"                    # Extract ID from JWT 'sub' claim
# or
client_id.http_header = "X-Client-ID"          # Extract ID from HTTP header

# Optional: Limit groups per user (at most one allowed)
group_id.jwt_claim = "groups"                  # JWT claim containing user's group
# or
group_id.http_header = "X-Group-ID"            # Extract ID from HTTP header

# You must provide a list of allowed groups
[server.client_identification.validation]
group_values = ["free", "pro", "max"]

Without client identification, rate limits cannot be enforced and requests will fail with a configuration error.

Configuration Hierarchy

Token rate limits can be configured at four levels, from most to least specific:

Model per user + group: Specific model for specific each user in a group
Model per user: Specific model for each user
Provider per user + group: All models from provider for each user in a group
Provider per user: All models from provider for each user

The most specific applicable limit is always used.

Basic Configuration

# Provider-level default rate limit (applies to all models)
[llm.providers.openai.rate_limits.per_user]
input_token_limit = 100000        # 100K input tokens
interval = "1m"                   # Per minute

# Model-specific rate limit (overrides provider default)
[llm.providers.openai.models.gpt-4.rate_limits.per_user]
input_token_limit = 50000         # More restrictive for expensive model
interval = "30s"

Group-Based Rate Limits

Configure different limits for user groups (requires group_id and group_values in client identification):

# Provider-level group limits
[llm.providers.openai.rate_limits.per_user.groups]
free = { input_token_limit = 10000, interval = "60s" }
pro = { input_token_limit = 100000, interval = "60s" }
enterprise = { input_token_limit = 1000000, interval = "60s" }

# Model-specific group limits (override provider groups)
[llm.providers.openai.models.gpt-4.rate_limits.per_user.groups]
free = { input_token_limit = 5000, interval = "60s" }
pro = { input_token_limit = 50000, interval = "60s" }
enterprise = { input_token_limit = 500000, interval = "60s" }

The limits are per user, but you can define different limits if the user is part of a specific group. If the user does not belong to any group, they will be assigned to the per-user limits.

Complete Example

# Client identification (REQUIRED for rate limiting)
[server.client_identification]
enabled = true
client_id.jwt_claim = "sub"
group_id.jwt_claim = "subscription_tier"
[server.client_identification.validation]
group_values = ["free", "pro", "enterprise"]

# OpenAI provider with rate limiting
[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"

# Provider-level defaults
[llm.providers.openai.rate_limits.per_user]
input_token_limit = 100000
interval = "60s"

[llm.providers.openai.rate_limits.per_user.groups]
free = { input_token_limit = 10000, interval = "60s" }
pro = { input_token_limit = 100000, interval = "60s" }

# GPT-4 specific limits (more restrictive)
[llm.providers.openai.models.gpt-4]
[llm.providers.openai.models.gpt-4.rate_limits.per_user]
input_token_limit = 50000
interval = "60s"

[llm.providers.openai.models.gpt-4.rate_limits.per_user.groups]
free = { input_token_limit = 5000, interval = "60s" }
pro = { input_token_limit = 50000, interval = "60s" }

# GPT-3.5 uses provider defaults
[llm.providers.openai.models.gpt-3-5-turbo]

How Token Counting Works

Input Tokens Only: Rate limiting is based solely on input tokens counted from the request's messages and system prompts
No Output Buffering: Output tokens and max_tokens parameter are NOT considered in rate limit calculations
Pre-check: Input tokens are checked against rate limits before processing
Token Accumulation: Uses a sliding window algorithm to track usage over time

Note: The rate limiting is designed to be predictable and based only on what the client sends, not on variable output sizes.

Rate Limit Response

When rate limited, the server returns a 429 status code. No Retry-After headers are sent to maintain consistency with downstream LLM provider behavior.

Error Responses

When rate limits are exceeded:

{
  "error": {
    "message": "Rate limit exceeded: Token rate limit exceeded. Please try again later.",
    "type": "rate_limit_error",
    "code": 429
  }
}

Important Notes

Per-User Limits: All limits are per individual user/client ID
No Shared Pools: Currently, there are no shared/global token pools
Streaming Support: Token counting works with both regular and streaming responses
Provider Agnostic: Works consistently across all LLM providers
Validation: Configuration is validated at startup; invalid group names will cause errors

TLS Configuration

Configure TLS for downstream connections:

[mcp.servers.my_server.tls]
verify_certs = true
accept_invalid_hostnames = false
root_ca_cert_path = "/path/to/ca.pem"
client_cert_path = "/path/to/client.pem"
client_key_path = "/path/to/client.key"

LLM Provider Configuration

Nexus provides a unified interface for multiple LLM providers, allowing you to route chat completions through various services with a consistent API.

Enabling LLM Routing

[llm]
enabled = true  # Enable LLM functionality (default: true)

# OpenAI protocol endpoint configuration
[llm.protocols.openai]
enabled = true      # Enable OpenAI protocol (default: true)
path = "/llm"       # Custom path (default: "/llm/openai")

# Anthropic protocol endpoint configuration
[llm.protocols.anthropic]
enabled = true      # Enable Anthropic protocol (default: false)
path = "/claude"    # Custom path (default: "/llm/anthropic")

Supported Providers

Nexus currently supports four major LLM providers with full tool calling capabilities:

OpenAI (including OpenAI-compatible APIs) - Full tool calling and parallel execution support
Anthropic (Claude models) - Tool calling with function definitions and tool choice controls
Google (Gemini models) - Function calling with parameter schemas and tool selection
AWS Bedrock (Multiple model families via AWS) - Tool calling support across all supported model families

Provider Configuration

Configure one or more LLM providers in your nexus.toml:

OpenAI Provider

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
# Optional: Use a custom base URL (for Azure OpenAI, proxies, or compatible APIs)
base_url = "https://api.openai.com/v1"  # Default

# Model Configuration (REQUIRED - at least one model must be configured)
[llm.providers.openai.models.gpt-4]
# Optional: Rename the model for your users
# rename = "smart-model"  # Users will see "openai/smart-model"

[llm.providers.openai.models.gpt-3-5-turbo]
# Models without rename use their original ID

Anthropic Provider

[llm.providers.anthropic]
type = "anthropic"
api_key = "{{ env.ANTHROPIC_API_KEY }}"
# Optional: Use a custom base URL
base_url = "https://api.anthropic.com/v1"  # Default

# Model Configuration (REQUIRED - at least one model must be configured)
[llm.providers.anthropic.models.claude-3-opus-20240229]

[llm.providers.anthropic.models.claude-3-5-sonnet-20241022]

Google Provider

[llm.providers.google]
type = "google"
api_key = "{{ env.GOOGLE_API_KEY }}"
# Optional: Use a custom base URL
base_url = "https://generativelanguage.googleapis.com/v1beta"  # Default

# Model Configuration (REQUIRED - at least one model must be configured)
# Note: Model IDs with dots must be quoted in TOML
[llm.providers.google.models."gemini-1.5-flash"]

[llm.providers.google.models.gemini-pro]

AWS Bedrock Provider

[llm.providers.bedrock]
type = "bedrock"
# Optional: AWS profile to use (defaults to environment settings)
profile = "{{ env.AWS_PROFILE }}"
# Optional: AWS region (defaults to environment or us-east-1)
region = "us-west-2"

# Model Configuration (REQUIRED - at least one model must be configured)
# Bedrock uses model IDs with dots, so they must be quoted
[llm.providers.bedrock.models."anthropic.claude-3-5-sonnet-20241022-v2:0"]

[llm.providers.bedrock.models."anthropic.claude-3-opus-20240229-v1:0"]

[llm.providers.bedrock.models."amazon.nova-micro-v1:0"]

[llm.providers.bedrock.models."meta.llama3-8b-instruct-v1:0"]

[llm.providers.bedrock.models."ai21.jamba-1.5-mini-v1:0"]

# Rename models for simpler access
[llm.providers.bedrock.models.claude-haiku]
rename = "anthropic.claude-3-5-haiku-20241022-v1:0"  # Users will access as "bedrock/claude-haiku"

[llm.providers.bedrock.models.jamba-mini]
rename = "ai21.jamba-1.5-mini-v1:0"  # Users will access as "bedrock/jamba-mini"

AWS Bedrock provides access to multiple foundation models through a single API. Key features:

Unified Access: Use models from Anthropic, Amazon, Meta, Cohere, and more through one interface
AWS Integration: Leverages AWS credentials and IAM for authentication
Regional Availability: Models may vary by AWS region
Native Streaming: Full streaming support for all compatible models

Authentication: Bedrock uses standard AWS credential chain:

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
AWS profile (specified in config or via AWS_PROFILE environment variable)
IAM role (when running on EC2/ECS/Lambda)
AWS SSO credentials

Supported Model Families:

AI21 Jamba: Jamba 1.5 Mini and Large models with 256K context window
Anthropic Claude: All Claude 3 models (Opus, Sonnet, Haiku) and Claude Instant
Amazon Nova: Nova Micro, Lite, Pro models
Amazon Titan: Titan Text and Embeddings models
Meta Llama: Llama 2 and Llama 3 models
Cohere Command: Command and Command R models
DeepSeek: DeepSeek R1 reasoning models
Mistral: Mistral 7B and Mixtral models

Model ID Format: Bedrock model IDs follow the pattern provider.model-name-version:revision, for example:

ai21.jamba-1.5-mini-v1:0
anthropic.claude-3-5-sonnet-20241022-v2:0
amazon.nova-micro-v1:0
meta.llama3-8b-instruct-v1:0

Model Configuration

Nexus automatically discovers models from every provider. At startup the server performs an initial discovery; a background task then refreshes provider inventories every five minutes and publishes the results through a watch channel for lock-free lookups.

Automatic Discovery (default)

Simply configure a provider—no model list is required. Nexus fetches the provider's models and exposes them as bare model names:

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"

Discovered models appear in /v1/models (e.g., gpt-4, gpt-4o-mini). Requests using these bare names route automatically.

Restrict Discovery with `model_filter`

Use model_filter to limit which discovered models are exposed. The regex is case-insensitive, must not be empty, and follows the regex crate syntax.

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model_filter = "^gpt-4"  # Only expose GPT-4-family models from discovery

[llm.providers.anthropic]
type = "anthropic"
api_key = "{{ env.ANTHROPIC_API_KEY }}"
model_filter = "^claude-"

If discovery fails for any provider at startup, Nexus exits so you can fix the configuration. Refresh failures log errors but keep the last successful snapshot.

Combine Discovery with Explicit Models

Explicit models remain useful for renames, rate limits, and legacy prefixes. They always appear with provider/model even when a filter would exclude them.

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model_filter = "^gpt-4"

[llm.providers.openai.models.gpt-3-5-turbo]
rename = "fast-model"

[llm.providers.openai.models.gpt-4]
[llm.providers.openai.models.gpt-4.rate_limits.per_user]
input_token_limit = 50000
interval = "60s"

Discovery guarantees:

Provider order in the TOML determines duplicate resolution (first provider wins); skipped duplicates emit warn-level logs
/v1/models lists bare discovered models first, followed by explicit provider/model entries
The watch channel keeps request handlers in sync without locking

How model resolution works:

If the model name contains /, route to the specified provider (openai/gpt-4)
Otherwise look up the bare name in the discovered model map (gpt-4 → provider)
Return "model not found" if the name is missing from both

Accessing models:

Discovered models: Use bare names (e.g., gpt-4, claude-3-opus)
Explicit models: Use provider/model format (e.g., openai/fast-model)
Both formats share rate limits and telemetry accounting

Model Renaming

You can rename models to provide custom identifiers for your users:

[llm.providers.openai.models.gpt-4]
rename = "smart-model"  # Users will access this as "openai/smart-model"

[llm.providers.openai.models.gpt-3-5-turbo]
rename = "fast-model"   # Users will access this as "openai/fast-model"

This is useful for:

Creating business-friendly model names
Abstracting away provider-specific model names
Providing consistent naming across different providers

TOML Syntax for Model IDs

Model IDs that contain dots must be quoted in TOML:

# Correct - dots in model IDs require quotes
[llm.providers.google.models."gemini-1.5-flash"]
[llm.providers.google.models."gemini-1.5-pro"]

# Also correct - no dots, no quotes needed
[llm.providers.google.models.gemini-pro]

Multiple Provider Configuration

You can configure multiple instances of the same provider type with different names:

# Primary OpenAI account
[llm.providers.openai_primary]
type = "openai"
api_key = "{{ env.OPENAI_PRIMARY_KEY }}"

[llm.providers.openai_primary.models.gpt-4]
[llm.providers.openai_primary.models.gpt-3-5-turbo]

# Secondary OpenAI account or Azure OpenAI
[llm.providers.openai_secondary]
type = "openai"
api_key = "{{ env.OPENAI_SECONDARY_KEY }}"
base_url = "https://my-azure-instance.openai.azure.com/v1"

[llm.providers.openai_secondary.models.gpt-4]
rename = "azure-gpt-4"  # Distinguish from primary account

# Anthropic
[llm.providers.claude]
type = "anthropic"
api_key = "{{ env.ANTHROPIC_API_KEY }}"

[llm.providers.claude.models.claude-3-opus-20240229]

# Google Gemini
[llm.providers.gemini]
type = "google"
api_key = "{{ env.GOOGLE_API_KEY }}"

[llm.providers.gemini.models."gemini-1.5-flash"]

Token Forwarding

Nexus supports token forwarding, allowing users to provide their own API keys at request time instead of using the configured keys. This feature is opt-in and disabled by default.

Configuring Token Forwarding

Enable token forwarding for any provider by setting forward_token = true:

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"  # Fallback key (optional with token forwarding)
forward_token = true  # Enable token forwarding for this provider

[llm.providers.openai.models.gpt-4]
[llm.providers.openai.models.gpt-3-5-turbo]

[llm.providers.anthropic]
type = "anthropic"
# No api_key required when token forwarding is enabled
forward_token = true

[llm.providers.anthropic.models.claude-3-5-sonnet-20241022]

[llm.providers.google]
type = "google"
api_key = "{{ env.GOOGLE_API_KEY }}"
forward_token = false  # Explicitly disabled (default)

[llm.providers.google.models."gemini-1.5-flash"]

Using Token Forwarding

When token forwarding is enabled for a provider, users can pass their own API key using the X-Provider-API-Key header:

# Using your own OpenAI key
curl -X POST http://localhost:8000/llm/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Provider-API-Key: sk-your-openai-key" \
  -d '{
    "model": "openai/gpt-4",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

# Using your own Anthropic key
curl -X POST http://localhost:8000/llm/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Provider-API-Key: sk-ant-your-anthropic-key" \
  -d '{
    "model": "anthropic/claude-3-opus-20240229",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Token Forwarding Behavior

When token forwarding is enabled (forward_token = true):
- User-provided keys (via header) take priority
- Falls back to configured key if no header is provided
- Returns 401 error if neither key is available
When token forwarding is disabled (forward_token = false, default):
- Always uses the configured API key
- Ignores the X-Provider-API-Key header
- Returns 401 error if no configured key exists

Security Considerations

OAuth2 Integration: When OAuth2 is enabled, users must still authenticate with Nexus even when using token forwarding
Key Validation: API keys are validated by the provider's API
No Logging: User-provided keys are never logged
HTTPS Recommended: Always use HTTPS in production to protect API keys in transit

Provider Limitations

AWS Bedrock: Token forwarding is not supported for Bedrock providers. Bedrock uses AWS IAM credentials and request signing, which cannot be provided via simple API key headers. You must configure AWS credentials at the provider level (via environment variables, AWS profile, or explicit credentials in configuration).

Header Transformation for LLM Providers

Nexus supports header transformation for LLM providers, allowing you to forward, insert, remove, or rename headers when making requests to LLM APIs.

Provider-Level Headers

Configure header rules at the provider level:

[llm.providers.openai]
type = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"

# Forward headers from incoming requests
[[llm.providers.openai.headers]]
rule = "forward"
name = "X-Request-ID"

# Forward with a default value if not present
[[llm.providers.openai.headers]]
rule = "forward"
name = "X-Trace-ID"
default = "generated-trace-id"

# Forward and rename
[[llm.providers.openai.headers]]
rule = "forward"
name = "X-Custom-Header"
rename = "X-OpenAI-Custom"

# Insert static headers
[[llm.providers.openai.headers]]
rule = "insert"
name = "X-OpenAI-Beta"
value = "assistants=v2"

# Remove headers
[[llm.providers.openai.headers]]
rule = "remove"
name = "X-Internal-Secret"

# Pattern-based forwarding (regex)
[[llm.providers.openai.headers]]
rule = "forward"
pattern = "X-Debug-.*"

# Pattern-based removal
[[llm.providers.openai.headers]]
rule = "remove"
pattern = "X-Internal-.*"

# Rename and duplicate (keeps original and adds renamed copy)
[[llm.providers.openai.headers]]
rule = "rename_duplicate"
name = "X-User-ID"
rename = "X-OpenAI-User"

Model-Level Headers

Configure headers for specific models (overrides provider-level rules):

[llm.providers.openai.models.gpt-4]
# Model-specific headers override provider headers
[[llm.providers.openai.models.gpt-4.headers]]
rule = "insert"
name = "X-Model-Config"
value = "premium"

[[llm.providers.openai.models.gpt-4.headers]]
rule = "forward"
pattern = "X-Premium-.*"

Header Rule Types

forward: Pass headers from incoming requests to the LLM provider
- name: Single header name to forward
- pattern: Regex pattern to match multiple headers
- default: Optional default value if header is missing
- rename: Optional new name for the forwarded header
insert: Add static headers to requests
- name: Header name
- value: Static value (supports {{ env.VAR }} substitution)
remove: Remove headers before sending to provider
- name: Single header name to remove
- pattern: Regex pattern to match headers to remove
rename_duplicate: Forward header with both original and new name
- name: Original header name
- rename: New header name for the duplicate
- default: Optional default if header is missing

Important Notes

AWS Bedrock: Does not support custom headers due to SigV4 signing requirements
Priority: Model-level rules override provider-level rules
Token Forwarding: The X-Provider-API-Key header is handled separately for token forwarding
Pattern Matching: Patterns are case-insensitive regex expressions

Using the LLM API

Once configured, you can interact with LLM providers through Nexus's unified API:

List Available Models

curl http://localhost:8000/llm/models

Response:

{
  "object": "list",
  "data": [
    {
      "id": "openai_primary/gpt-4-turbo",
      "object": "model",
      "created": 1677651200,
      "owned_by": "openai"
    },
    {
      "id": "claude/claude-3-5-sonnet-20241022",
      "object": "model",
      "created": 1709164800,
      "owned_by": "anthropic"
    },
    {
      "id": "gemini/gemini-1.5-pro",
      "object": "model",
      "created": 1710000000,
      "owned_by": "google"
    }
  ]
}

Tool Calling (Function Calling)

Nexus supports advanced tool calling across all LLM providers, allowing models to invoke external functions with structured parameters. All providers use the standardized OpenAI tool calling format for consistency.

Basic Tool Calling Example:

curl -X POST http://localhost:8000/llm/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "messages": [
      {"role": "user", "content": "What'\''s the weather in San Francisco?"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "City and state"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
          },
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

Tool Choice Options:

"auto" (default): Model decides whether to call tools
"none": Model won't call any tools
"required": Model must call at least one tool
{"type": "function", "function": {"name": "specific_function"}}: Force a specific tool

Parallel Tool Calls:

When supported by the provider, models can call multiple tools simultaneously:

{
  "model": "openai/gpt-4",
  "messages": [{"role": "user", "content": "Get weather for NYC and LA"}],
  "tools": [/* tool definitions */],
  "parallel_tool_calls": true
}

Tool Conversation Flow:

Tool calling creates a multi-turn conversation:

User message → asks a question requiring tool use
Assistant message → responds with tool calls (no content)
Tool messages → provide results from tool execution
Assistant message → synthesizes final response

{
  "messages": [
    {"role": "user", "content": "What's the weather in Paris?"},
    {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_123",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\": \"Paris\"}"}
      }]
    },
    {
      "role": "tool",
      "tool_call_id": "call_123",
      "content": "Weather in Paris: 22°C, sunny"
    },
    {
      "role": "assistant",
      "content": "The weather in Paris is currently 22°C and sunny!"
    }
  ]
}

Provider-Specific Tool Support:

All providers support the standardized format, but have different capabilities:

OpenAI: Full support including parallel calls and streaming tool calls
Anthropic: Tool calling with robust function definitions and tool choice controls
Google: Function calling with JSON schema validation and tool selection
AWS Bedrock: Tool calling support varies by model family (Claude, Nova, etc.)

Streaming Tool Calls:

Tool calls can be streamed just like regular responses:

curl -X POST http://localhost:8000/llm/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4",
    "messages": [{"role": "user", "content": "Search for Python tutorials"}],
    "tools": [/* tool definitions */],
    "stream": true
  }'

The stream will include tool call chunks as they're generated, followed by the tool execution results.

Chat Completions

Send a chat completion request using the OpenAI-compatible format:

curl -X POST http://localhost:8000/llm/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai_primary/gpt-4-turbo",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7,
    "max_tokens": 150
  }'

The model name format is <provider_name>/<model_id>. Nexus automatically routes the request to the appropriate provider and transforms the request/response as needed.

Streaming Responses

Nexus supports streaming responses for all LLM providers using Server-Sent Events (SSE):

curl -X POST http://localhost:8000/llm/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "messages": [
      {"role": "user", "content": "Write a short poem"}
    ],
    "stream": true,
    "max_tokens": 100
  }'

When stream: true is set, the response will be streamed as Server-Sent Events with the following format:

data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{"role":"assistant","content":"Here"}}]}

data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{"content":" is"}}]}

data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":25,"total_tokens":35}}

data: [DONE]

Streaming is supported for all providers (OpenAI, Anthropic, Google) and provides:

Real-time token delivery as they're generated
Consistent chunk format across all providers
Usage statistics in the final chunk
Standard SSE format compatible with OpenAI SDKs

Provider-Specific Considerations

OpenAI

Supports all standard OpenAI models (GPT-3.5, GPT-4, etc.)
Compatible with Azure OpenAI endpoints
Tool Calling: Full support including parallel tool calls and streaming tool calls
Function Definitions: JSON schema support for parameters
Tool Choice: Supports all tool choice modes including specific function forcing
Supports streaming responses with Server-Sent Events (SSE)

Anthropic

System messages are automatically extracted and placed in the system field
Messages must alternate between user and assistant roles
Requires explicit max_tokens parameter (defaults to 4096 if not specified)
Tool Calling: Robust tool calling with function definitions and tool choice controls
Tool Use: Supports tool_use blocks with structured parameter validation
Streaming Tools: Tool calls can be streamed incrementally
Supports all Claude models (Opus, Sonnet, Haiku)
Supports streaming responses with Server-Sent Events (SSE)

Google

Assistant role is automatically mapped to "model" role
System messages are placed in the systemInstruction field
Function Calling: Native function calling with JSON schema valida

Nexus

README

Features

Installation

Quick Install (Linux/Windows (WSL)/macOS)

Docker

Build from Source

Running Nexus

Using the Binary

Using Docker

Docker Compose Example

Configuration

Configuration Options

Server Configuration

LLM Configuration

MCP Configuration

MCP Server Types

Authentication

Header Insertion for MCP Servers

Global MCP Headers

Server-Specific Headers

MCP Header Features

MCP Access Control

OAuth2 Authentication

Rate Limiting

LLM Token Rate Limiting

Prerequisites

Configuration Hierarchy

Basic Configuration

Group-Based Rate Limits

Complete Example

How Token Counting Works

Rate Limit Response

Error Responses

Important Notes

TLS Configuration

LLM Provider Configuration

Enabling LLM Routing

Supported Providers

Provider Configuration

OpenAI Provider

Anthropic Provider

Google Provider

AWS Bedrock Provider

Model Configuration

Automatic Discovery (default)

Restrict Discovery with model_filter

Combine Discovery with Explicit Models

Model Renaming

TOML Syntax for Model IDs

Multiple Provider Configuration

Token Forwarding

Configuring Token Forwarding

Using Token Forwarding

Token Forwarding Behavior

Security Considerations

Provider Limitations

Header Transformation for LLM Providers

Provider-Level Headers

Model-Level Headers

Header Rule Types

Important Notes

Using the LLM API

List Available Models

Tool Calling (Function Calling)

Chat Completions

Streaming Responses

Provider-Specific Considerations

OpenAI

Anthropic

Google

Comments (0)

Restrict Discovery with `model_filter`