Pivot Web Search — Multi-Provider MCP Search Plugin for Claude Code

A resilient, multi-provider web search and content extraction tool for Claude Code on Amazon Bedrock and other API providers.

What Is This?

Claude Code users on Amazon Bedrock (or other API providers) don't get Anthropic's built-in WebSearch and WebFetch tools. This Model Context Protocol (MCP) search server fills that gap — giving you a fully self-hosted, multi-provider search failover engine and a local content extractor, with no Anthropic API dependency.

Key Features

• Priority-group routing — providers grouped by priority and executed with hedged requests. Same-priority providers fire concurrently with 200ms stagger; first quality-gate pass wins. Groups tried sequentially from highest to lowest priority.
• Smart defaults — quality-first ordering (LLM Search > Tavily/Brave/Gemini > SearXNG/json_api > DDG) applied automatically. No manual priority tuning needed for common setups.
• 3-tier quality gate — AI answer presence, URL count, and keyword overlap drive automatic failover decisions. Partial results are kept as fallback while better sources are tried.
• Circuit breaker — per-provider health tracking. After 3 consecutive failures, a provider is temporarily bypassed (60s cooldown) with automatic recovery probing.
• Super mode — queries all providers in parallel, deduplicates by URL, and ranks by cross-provider agreement.
• Fully async I/O — all network calls use httpx.AsyncClient with connection pooling. No thread pools for HTTP — enables efficient concurrent requests and prepares for streamable-HTTP remote transport.
• Local content extraction — fetches and extracts full page content via trafilatura. No external API needed. Includes Next.js/Nuxt.js SPA fallback.
• Quota-aware scheduling — tracks API usage per provider, prefers cheapest available, skips exhausted providers.
• Per-host proxy cache — remembers which connection path worked for each hostname. Re-probes automatically on network changes.
• Hot-reloadable config — add/remove/reorder providers and proxies via YAML. Changes apply on the next request.
• Pluggable adapters — built-in support for DuckDuckGo (DDG), Tavily, Brave, Gemini, SearXNG, a generic json_api adapter, and llm_search for any LLM with web search grounding. Writing a new adapter is one class.
• Structured error diagnostics — when all providers fail, returns per-provider failure reasons and actionable suggestions instead of a generic error.
• Debug logging — set PIVOT_WEB_SEARCH_DEBUG=1 to get timestamped verbose logs at ~/.cache/pivot-web-search/server.log.

Why Multi-Provider Failover?

Most MCP search tools bind to a single provider. If the API is down, rate-limited, or blocked in your network, you get nothing. Single-provider tools also can't compare results across sources or handle quota exhaustion gracefully.

This plugin solves these problems by combining multiple search backends with a quality gate — if results don't pass (too few URLs, no keyword overlap), failover continues automatically to the next priority group. In super mode, all providers are queried in parallel for maximum coverage.

Prerequisites

• Claude Code installed and configured
• uv — the plugin launcher (manages Python and dependencies automatically)
• At least one search provider configured (DDG works with no API key)

Recommended: Configure at least one free API key — Tavily (1000 credits/month, no credit card) or Brave (1000 queries/month, credit card required). DDG is a free fallback but can be unreliable under heavy use.

Plugin Installation

Step 1: Add the marketplace (one-time):

claude plugin marketplace add https://github.com/Tannnnhauser/pivot-web-search.git

Step 2: Install:

claude plugin install pivot-web-search

The plugin prompts for configuration at install time:

Providers are enabled automatically based on which keys you supply: each provider with a key (or its corresponding env var) is added to the routing chain. DDG is always enabled (no key needed). Routing order and timeouts are determined by smart defaults; the order you supply keys does not matter.

For advanced setups (SearXNG, custom JSON APIs, LLM-search providers, SOCKS5 proxies, forced-proxy with no direct fallback), see ~/.pivot-web-search/.

Key resolution order — for each provider key, the plugin reads:

1. The standard env var (e.g. TAVILY_API_KEY) inherited from the parent shell. Wins if set.
2. The /plugin UI value (injected as PIVOT_USERCONFIG_TAVILY_API_KEY).

This means a value exported in your shell always takes precedence over the UI config. To use the UI value instead, unset TAVILY_API_KEY in your shell.

macOS GUI launch caveat: When Claude Code is started from Spotlight or the Dock, it does not see your ~/.zshrc exports. Either start it from a terminal, set keys via the /plugin UI, or add them to the env block in ~/.claude/config.json.

You can reconfigure anytime via claude plugin configure pivot-web-search.

Verify installation

After installing, ask Claude Code to run WebSearchConfig with action status. You should see a provider health report showing which providers are online and their quota usage.

Manual install

For users who want to skip the plugin marketplace entirely, run the MCP server remotely via uvx. Requires only uv — no clone, no venv, no pip install. uvx fetches the package directly from GitHub and runs it in a managed cache.

Add this to your .mcp.json (works in Claude Code, Claude Desktop, Cursor, and any MCP-aware host):

{
  "mcpServers": {
    "pivot-web-search": {
      "command": "uvx",
      "args": [
        "git+https://github.com/Tannnnhauser/pivot-web-search.git#subdirectory=plugins/pivot-web-search"
      ],
      "env": {
        "TAVILY_API_KEY": "tvly-...",
        "BRAVE_API_KEY": "BSA...",
        "GEMINI_SEARCH_API_KEY": "AI..."
      }
    }
  }
}

Or one-shot from the shell (e.g. for debugging):

uvx "git+https://github.com/Tannnnhauser/pivot-web-search.git#subdirectory=plugins/pivot-web-search"

First run takes a few seconds (clone + dependency resolve); subsequent runs reuse the uvx cache. Pin a specific version with @v1.0.2:

git+https://github.com/Tannnnhauser/[email protected]#subdirectory=plugins/pivot-web-search

For advanced provider/proxy config, drop YAML files at ~/.pivot-web-search/providers.yaml and ~/.pivot-web-search/proxies.yaml (templates in examples/) — they apply to uvx launches the same way they apply to plugin installs.

Uninstall

claude plugin uninstall pivot-web-search

Upgrade

claude plugin update pivot-web-search
# For uvx-based installs: refresh the cache, or pin a new @vX.Y.Z tag
uvx --refresh "git+https://github.com/Tannnnhauser/pivot-web-search.git#subdirectory=plugins/pivot-web-search"

Local development

claude --plugin-dir /path/to/pivot-web-search/

MCP Tool Reference

When called via MCP, tools are prefixed: mcp__pivot-web-search__WebSearch, mcp__pivot-web-search__WebFetch, mcp__pivot-web-search__WebSearchConfig.

WebSearch

WebFetch

Behaviors:

• 15-minute response cache per URL
• 100K character truncation (configurable via max_chars)
• Binary content detection and rejection
• Cross-host redirect safety (blocks before following)
• SPA fallback: Next.js __NEXT_DATA__ / RSC payload / Nuxt __NUXT_DATA__

WebSearchConfig

The status action returns:

• Provider health (available/unavailable with details)
• Quota usage per provider
• Config source annotations — shows where each setting comes from: environment variable (with var name), YAML file (with path), or built-in default

Configuration

The plugin runs on auto-detection by default — supply API keys via the install-time UI (or shell env vars), and the matching providers are enabled automatically with smart routing defaults. No YAML required for the common case.

For advanced setups, drop YAML files into ~/.pivot-web-search/:

Precedence is "all-or-nothing per file": if ~/.pivot-web-search/providers.yaml exists, auto-detection is bypassed entirely — list every provider you want enabled (including DDG). Same for proxies. Templates live in examples/ — copy what you need.

~/.pivot-web-search/providers.yaml

Providers are tried by priority (lower number = tried first). Same-priority providers are hedged — queried concurrently with staggered starts, first quality result wins. If no priority is set, smart defaults apply based on provider type.

providers:
  - name: tavily
    type: tavily
    api_key_env: TAVILY_API_KEY
    # priority: 20 (smart default — hedged with brave)
    # timeout: 4 (smart default)

  - name: brave
    type: brave
    api_key_env: BRAVE_API_KEY
    # priority: 20 (smart default — hedged with tavily)
    # timeout: 4 (smart default)

  - name: gemini
    type: gemini
    api_key_env: GEMINI_SEARCH_API_KEY
    model: gemini-2.5-flash
    # priority: 20 (smart default — hedged with tavily/brave)
    # timeout: 20 (smart default)

  - name: ddg
    type: ddg
    # priority: 90 (smart default — exhaustion fallback)
    # timeout: 6 (smart default)

  # Self-hosted SearXNG
  # - name: searxng-local
  #   type: searxng
  #   endpoint: "http://localhost:8888/search"

  # Generic JSON API (Serper, Google CSE, etc.)
  # Multiple json_api instances allowed — each gets independent
  # priority, quota tracking, and circuit breaker state.
  # - name: serper
  #   type: json_api
  #   endpoint: "https://google.serper.dev/search"
  #   api_key_env: SERPER_API_KEY
  #   method: POST
  #   request_body:
  #     q: "{{query}}"
  #     num: "{{max_results}}"
  #   response_mapping:
  #     results_path: "organic"
  #     title: "title"
  #     url: "link"
  #     snippet: "snippet"

Smart default priorities (when no explicit priority is set):

Note: Enabling an llm_search provider trades latency for quality — at priority 10 it runs ahead of Tavily/Brave with a 15s timeout, so every query may take 15+s. Bump its priority above 20 (or omit it from the config) if latency matters more than answer quality.

LLM Search Providers (type: llm_search)

For LLM-based search — any model with built-in web search grounding (Perplexity Sonar Pro, OpenAI with web_search, SAP AI Core, etc.). These providers return an AI-generated answer plus cited URLs extracted from the response.

This is a power-user feature. Configure by creating ~/.pivot-web-search/providers.yaml (template at examples/providers.yaml).

Two api_format paradigms are supported:

chat_completions — any /chat/completions-compatible endpoint with built-in search:

  - name: sonar-pro
    type: llm_search
    api_format: chat_completions
    endpoint: "https://api.perplexity.ai/chat/completions"
    model: sonar-pro
    api_key_env: PERPLEXITY_API_KEY
    timeout: 15
    # priority: 10 (smart default)

Response parsing uses a data-driven fallback chain:

1. search_results array (Perplexity/Sonar style)
2. annotations with type: url_citation in message (OpenAI Chat Completions style)
3. Top-level citations URL array

responses — OpenAI Responses API (/responses) with web_search tool:

  - name: gpt-web-search
    type: llm_search
    api_format: responses
    endpoint: "https://api.openai.com/v1/responses"
    model: gpt-4o
    api_key_env: OPENAI_API_KEY
    timeout: 45
    search_tool: web_search
    search_context_size: medium

Additional responses format options: filters (domain filtering object), user_location (location context).

Common fields:

The existing gemini type is also backed by LLM search internally (using Google's Search grounding) but keeps its own type: gemini for backward compatibility and dual-key fallback logic.

~/.pivot-web-search/proxies.yaml

When this file exists it takes over completely — the install-time Proxy URLs field is ignored, and direct is not auto-appended (escape hatch for forced-proxy / no-direct-fallback setups).

proxies:
  - name: direct
    url: null             # null = direct connection
    enabled: true
    priority: 1

  - name: myproxy1
    url: "http://myproxy1.example:8080"
    enabled: true
    priority: 2

  # SOCKS5 (requires PySocks: uv pip install pysocks)
  # - name: ssh-tunnel
  #   url: "socks5://127.0.0.1:1080"
  #   enabled: true
  #   priority: 3

config/fetch.yaml

Controls WebFetch behavior including JavaScript rendering fallback:

js_renderer: none         # none (default), "playwright", or "tavily"
max_chars: 100000         # content truncation limit

Set js_renderer: playwright for JavaScript-heavy sites (requires uv sync --extra browser or pip install pivot-web-search-mcp[browser]).

How Routing and Failover Works

Request
  │
  ├─ normal mode: priority-group routing
  │   ┌─ Group 1 (priority 10): LLM Search (Perplexity, OpenAI, etc.)
  │   ├─ Group 2 (priority 20): Tavily + Brave + Gemini ← hedged (200ms stagger, first quality-gate pass wins)
  │   ├─ Group 3 (priority 30): SearXNG / json_api
  │   └─ Group 4 (priority 90): DDG (free exhaustion fallback)
  │
  │   Gates: quota-exhausted → skip | circuit-open → skip | affinity mismatch → skip
  │   Quality gate (3-tier): AI answer ≥40 chars? → unique URLs ≥2? → keyword overlap?
  │   Circuit breaker: 3 consecutive failures → OPEN (60s cooldown) → HALF_OPEN → probe
  │
  └─ super mode:    Tavily ┐
                    Brave  ┤ parallel (skip exhausted) → dedup → rank by provider count
                    Gemini ┤
                    DDG    ┘

Each provider independently tries all configured proxies (direct → myproxy1 → myproxy2) with per-host caching. The proxy cache persists to ~/.cache/pivot-web-search-proxy-cache.json across sessions.

Quota Management

API usage is tracked across sessions in ~/.cache/pivot-web-search/quota.json:

Quota-aware scheduling skips providers at 100% usage. Providers are ordered by priority groups (smart defaults or explicit config). Resets automatically on calendar month rollover.

Architecture

plugins/pivot-web-search/        Plugin payload — what gets installed into the user's plugin cache
  .claude-plugin/plugin.json     Plugin manifest (userConfig schema)
  .mcp.json                      MCP server registration (uv run -m pivot_web_search_mcp)
  pyproject.toml + uv.lock       Runtime dependencies, resolved by uv at startup
  hooks/hooks.json               PreToolUse hook — blocks built-in WebSearch/WebFetch (fail-open)
                                 SessionStart hook — async health check on startup
  pivot_web_search_mcp/          FastMCP server (stdio) — fully async, exposes 3 MCP tools
    server.py                    Async tool handlers, failover orchestration, smart defaults
    routing.py                   Priority-group routing, hedged execution, circuit breaker, quality gate
    quality_gate.py              3-tier quality gate (answer/URLs/keywords)
    backends.py                  Async search backends (DDG/Tavily/Brave/Gemini, httpx)
    extraction.py                trafilatura wrapper for URL content extraction
    http_client.py               Shared httpx client + proxy failover
    results.py                   dedup_and_rank, markdown rendering
    validation.py                URL/SSRF validation
    config.py                    YAML config loaders (hot-reload)
    defaults.py                  Smart-defaults priority table
    providers/                   Subpackage: base (SearchProvider/SearchResult), adapters (6 built-ins + ADAPTER_MAP), registry (mtime hot-reload)
    llm_search_formats.py        Strategy pattern for LLM search API formats (chat_completions, responses, gemini)
    fetch.py                     SPA detection, async JS renderer dispatch (Playwright/Tavily)
    quota.py                     Cross-session quota tracking (filelock, cross-platform)
    logging.py                   Centralized logging (stderr + optional file via PIVOT_WEB_SEARCH_DEBUG)
  config/                        Bundled fetch.yaml defaults (provider/proxy YAML lives in ~/.pivot-web-search/)
  scripts/
    health-check.py              Startup probe — reports provider availability and quota
    pretool-check.py             PreToolUse hook script — fail-open tool blocker
  skills/pivot-web-search/       Skill definition surfaced to Claude Code

.claude-plugin/marketplace.json  Marketplace manifest pointing to ./plugins/pivot-web-search
examples/                        YAML templates for ~/.pivot-web-search/ — copy and edit for advanced setups
tests/                           308 offline tests + live integration (pytest-asyncio)
pyproject.toml                   Workspace shell — dev deps and lint/test config (no runtime code)

Testing

uv sync                               # install workspace + dev dependencies
pytest -m "not integration"           # 308 offline tests (~5s)
pytest                                # all tests including live integration (requires BRAVE/TAVILY keys)

Troubleshooting

Enable debug logging Set PIVOT_WEB_SEARCH_DEBUG=1 as an environment variable. Verbose timestamped logs are written to ~/.cache/pivot-web-search/server.log. Useful for diagnosing provider failures and proxy routing.

No results from any provider Run WebSearchConfig with action status to check provider health and see which config source is active for each setting. Ensure at least one provider has a valid API key configured (or that DDG is reachable from your network).

SSL certificate errors on macOS The plugin uses certifi for CA bundles. If you still see SSL errors: uv sync --upgrade-package certifi

Proxy timeouts / slow startup The SessionStart health check runs asynchronously and never blocks your session. If you're not behind a proxy, leave the Proxy URLs field empty — direct is always the final fallback.

DuckDuckGo rate limiting (403 errors) DDG occasionally rate-limits aggressive queries. The circuit breaker automatically detects consecutive failures and temporarily bypasses DDG (60s cooldown with probe-based recovery). DDG is restored once a probe request succeeds. For reliability, configure at least one API-backed provider.

trafilatura extraction returns empty Some JavaScript-heavy sites need a renderer. Set js_renderer: playwright in config/fetch.yaml and install: uv sync --extra browser (or pip install pivot-web-search-mcp[browser]) then playwright install chromium.

Playwright not installed error Playwright is optional. Install only when needed: uv sync --extra browser && playwright install chromium (or: pip install pivot-web-search-mcp[browser] && playwright install chromium)

Alternatives Comparison

License

Apache-2.0