Omnifeed

• web_search queries a SearXNG instance (Google/Bing/DDG, Reddit included) and returns ranked URLs with titles and snippets.
• fetch_url renders any URL through crawl4ai as clean markdown — and Reddit URLs (threads and /r/{sub} listings) come back as TOON, as do Hacker News threads and front-page / Ask / Show feeds (read directly from the Algolia HN API).

Why omnifeed

Demo

Quick start

# Fetch the compose file + SearXNG settings, then start:
curl -fsSL https://raw.githubusercontent.com/kinorai/omnifeed/main/docker-compose.yml -o docker-compose.yml
curl -fsSL --create-dirs https://raw.githubusercontent.com/kinorai/omnifeed/main/searxng/settings.yml -o searxng/settings.yml
docker compose up

Starts omnifeed + SearXNG + crawl4ai — tokenless out of the box (the compose file sets OMNIFEED_DEV_NO_AUTH=true), so docker compose up just works. Point Open WebUI at http://localhost:8080 with WEB_LOADER_ENGINE=external. (SearXNG is mounted with searxng/settings.yml, which enables the json format web_search needs.) See Authentication below to require a bearer token.

As an MCP server

Works with any MCP client — Claude Code, Cursor, Codex, Gemini CLI, OpenCode, Windsurf, Pi, and more.

HTTP — recommended. docker compose up already runs the MCP server on :8081 (tokenless in dev mode), so the simplest setup is no extra container at all — point your client at the URL:

{ "mcpServers": { "omnifeed": { "url": "http://localhost:8081/mcp" } } }

Stdio — for clients that only speak stdio. A stdio server is spawned and owned by your client (it pipes JSON-RPC over the process's stdin/stdout), so it can't be a long-running compose service — but you can launch the mcp profile from this compose file, which keeps every setting (upstreams, network, image) in one place:

{
  "mcpServers": {
    "omnifeed": {
      "command": "docker",
      "args": ["compose", "-f", "/abs/path/to/docker-compose.yml", "run", "-T", "--rm", "mcp"]
    }
  }
}

run -T disables the TTY so JSON-RPC pipes cleanly; the container joins the stack's network and reuses crawl4ai/SearXNG. Bring the stack up first (docker compose up -d) so the upstreams are healthy.

Spawn the container directly and tell it where crawl4ai/SearXNG are reachable (omnifeed exits at startup without OMNIFEED_CRAWL4AI_URL):

{
  "mcpServers": {
    "omnifeed": {
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "-e", "OMNIFEED_CRAWL4AI_URL=http://host.docker.internal:11235/crawl",
        "-e", "OMNIFEED_SEARXNG_URL=http://host.docker.internal:8080",
        "kinorai/omnifeed:latest", "--mcp-stdio"
      ]
    }
  }
}

On Linux, add "--add-host=host.docker.internal:host-gateway" to the args so host.docker.internal resolves.

Tools: fetch_url (always) and web_search (only when OMNIFEED_SEARXNG_URL is set). The intended loop is web_search → pick URLs → fetch_url.

/crawl returns [{"page_content": "...", "metadata": {...}}] — already the shape of a LangChain / LlamaIndex Document, so wrapping it as a custom document loader takes only a few lines.

Authentication

The compose stack runs tokenless for local use (OMNIFEED_DEV_NO_AUTH=true). To require a bearer token instead, generate one — this is the value clients send as Authorization: Bearer <token>, so copy it:

openssl rand -hex 32        # ← your token; copy this

Then in docker-compose.yml set OMNIFEED_API_KEY to that value and remove OMNIFEED_DEV_NO_AUTH. Without a key (and without OMNIFEED_DEV_NO_AUTH=true) the proxy refuses to start, so it can't be left open by accident. Stdio MCP needs no token — it inherits the trust of the process that spawned it.

Configuration

Everything is configured with OMNIFEED_-prefixed environment variables. In practice you only ever set three — OMNIFEED_API_KEY, OMNIFEED_CRAWL4AI_URL, and (optionally) OMNIFEED_SEARXNG_URL. The rest have sane defaults.

Controlling Reddit response size

A Reddit thread's comment tree can be huge. The size knobs come in two kinds — it matters which is which:

• Upstream Reddit params — forwarded verbatim to Reddit's API, so Reddit owns their behavior: OMNIFEED_REDDIT_FETCH_LIMIT → limit, OMNIFEED_REDDIT_DEPTH → depth, OMNIFEED_REDDIT_SORT → sort. They shape what Reddit sends back (less latency, fewer tokens) but are approximate, and limit/depth bound only the initial fetch. Semantics are Reddit's, not ours — see https://www.reddit.com/dev/api/ → GET [/r/subreddit]/comments/article (limit = "maximum number of comments to return", depth = "maximum depth of subtrees").
• omnifeed engine caps — our own, applied after fetch + expansion, so they're exact and independent of Reddit: OMNIFEED_REDDIT_MAX_COMMENTS (truncate the flat comment list) and OMNIFEED_REDDIT_MAX_TOP_LEVEL (keep the first N top-level threads, in sort order, with their replies).

Rule of thumb: reach for the upstream params to fetch less from Reddit; reach for the engine caps when you need a guaranteed ceiling — OMNIFEED_REDDIT_MAX_ROUNDS expansion adds comments on top of limit, so only the caps bound the final total. All five are also per-request on the fetch_url MCP tool (limit, depth, sort, max_comments, max_top_level); a positive value overrides the env default.

Architecture

Two ports answer two questions. The Searcher port answers query → URLs; the Engine port answers URL → content. MCP tools and REST handlers compose them; transports stay thin.

%%{init: {"theme":"base","themeVariables":{"background":"transparent","mainBkg":"#161b22","primaryColor":"#161b22","primaryTextColor":"#e6edf3","primaryBorderColor":"#FF4500","lineColor":"#8b949e","secondaryColor":"#161b22","tertiaryColor":"#161b22"},"flowchart":{"curve":"basis","htmlLabels":false}}}%%
flowchart TB
  crawl["POST /crawl"] e1@--> owt["Open WebUI<br/>transport"]
  search["POST /search"] e2@--> sat["SearchAPI<br/>transport"]
  mcpStdio["MCP stdio"] e3@--> mcp["MCP server"]
  mcpHTTP["MCP HTTP /mcp"] e4@--> mcp

  owt e5@--> reg["Engine Registry"]
  mcp -- crawl tools --> reg
  sat e6@--> searcher["Searcher<br/>(SearXNG)"]
  mcp -- search tool --> searcher

  reg e7@--> reddit["Reddit engine<br/>(TOON)"]
  reg e12@--> hn["Hacker News engine<br/>(TOON)"]
  reg e8@--> generic["Generic fallback<br/>(markdown)"]
  reddit e9@--> c4["crawl4ai upstream<br/>(headless browser)"]
  generic e10@--> c4
  hn e13@--> algolia["Algolia HN API<br/>(hn.algolia.com)"]
  searcher e11@--> sx["SearXNG upstream<br/>(Google / Bing / DDG)"]

  classDef box fill:#161b22,stroke:#30363d,stroke-width:1px,color:#e6edf3;
  classDef accent fill:#0d1117,stroke:#FF4500,stroke-width:2px,color:#ffd9b3;
  classDef animate stroke:#FF4500,stroke-width:2px,stroke-dasharray:10 6,stroke-dashoffset:900,animation:dash 14s linear infinite;
  class crawl,search,mcpStdio,mcpHTTP,owt,sat box;
  class mcp,reg,searcher,reddit,hn,generic,c4,sx,algolia accent;
  class e1,e2,e3,e4,e5,e6,e7,e8,e9,e10,e11,e12,e13 animate;

Reddit anti-bot handling

Reddit's edge 403-blocks non-browser HTTP clients (it fingerprints the TLS/JA3 handshake), so the Reddit engine never calls Reddit directly. It drives crawl4ai's headless Chromium to a www.reddit.com page (which clears the bot wall), then runs a same-origin fetch() of the .json and /api/morechildren endpoints from inside that page. No Reddit auth, cookies, or API key. A per-thread crawl4ai session_id is reused across expansion rounds to keep one warmed context.

Sustained scraping can raise your source IP's risk score. If fetches start returning the block page, slow down, keep expand modest, or route crawl4ai through a residential proxy.

Extending it

New engines (Hacker News, Stack Overflow, …), searchers (Brave, Tavily, …), MCP tools, and transports each plug into one small port without touching the rest. See AGENTS.md → Adding things for the architecture and a step-by-step.

Development

git clone https://github.com/kinorai/omnifeed.git && cd omnifeed
make check        # vet + lint + test — hermetic, no upstreams or token needed
docker compose up # run the full stack locally (tokenless: ports 8080 / 8081 / 9090)

See CONTRIBUTING.md for the full workflow, SECURITY.md for vulnerability reporting, and AGENTS.md if you're a coding agent working in this repo.

Star history

Contributing

Star it if it's useful — it helps other AI builders find omnifeed.

New engines, searchers, MCP tools, and transports are all welcome — start with AGENTS.md and CONTRIBUTING.md.

License

MIT © kinorai