Discord Archive

Archive Discord servers to PostgreSQL and query them from Claude Code over MCP — structured SQL, a user interaction graph, and semantic search (NV-Embed-v2 + LanceDB) — plus a 3D galaxy visualizer of the embedding space.

Architecture

graph LR
    subgraph Ingest
        A[Discord API] -->|REST| B[IngestOrchestrator]
    end

    subgraph Storage
        B --> C[(PostgreSQL)]
    end

    subgraph RAG Pipeline
        C -->|messages| D[Chunking]
        D -->|chunk_texts| E[NV-Embed-v2]
        E -->|4096-dim vectors| F[(LanceDB)]
    end

    subgraph Graph
        C -->|replies + mentions| M[Graph builder]
        M -->|user_interactions| C
    end

    subgraph Agent
        F --> G[MCP Server]
        C --> G
        G -->|stdio| H[Claude Code]
    end

    subgraph Galaxy
        F --> I[GPU PCA + UMAP]
        I -->|3D coordinates| J[FastAPI]
        C --> J
        J --> K[React + Three.js]
    end

Five layers, each runnable independently:

Setup

Requirements: Python 3.11–3.12, Docker, Node.js (galaxy frontend only). Embedding and projection need a CUDA GPU — NV-Embed-v2 runs in fp16, which is ~16 GB of VRAM for the weights alone.

cp .env.example .env     # edit with your PostgreSQL credentials
docker compose up -d     # start PostgreSQL

cp config.example.toml config.toml  # edit with your database URL and Discord tokens

uv sync                  # core only (ingest)
uv sync --extra rag      # + RAG pipeline and MCP server
uv sync --extra galaxy   # + 3D visualization

cd web && npm install && npm run build   # galaxy frontend → web/dist (galaxy only)

Note: archiving with a user token is self-botting, which violates Discord's Terms of Service. Use at your own risk.

Database migrations

The schema is managed with Alembic. Every pipeline command upgrades the database to the latest revision on startup, so no manual step is needed — a pre-Alembic database is detected and stamped automatically. To manage migrations directly:

uv run alembic upgrade head                          # apply pending migrations
uv run alembic revision --autogenerate -m "message"  # generate a migration after model changes

Ingest

Downloads guilds, channels, roles, messages, attachments, reactions, emojis, stickers, and scheduled events. Supports historical backfill and incremental sync with per-channel checkpoints.

discord-archive ingest                          # all configured guilds
discord-archive ingest --guild-id 123           # specific guild
discord-archive ingest --channel-id 456         # specific channel
discord-archive ingest -v                       # verbose logging

Rate limits are handled automatically (429 → wait, 5xx → exponential backoff, 403 → skip channel). Interrupted runs resume from checkpoint.

RAG Pipeline

Three-stage pipeline that turns messages into searchable vector embeddings:

1. Chunking — Groups messages into semantic chunks using three strategies (sliding window, author group, reply chain). Token-aware boundaries.

2. Embedding — Encodes chunks with NV-Embed-v2 (4096-dim, L2-normalized). Batched by token budget to maximize GPU throughput.

3. Storage — Vectors stored in LanceDB with metadata (guild, channel, authors, timestamps) for filtered ANN search.

discord-archive chunk                           # create chunks
discord-archive embed                           # encode to vectors

Both commands accept --guild-id and --channel-id filters.

Interaction Graph

Extracts directed reply and mention edges from messages into a per-guild user interaction graph (edge weight = interaction count). Reply auto-pings are excluded from mention edges so a pinged reply isn't counted twice. Incremental via per-guild checkpoints.

discord-archive graph                           # process new messages since last run
discord-archive graph --rebuild                 # recount from scratch (after backfilling older history)

Retrieval (MCP Server)

A FastMCP server that exposes the archive to Claude Code as five tools:

Registered in .mcp.json and auto-started by Claude Code. Lazy-loads the embedding model on first search.

In practice, structured queries carry most lookups — short, context-heavy chat messages are a hard target for embedding retrieval. Treat semantic search as a fuzzy entry point and SQL as the source of truth.

Galaxy

3D semantic visualization of the archive. Purely visual — it shows the shape of a community rather than answering questions about it.

1. Projection — GPU PCA (4096 → 200) then UMAP (200 → 3D). Exports per-guild binary point clouds.
2. Server — FastAPI serving projection data, chunk details, and search API.
3. Frontend — React + Three.js with custom GLSL shaders for point rendering and GPU picking.

discord-archive project                         # compute 3D coordinates
discord-archive serve                           # start web server (port 8000)

Project Structure

alembic/                 # Database migrations
discord_archive/
├── config/              # Pydantic settings
├── core/                # BaseOrchestrator
├── db/
│   ├── models/          # SQLAlchemy ORM (16 tables)
│   └── repositories/    # Data access layer
├── ingest/              # Discord API → PostgreSQL
├── rag/
│   ├── chunking/        # Messages → semantic chunks
│   ├── embedding/       # NV-Embed-v2 + LanceDB
│   ├── graph/           # Reply/mention edges → interaction graph
│   ├── projection/      # GPU PCA + UMAP → 3D
│   └── retrieval/       # MCP server
├── galaxy/              # FastAPI web server
└── utils/               # Snowflake, permissions, logging
web/                     # React + Three.js frontend

License

MIT