RAGent - RAG System Builder for Markdown Documents

日本語版 (Japanese) / 日本語 README

RAGent is a CLI tool for building a RAG (Retrieval-Augmented Generation) system from markdown documents using hybrid search capabilities (BM25 + vector search) with Amazon S3 Vectors and OpenSearch.

Table of Contents

Features

  • Vectorization: Convert source files (markdown, CSV, and PDF) from local directories, S3, or GitHub repositories to embeddings using Amazon Bedrock
  • GitHub Data Source: Clone GitHub repositories and vectorize their markdown/CSV files with auto-generated metadata
  • S3 Vector Integration: Store generated vectors in Amazon S3 Vectors
  • Hybrid Search: Combined BM25 + vector search using OpenSearch
  • Slack Search Integration: Blend document results with Slack conversations via an iterative enrichment pipeline
  • Semantic Search: Semantic similarity search using S3 Vector Index
  • Interactive RAG Chat: Chat interface with context-aware responses
  • Vector Management: List vectors stored in S3
  • Embedding-Agnostic RAG: Optional search path that answers directly from Slack without requiring pre-built vectors
  • MCP Server: Model Context Protocol server for Claude Desktop integration
  • OIDC Authentication: OpenID Connect authentication with multiple providers
  • IP-based Security: IP address-based access control with configurable bypass ranges and audit logging
  • Dual Transport: HTTP and Server-Sent Events (SSE) transport support
  • AWS Secrets Manager Integration: Automatic secret injection as environment variable fallback

Slack Search Integration

Slack search extends every retrieval workflow by streaming recent conversations, threads, and channel timelines directly from Slack’s Web API. When SLACK_SEARCH_ENABLED=true, the following behavior is enabled automatically:

  • query exposes --enable-slack-search flag to opt in per request.
  • chat surfaces Slack context in addition to documents without requiring extra flags.
  • slack-bot responds with combined document + Slack answers inside Block Kit messages.
  • mcp-server adds enable_slack_search parameter to the hybrid_search tool.

Each Slack lookup performs iterative query refinement, merges timeline context from threads, and runs a sufficiency check before the final answer is generated. Results include permalinks so operators can pivot back to the original conversation instantly.

Slack URL Reference

You can include Slack message URLs directly in your queries. RAGent automatically detects and fetches the referenced messages, including thread replies, making them available as context for your question. This works across all commands (query, chat, slack-bot, mcp-server) regardless of whether SLACK_SEARCH_ENABLED is set.

# Fetch and analyze a specific Slack conversation
RAGent query -q "Summarize this discussion: https://your-workspace.slack.com/archives/C12345678/p1234567890123456"

# Reference multiple Slack messages
RAGent chat
> Compare these two threads: https://slack.com/.../p111 and https://slack.com/.../p222

Quick example:

# Search across documents and Slack in one command
RAGent query -q "incident timeline" --enable-slack-search

Embedding-Agnostic RAG

Slack search is intentionally embedding-agnostic: the system fetches live messages at query time instead of relying on pre-generated vectors. This provides several benefits:

  • Operational cost savings: no additional vector storage or nightly re-indexing jobs are required for Slack data.
  • Real-time awareness: answers incorporate messages posted seconds ago, keeping incident timelines and release notes fresh.
  • Time-series preservation: the pipeline keeps message order and thread structure so analysts can replay discussions accurately.
  • Seamless fallbacks: when Slack returns no hits the document-only hybrid search continues without interruption.

Slack-only output is still fused with document references, giving operators a single consolidated view while staying within compliance rules.

Environment Variable Setup Guide

RAGent integrates many services, and the required environment variables differ depending on your choices. This section provides visual flowcharts for the two main workflows: vectorization (data ingestion) and output (search & usage).

Vectorization Flow (vectorize command)

Follow this flowchart to determine which environment variables are needed for vectorization.

flowchart TD
    Start([Run vectorize]) --> AuthMethod

    %% ── Authentication method ──
    AuthMethod{{"AWS authentication method?"}}

    AuthMethod -->|"IAM Credentials"| IAMAuth["IAM Credentials
    ━━━━━━━━━━━━━━━━━━━
    AWS_REGION
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY
    BEDROCK_REGION (default: us-east-1)"]

    AuthMethod -->|"Bearer Token"| BearerAuth["Bearer Token Auth
    ━━━━━━━━━━━━━━━━━━━
    BEDROCK_REGION
    * AWS_BEARER_TOKEN_BEDROCK
    ━━━━━━━━━━━━━━━━━━━
    AWS_REGION etc. not required"]

    IAMAuth --> OpenSearch
    BearerAuth --> OpenSearch

    %% ── OpenSearch (always required) ──
    OpenSearch["OpenSearch Config - BM25 Index
    ━━━━━━━━━━━━━━━━━━━
    * OPENSEARCH_ENDPOINT
    * OPENSEARCH_INDEX
    OPENSEARCH_REGION (default: us-east-1)"]

    OpenSearch --> VectorDB

    %% ── Vector DB backend selection ──
    VectorDB{{"Vector storage backend?
    VECTOR_DB_BACKEND"}}

    VectorDB -->|"s3 (default)"| S3Config["S3 Vectors Config
    ━━━━━━━━━━━━━━━━━━━
    * AWS_S3_VECTOR_BUCKET
    * AWS_S3_VECTOR_INDEX
    S3_VECTOR_REGION (default: us-east-1)"]

    VectorDB -->|sqlite| SQLiteConfig["SQLite-vec Config
    ━━━━━━━━━━━━━━━━━━━
    SQLITE_VEC_DB_PATH
    (default: ~/.ragent/vectors.db)"]

    S3Config --> DataSource
    SQLiteConfig --> DataSource

    %% ── Data source selection ──
    DataSource{{"Data source?"}}

    DataSource -->|"Local (default)"| LocalSource["Local Directory
    ━━━━━━━━━━━━━━━━━━━
    --directory ./source
    No additional env vars"]

    DataSource -->|S3 Bucket| S3Source["S3 Source Config
    ━━━━━━━━━━━━━━━━━━━
    --enable-s3 flag
    --s3-bucket bucket-name
    S3_SOURCE_REGION (default: us-east-1)"]

    DataSource -->|GitHub Repository| GitHubSource["GitHub Config
    ━━━━━━━━━━━━━━━━━━━
    --github-repos owner/repo
    GITHUB_TOKEN
    (required for private repos)"]

    DataSource -->|Multiple| MultiSource["Multiple Sources
    ━━━━━━━━━━━━━━━━━━━
    Local + S3 + GitHub can be combined"]

    LocalSource --> OCR
    S3Source --> OCR
    GitHubSource --> OCR
    MultiSource --> OCR

    %% ── OCR (PDF support) ──
    OCR{{"Process PDFs?
    OCR_PROVIDER"}}

    OCR -->|"No (default)"| NoOCR["PDFs are skipped"]
    OCR -->|bedrock| OCRBedrock["Bedrock OCR Config
    ━━━━━━━━━━━━━━━━━━━
    OCR_PROVIDER=bedrock
    OCR_MODEL (optional)
    OCR_TIMEOUT (default: 600s)
    OCR_CONCURRENCY (default: 5)"]
    OCR -->|gemini| OCRGemini["Gemini OCR Config
    ━━━━━━━━━━━━━━━━━━━
    OCR_PROVIDER=gemini
    GEMINI_API_KEY (API key auth)
    ━━━ OR ━━━
    GOOGLE_APPLICATION_CREDENTIALS
    GEMINI_GCP_PROJECT
    GEMINI_GCP_LOCATION (default: us-central1)
    ━━━━━━━━━━━━━━━━━━━
    OCR_MODEL (optional)"]

    NoOCR --> Optional
    OCRBedrock --> Optional
    OCRGemini --> Optional

    %% ── Optional settings ──
    Optional["Optional Settings
    ━━━━━━━━━━━━━━━━━━━
    VECTORIZER_CONCURRENCY (default: 10)
    VECTORIZER_RETRY_ATTEMPTS (default: 10)
    EXCLUDE_CATEGORIES"]

    Optional --> Ready([Ready to run vectorize])

    %% ── Styles ──
    style Start fill:#4CAF50,color:#fff
    style Ready fill:#4CAF50,color:#fff
    style AuthMethod fill:#FF9800,color:#fff
    style VectorDB fill:#FF9800,color:#fff
    style DataSource fill:#FF9800,color:#fff
    style OCR fill:#FF9800,color:#fff
    style IAMAuth fill:#e3f2fd
    style BearerAuth fill:#e3f2fd
    style OpenSearch fill:#e3f2fd
    style S3Config fill:#fff3e0
    style SQLiteConfig fill:#fff3e0
    style LocalSource fill:#e8f5e9
    style S3Source fill:#e8f5e9
    style GitHubSource fill:#e8f5e9
    style MultiSource fill:#e8f5e9
    style OCRBedrock fill:#fce4ec
    style OCRGemini fill:#fce4ec
    style NoOCR fill:#f5f5f5
    style Optional fill:#f3e5f5

Legend: * = required (error if unset), default: xxx = has a default value. IAM credentials and Bearer Token (AWS_BEARER_TOKEN_BEDROCK) are mutually exclusive.

Output Flow (Search & Usage Commands)

The required environment variables depend on which command you use and which search mode you choose.

flowchart TD
    Start([Use vectorized data]) --> Command

    %% ── Command selection ──
    Command{{"Which command?"}}

    Command -->|query| QueryCmd["query
    ━━━━━━━━━━━━━━━━
    Semantic search
    Terminal output"]

    Command -->|chat| ChatCmd["chat
    ━━━━━━━━━━━━━━━━
    Interactive RAG chat
    Claude-powered answers"]

    Command -->|slack-bot| SlackBotCmd["slack-bot
    ━━━━━━━━━━━━━━━━
    Slack Bot
    Auto-respond to mentions"]

    Command -->|mcp-server| MCPCmd["mcp-server
    ━━━━━━━━━━━━━━━━
    MCP Protocol Server
    Claude Desktop integration"]

    QueryCmd --> SearchMode
    ChatCmd --> SearchMode
    SlackBotCmd --> SlackBotConfig
    MCPCmd --> MCPAuth

    %% ── Search mode (query / chat) ──
    SearchMode{{"Search mode?"}}

    SearchMode -->|"Hybrid search (default)"| HybridBase
    SearchMode -->|"--only-slack"| SlackOnly

    %% ── Hybrid search common config ──
    HybridBase["Common Config (required)
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * OPENSEARCH_ENDPOINT
    * OPENSEARCH_INDEX
    CHAT_MODEL (for chat command)"]

    HybridBase --> SlackSearch

    %% ── Slack search addition ──
    SlackSearch{{"Also use Slack search?
    SLACK_SEARCH_ENABLED"}}

    SlackSearch -->|"false (default)"| HybridOnly["Hybrid search only
    ━━━━━━━━━━━━━━━━━━━
    No additional env vars"]

    SlackSearch -->|true| HybridPlusSlack["Add Slack Search Config
    ━━━━━━━━━━━━━━━━━━━
    SLACK_SEARCH_ENABLED=true
    * SLACK_USER_TOKEN
    ━━━━━ Optional ━━━━━
    SLACK_SEARCH_MAX_RESULTS (default: 20)
    SLACK_SEARCH_MAX_ITERATIONS (default: 3)
    SLACK_SEARCH_TIMEOUT_SECONDS (default: 60)"]

    %% ── Slack-Only mode ──
    SlackOnly["Slack-Only Config
    ━━━━━━━━━━━━━━━━━━━
    OpenSearch not required
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * SLACK_USER_TOKEN
    SLACK_SEARCH_ENABLED=true"]

    %% ── Slack Bot specific config ──
    SlackBotConfig["Slack Bot Config
    ━━━━━━━━━━━━━━━━━━━
    * SLACK_BOT_TOKEN
    SLACK_APP_TOKEN (for Socket Mode)
    SLACK_RESPONSE_TIMEOUT (default: 5s)
    SLACK_ENABLE_THREADING (default: true)"]

    SlackBotConfig --> SlackBotSearch

    SlackBotSearch{{"Search mode?"}}

    SlackBotSearch -->|"Hybrid (default)"| SlackBotHybrid["Common Config (required)
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * OPENSEARCH_ENDPOINT
    * OPENSEARCH_INDEX"]

    SlackBotSearch -->|"--only-slack"| SlackBotOnly["Slack-Only Config
    ━━━━━━━━━━━━━━━━━━━
    OpenSearch not required
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * SLACK_USER_TOKEN
    SLACK_SEARCH_ENABLED=true"]

    SlackBotHybrid --> SlackBotSlackOpt
    SlackBotSlackOpt{{"Also use Slack search?"}}
    SlackBotSlackOpt -->|false| SlackBotReady([Ready to run slack-bot])
    SlackBotSlackOpt -->|true| SlackBotSlack["SLACK_SEARCH_ENABLED=true
    * SLACK_USER_TOKEN"]
    SlackBotSlack --> SlackBotReady
    SlackBotOnly --> SlackBotReady

    %% ── MCP Server auth config ──
    MCPAuth{{"Auth method?
    --auth-method"}}

    MCPAuth -->|"ip (default)"| MCPIP["IP Auth
    ━━━━━━━━━━━━━━━━━━━
    MCP_ALLOWED_IPS (default: 127.0.0.1,::1)"]

    MCPAuth -->|oidc| MCPOIDC["OIDC Auth
    ━━━━━━━━━━━━━━━━━━━
    * OIDC_ISSUER
    * OIDC_CLIENT_ID
    * OIDC_CLIENT_SECRET"]

    MCPAuth -->|both| MCPBoth["IP + OIDC both required
    ━━━━━━━━━━━━━━━━━━━
    IP auth config +
    OIDC auth config"]

    MCPAuth -->|either| MCPEither["IP or OIDC either one
    ━━━━━━━━━━━━━━━━━━━
    IP auth config or
    OIDC auth config"]

    MCPIP --> MCPServer
    MCPOIDC --> MCPServer
    MCPBoth --> MCPServer
    MCPEither --> MCPServer

    MCPServer["MCP Server Config
    ━━━━━━━━━━━━━━━━━━━
    MCP_SERVER_HOST (default: localhost)
    MCP_SERVER_PORT (default: 8080)"]

    MCPServer --> MCPSearch

    MCPSearch{{"Search mode?"}}

    MCPSearch -->|"Hybrid (default)"| MCPHybrid["Common Config (required)
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * OPENSEARCH_ENDPOINT
    * OPENSEARCH_INDEX"]

    MCPSearch -->|"--only-slack"| MCPSlackOnly["Slack-Only Config
    ━━━━━━━━━━━━━━━━━━━
    OpenSearch not required
    ━━━━━━━━━━━━━━━━━━━
    AWS Auth (IAM or Bearer Token)
    BEDROCK_REGION
    * SLACK_USER_TOKEN
    SLACK_SEARCH_ENABLED=true"]

    MCPHybrid --> MCPReady([Ready to run mcp-server])
    MCPSlackOnly --> MCPReady

    HybridOnly --> Ready([Ready])
    HybridPlusSlack --> Ready
    SlackOnly --> Ready

    %% ── Styles ──
    style Start fill:#4CAF50,color:#fff
    style Ready fill:#4CAF50,color:#fff
    style SlackBotReady fill:#4CAF50,color:#fff
    style MCPReady fill:#4CAF50,color:#fff

    style Command fill:#2196F3,color:#fff
    style SearchMode fill:#FF9800,color:#fff
    style SlackSearch fill:#FF9800,color:#fff
    style SlackBotSearch fill:#FF9800,color:#fff
    style SlackBotSlackOpt fill:#FF9800,color:#fff
    style MCPAuth fill:#FF9800,color:#fff
    style MCPSearch fill:#FF9800,color:#fff

    style QueryCmd fill:#e3f2fd
    style ChatCmd fill:#fce4ec
    style SlackBotCmd fill:#e8f5e9
    style MCPCmd fill:#f3e5f5

    style HybridBase fill:#e3f2fd
    style SlackBotConfig fill:#e8f5e9
    style MCPServer fill:#f3e5f5

    style HybridOnly fill:#e8f5e9
    style HybridPlusSlack fill:#fff3e0
    style SlackOnly fill:#fff3e0

    style MCPIP fill:#f3e5f5
    style MCPOIDC fill:#f3e5f5
    style MCPBoth fill:#f3e5f5
    style MCPEither fill:#f3e5f5

Architecture Overview

The diagram below highlights how document and Slack pipelines converge before the answer is generated.

graph LR
    subgraph Sources["Data Sources"]
        MD[Markdown Files]
        CSV[CSV Files]
        S3[Amazon S3 Bucket]
        GH[GitHub Repositories]
    end

    Sources -->|Vectorize| VE[Amazon S3 Vectors]
    VE --> HY[Hybrid Search Engine]
    OS[(Amazon OpenSearch)] --> HY
    SL[Slack Workspace] -->|Conversations API| SS[SlackSearch Service]
    SS --> HY
    HY --> CT[Context Builder]
    CT --> AN["Answer Generation (Claude, Bedrock Chat)"]

Detailed Architecture Diagrams

Command Overview

RAGent provides five main commands, each serving a specific purpose in the RAG workflow:

flowchart LR
    User([User])

    User -->|1| Vectorize[vectorize<br/>Markdown → Vectors]
    User -->|2| Query[query<br/>Semantic Search]
    User -->|3| Chat[chat<br/>Interactive RAG]
    User -->|4| SlackBot[slack-bot<br/>Slack Integration]
    User -->|5| MCPServer[mcp-server<br/>Claude Desktop]

    Vectorize -->|Store| S3V[(S3 Vectors)]
    Vectorize -->|Index| OS[(OpenSearch)]

    Query --> Hybrid[Hybrid Search<br/>BM25 + Vector]
    Chat --> Hybrid
    SlackBot --> Hybrid
    MCPServer --> Hybrid

    Hybrid --> S3V
    Hybrid --> OS
    Hybrid -->|Optional| Slack[(Slack API)]

    Hybrid --> Results[Search Results]
    Results --> Answer[Generated Answer<br/>via Bedrock Claude]

    style Vectorize fill:#e1f5ff
    style Query fill:#fff4e1
    style Chat fill:#ffe1f5
    style SlackBot fill:#e1ffe1
    style MCPServer fill:#f5e1ff

Hybrid Search Flow

The hybrid search engine combines BM25 keyword matching with vector similarity search:

sequenceDiagram
    participant User
    participant CLI as CLI Command
    participant Bedrock as Amazon Bedrock
    participant OS as OpenSearch
    participant Slack as Slack API
    participant Engine as Hybrid Engine

    User->>CLI: query "検索クエリ"
    CLI->>Bedrock: GenerateEmbedding(query)
    Bedrock-->>CLI: Vector[1024]

    par BM25 Search
        CLI->>OS: BM25 Search (kuromoji tokenizer)
        OS-->>CLI: BM25 Results (200 docs)
    and Vector Search
        CLI->>OS: k-NN Search (cosine similarity)
        OS-->>CLI: Vector Results (100 docs)
    end

    CLI->>Engine: Fuse Results (RRF/weighted_sum)
    Engine-->>CLI: Fused Results (top-k)

    opt Slack Search Enabled
        CLI->>Slack: search.messages(query)
        Slack-->>CLI: Slack Messages
        CLI->>Slack: conversations.history(threads)
        Slack-->>CLI: Thread Context
        CLI->>Engine: Merge Slack Context
        Engine-->>CLI: Enriched Results
    end

    CLI-->>User: Combined Results + References

Slack Bot Processing

Slack Bot listens for mentions and responds with RAG-powered answers:

sequenceDiagram
    participant Slack as Slack Workspace
    participant Bot as Slack Bot (RTM/Socket)
    participant Detector as Mention Detector
    participant Extractor as Query Extractor
    participant Hybrid as Hybrid Search
    participant SlackSearch as Slack Search Service
    participant Formatter as Block Kit Formatter

    Slack->>Bot: @ragent-bot "質問内容"
    Bot->>Detector: Detect Mention
    Detector-->>Bot: Mention Event

    Bot->>Extractor: Extract Query
    Extractor-->>Bot: Query Text

    Bot->>Hybrid: Search(query)

    par Document Search
        Hybrid->>Hybrid: BM25 + k-NN
        Hybrid-->>Bot: Document Results
    and Slack Context Search (Optional)
        Bot->>SlackSearch: SearchSlack(query, channels)
        SlackSearch->>SlackSearch: Query Refinement Loop
        SlackSearch->>Slack: search.messages API
        Slack-->>SlackSearch: Messages
        SlackSearch->>Slack: conversations.history (threads)
        Slack-->>SlackSearch: Thread Timeline
        SlackSearch->>SlackSearch: Sufficiency Check
        SlackSearch-->>Bot: Slack Context
    end

    Bot->>Formatter: Format Results (Block Kit)
    Formatter-->>Bot: Slack Blocks

    Bot->>Slack: Post Message (with Blocks)
    Slack-->>Slack: Display Answer + References

MCP Server Integration

MCP Server exposes RAGent's hybrid search to Claude Desktop and other MCP-compatible tools:

sequenceDiagram
    participant Claude as Claude Desktop
    participant MCP as MCP Server
    participant Auth as Auth Middleware
    participant Handler as Hybrid Search Handler
    participant Bedrock as Amazon Bedrock
    participant OS as OpenSearch
    participant Slack as Slack API

    Claude->>MCP: JSON-RPC Request<br/>tools/call "hybrid_search"

    MCP->>Auth: Authenticate Request

    alt IP Authentication
        Auth->>Auth: Check Client IP
    else OIDC Authentication
        Auth->>Auth: Verify JWT Token
    else Bypass Range
        Auth->>Auth: Check Bypass CIDR
    end

    Auth-->>MCP: Authentication OK

    MCP->>Handler: Handle hybrid_search(query, options)

    Handler->>Bedrock: GenerateEmbedding(query)
    Bedrock-->>Handler: Vector[1024]

    par OpenSearch BM25
        Handler->>OS: BM25 Query
        OS-->>Handler: BM25 Results
    and OpenSearch k-NN
        Handler->>OS: k-NN Query
        OS-->>Handler: Vector Results
    end

    Handler->>Handler: Fuse Results (weighted_sum/RRF)

    opt enable_slack_search=true
        Handler->>Slack: Search Messages + Threads
        Slack-->>Handler: Slack Context
        Handler->>Handler: Merge Slack Results
    end

    Handler-->>MCP: Search Results (JSON)
    MCP-->>Claude: JSON-RPC Response<br/>{results, slack_results, metadata}

Vectorization Pipeline

The vectorize command processes source documents (Markdown, CSV, and S3 objects) and stores them in dual backends:

flowchart TD
    Start([vectorize command]) --> CheckFlags{Check Flags}

    CheckFlags -->|--clear| Clear[Delete All Vectors<br/>+ Recreate Index]
    CheckFlags -->|Normal| ScanSources
    Clear --> ScanSources

    subgraph ScanSources[Scan Data Sources]
        LocalScan[Scan Local Directory<br/>./source]
        S3Scan[Scan S3 Bucket<br/>--enable-s3]
    end

    ScanSources --> Files[List Files<br/>.md, .markdown, .csv]

    Files --> Loop{For Each File<br/>Parallel Processing}

    Loop --> FileType{File Type?}

    FileType -->|Markdown| ReadMD[Read File Content]
    FileType -->|CSV| ReadCSV[Read CSV & Expand Rows<br/>Each row → 1 document]

    ReadMD --> Extract[Extract Metadata<br/>FrontMatter Parser]
    ReadCSV --> ExtractCSV[Extract Metadata<br/>CSV Column Mapping]

    Extract --> Embed[Generate Embedding<br/>Bedrock Titan v2]
    ExtractCSV --> Embed

    Embed --> Dual{Dual Backend Storage}

    Dual --> S3Store[Store to S3 Vector<br/>with Metadata]
    Dual --> OSIndex[Index to OpenSearch<br/>BM25 + k-NN fields]

    S3Store --> Stats1[Update Stats<br/>S3: Success/Fail]
    OSIndex --> Stats2[Update Stats<br/>OS: Indexed/Skipped/Retry]

    Stats1 --> Check{More Files?}
    Stats2 --> Check

    Check -->|Yes| Loop
    Check -->|No| Report[Display Statistics<br/>Success Rate, Errors]

    Report --> Follow{Follow Mode?}
    Follow -->|Yes| Wait[Wait Interval<br/>Default: 30m]
    Wait --> ScanSources
    Follow -->|No| End([Completed])

    style Start fill:#e1f5ff
    style Clear fill:#ffe1e1
    style ScanSources fill:#e8f4f8
    style Embed fill:#fff4e1
    style Dual fill:#f5e1ff
    style End fill:#e1ffe1

Prerequisites

Prepare Source Documents

Before using RAGent, you need to prepare source documents in a source/ directory. These documents should contain the content you want to make searchable through the RAG system.

Supported file types:

  • Markdown (.md, .markdown): Each file becomes one document
  • CSV (.csv): Each row becomes one document (header row required)
  • PDF (.pdf): Each page becomes one document (requires OCR_PROVIDER=bedrock or OCR_PROVIDER=gemini)
# Create source directory
mkdir source

# Place your files in this directory
cp /path/to/your/documents/*.md source/
cp /path/to/your/data/*.csv source/

For CSV files, you can optionally provide a configuration file to specify column mappings:

# Copy example configuration
cp csv-config.yaml.example csv-config.yaml

# Run with CSV configuration
RAGent vectorize --csv-config csv-config.yaml

CSV Configuration Options

The csv-config.yaml supports the following options:

header_row (Header Row Position):

Use this option when your CSV file has metadata or summary rows before the actual header row. When header_row is specified, that row is used as the column headers, and all preceding rows are skipped.

csv:
  files:
    - pattern: "sample.csv"
      header_row: 7  # Row 7 is the header (1-indexed)
                     # Rows 1-6 are skipped, data starts from row 8
      content:
        columns: ["task", "category"]
      metadata:
        title: "task"
        category: "category"
  • If header_row is not specified, the default is 1 (first row is the header)
  • Row numbers are 1-indexed

For exporting notes from Kibela, use the separate export tool available in the export/ directory.

Required Environment Variables

Create a .env file in the project root and configure the following environment variables:

# AWS Configuration
AWS_REGION=your_aws_region
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key

# S3 Vector Configuration
S3_VECTOR_INDEX_NAME=your_vector_index_name
S3_BUCKET_NAME=your_s3_bucket_name
S3_VECTOR_REGION=us-east-1           # AWS region for S3 Vector bucket
S3_SOURCE_REGION=ap-northeast-1      # AWS region for source S3 bucket (--enable-s3)

# Vector DB Backend Selection
VECTOR_DB_BACKEND=s3            # Backend type: "s3" (Amazon S3 Vectors) or "sqlite" (local sqlite-vec) (default: s3)
SQLITE_VEC_DB_PATH=~/.ragent/vectors.db  # Path to sqlite-vec DB file (used when VECTOR_DB_BACKEND=sqlite)

# OpenSearch Configuration (for Hybrid RAG)
OPENSEARCH_ENDPOINT=your_opensearch_endpoint
OPENSEARCH_INDEX=your_opensearch_index
OPENSEARCH_REGION=us-east-1  # default

# GitHub Configuration (optional)
GITHUB_TOKEN=ghp_your_github_token  # Required for private repositories

# Chat Configuration
CHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0  # default
EXCLUDE_CATEGORIES=Personal,Daily  # Categories to exclude from search

# MCP Server Configuration
MCP_SERVER_HOST=localhost
MCP_SERVER_PORT=8080
MCP_IP_AUTH_ENABLED=true
MCP_ALLOWED_IPS=127.0.0.1,::1  # Comma-separated list

# MCP Bypass Configuration (optional)
MCP_BYPASS_IP_RANGE=10.0.0.0/8,172.16.0.0/12  # Comma-separated CIDR ranges
MCP_BYPASS_VERBOSE_LOG=false
MCP_BYPASS_AUDIT_LOG=true
MCP_TRUSTED_PROXIES=192.168.1.1,10.0.0.1  # Trusted proxy IPs for X-Forwarded-For

# OIDC Authentication (optional)
OIDC_ISSUER=https://accounts.google.com  # Your OIDC provider URL
OIDC_CLIENT_ID=your_client_id
OIDC_CLIENT_SECRET=your_client_secret

# Slack Bot Configuration
SLACK_BOT_TOKEN=xoxb-your-bot-token
# SLACK_USER_TOKEN is optional but covers more paths.
# - When set (xoxp): all entry points (slack-bot / query CLI / mcp-server)
#   use the legacy `search.messages` API and can reach private channels,
#   DMs and MPIMs that the user has access to.
# - When unset: only the slack-bot mention path can search Slack. It calls
#   the Real-time Search API (`assistant.search.context`) on the bot token
#   plus the per-event `action_token` and reaches public channels the bot
#   has not joined (requires the `search:read.public` bot scope). CLI/MCP
#   cannot obtain an `action_token` and therefore skip Slack search.
SLACK_USER_TOKEN=
SLACK_RESPONSE_TIMEOUT=5s
SLACK_MAX_RESULTS=5
SLACK_ENABLE_THREADING=false
SLACK_THREAD_CONTEXT_ENABLED=true
SLACK_THREAD_CONTEXT_MAX_MESSAGES=10

# Slack Search Configuration
SLACK_SEARCH_ENABLED=false                     # Enable Slack search pipeline (set true to activate)
SLACK_SEARCH_MAX_RESULTS=20                    # Max Slack messages retrieved per query (1-100)
SLACK_SEARCH_MAX_RETRIES=5                     # Retry count for Slack API rate limits (0-10)
SLACK_SEARCH_CONTEXT_WINDOW_MINUTES=30         # Time window (minutes) of surrounding message context
SLACK_SEARCH_MAX_ITERATIONS=5                  # Iterative refinements performed when answers insufficient
SLACK_SEARCH_MAX_CONTEXT_MESSAGES=100          # Max messages assembled into enriched context
SLACK_SEARCH_TIMEOUT_SECONDS=5                 # Slack API request timeout in seconds (1-60)

# OpenTelemetry Configuration (optional)
OTEL_ENABLED=false
OTEL_SERVICE_NAME=ragent
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=service.namespace=ragent,environment=dev
OTEL_TRACES_SAMPLER=always_on
OTEL_TRACES_SAMPLER_ARG=1.0

# OCR Configuration (for PDF vectorization)
OCR_PROVIDER=bedrock                                    # OCR provider ("bedrock" or "gemini"; omit to skip PDFs)
OCR_MODEL=anthropic.claude-3-5-sonnet-20241022-v2:0    # Model for OCR (Bedrock model ID or Gemini model name)
OCR_TIMEOUT=120s                                        # OCR request timeout (default: 120s)

# Embedding Configuration
EMBEDDING_PROVIDER=bedrock                              # Embedding provider ("bedrock" [default] or "gemini")
EMBEDDING_MODEL=                                        # Embedding model ID (optional; defaults to provider's default)
EMBEDDING_DIMENSION=                                    # Output vector dimension (optional; overrides model default)

# Gemini Configuration (OCR_PROVIDER=gemini and/or EMBEDDING_PROVIDER=gemini)
# Option 1: API key authentication
GEMINI_API_KEY=your_gemini_api_key                      # Google AI Studio API key
# Option 2: Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS)
GEMINI_GCP_PROJECT=your_gcp_project_id                  # GCP project ID for Vertex AI
GEMINI_GCP_LOCATION=us-central1                         # GCP region for Vertex AI (default: us-central1)

# AWS Secrets Manager Configuration (optional)
SECRET_MANAGER_SECRET_ID=ragent/app              # Secret ID in AWS Secrets Manager (omit to disable)
SECRET_MANAGER_REGION=us-east-1                  # AWS region for Secrets Manager (default: us-east-1)

Gemini Embedding Configuration

To use Gemini for embeddings instead of Amazon Bedrock, set EMBEDDING_PROVIDER=gemini. Two authentication methods are supported:

# Use Gemini for embeddings (with API Key)
export EMBEDDING_PROVIDER=gemini
export EMBEDDING_MODEL=text-embedding-004
export GEMINI_API_KEY=your-api-key

# Use Gemini for embeddings (with Vertex AI)
export EMBEDDING_PROVIDER=gemini
export GEMINI_GCP_PROJECT=your-project
export GEMINI_GCP_LOCATION=us-central1

When EMBEDDING_MODEL is omitted, Gemini defaults to text-embedding-004 (768 dimensions). Bedrock defaults to amazon.titan-embed-text-v2:0 (1024 dimensions).

Models such as gemini-embedding-2-preview support configurable output dimensions. Set EMBEDDING_DIMENSION to match your vector store index — for example EMBEDDING_DIMENSION=1024 for an OpenSearch index created with 1024 dimensions. When omitted, the model's default dimension is used.

Slack search requires SLACK_SEARCH_ENABLED=true and a valid SLACK_BOT_TOKEN. SLACK_USER_TOKEN (xoxp-) is optional but determines which entry points can search Slack:

  • With SLACK_USER_TOKEN — every entry point (slack-bot, query CLI, mcp-server) calls the legacy search.messages API with the user-token search:read scope. Results cover public channels, private channels the user belongs to, plus DMs / MPIMs.
  • Without SLACK_USER_TOKEN — only the slack-bot mention path can search Slack. It switches to the Real-time Search API (assistant.search.context) using the bot token together with the short-lived action_token carried by app_mention / DM message events. The bot needs the granular search:read.public scope and reaches public channels even ones the bot has not been invited to. The query CLI and mcp-server cannot obtain an action_token, so they refuse to run Slack search in this mode (other RAGent features continue to work).

The search-specific knobs let you tune throughput and cost per workspace without touching the core document pipeline. See docs/slack-user-token.md for the full rationale of when a user token is required.

MCP Bypass Configuration (Optional)

  • MCP_BYPASS_IP_RANGE: Comma-separated CIDR ranges that skip authentication for trusted networks.
  • MCP_BYPASS_VERBOSE_LOG: Enables detailed logging for bypass decisions to aid troubleshooting.
  • MCP_BYPASS_AUDIT_LOG: Emits JSON audit entries for bypassed requests (enabled by default for compliance).
  • MCP_TRUSTED_PROXIES: Comma-separated list of proxy IPs whose X-Forwarded-For headers are trusted during bypass checks.

AWS Secrets Manager Integration

RAGent supports AWS Secrets Manager as a fallback for environment variables. When configured, secrets stored in Secrets Manager are automatically injected as environment variables at startup — but only for keys that are not already set. This means existing environment variables always take priority and are never overwritten.

How It Works

  1. On startup, RAGent checks for the SECRET_MANAGER_SECRET_ID environment variable.
  2. If set, it fetches the secret from AWS Secrets Manager (the secret must be stored as a JSON key-value pair).
  3. For each key in the JSON, if the corresponding environment variable is not already set, the value is injected.
  4. If the environment variable is already set (via .env, shell export, etc.), the Secrets Manager value is ignored.

This design allows you to:

  • Store sensitive credentials (API keys, tokens) centrally in Secrets Manager
  • Override any secret locally by setting the environment variable directly
  • Use the same secret across multiple deployment environments

Configuration

Environment Variable Description Default
SECRET_MANAGER_SECRET_ID Secret ID or ARN in AWS Secrets Manager (not set — feature disabled)
SECRET_MANAGER_REGION AWS region for Secrets Manager us-east-1

Registering Secrets

Store your secrets as a JSON object in AWS Secrets Manager. Each key should match the environment variable name used by RAGent.

Using AWS CLI:

# Create a new secret
aws secretsmanager create-secret \
  --name "ragent/app" \
  --description "RAGent application secrets" \
  --secret-string '{
    "SLACK_BOT_TOKEN": "xoxb-your-bot-token",
    "SLACK_USER_TOKEN": "xoxp-your-user-token",
    "OIDC_CLIENT_SECRET": "your-oidc-secret",
    "GITHUB_TOKEN": "ghp_your-github-token",
    "GEMINI_API_KEY": "your-gemini-api-key"
  }' \
  --region us-east-1

# Update an existing secret
aws secretsmanager put-secret-value \
  --secret-id "ragent/app" \
  --secret-string '{
    "SLACK_BOT_TOKEN": "xoxb-new-token",
    "SLACK_USER_TOKEN": "xoxp-new-token",
    "OIDC_CLIENT_SECRET": "new-secret",
    "GITHUB_TOKEN": "ghp_new-token",
    "GEMINI_API_KEY": "new-api-key"
  }' \
  --region us-east-1

Using AWS Console:

  1. Open the AWS Secrets Manager Console
  2. Click Store a new secret
  3. Select Other type of secret
  4. Choose Plaintext and enter a JSON object with key-value pairs
  5. Name the secret (e.g., ragent/app) and complete the wizard

IAM Permissions

The IAM role or user running RAGent needs the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:ragent/app-*"
    }
  ]
}

Usage Example

# Set only the Secrets Manager config — all other secrets are fetched automatically
export SECRET_MANAGER_SECRET_ID=ragent/app
export SECRET_MANAGER_REGION=us-east-1

# Non-secret config still set via environment / .env
export OPENSEARCH_ENDPOINT=https://your-opensearch-endpoint
export OPENSEARCH_INDEX=your-index

# Start RAGent — SLACK_BOT_TOKEN, GITHUB_TOKEN, etc. are loaded from Secrets Manager
RAGent slack-bot

# Override a specific secret locally (takes priority over Secrets Manager)
export SECRET_MANAGER_SECRET_ID=ragent/app
export SLACK_BOT_TOKEN=xoxb-local-override-token  # This value wins

RAGent slack-bot  # Uses the local SLACK_BOT_TOKEN, other secrets from SM

Note: Only string values in the JSON are injected. Non-string values (objects, arrays, numbers, booleans, null) are silently skipped.

MCP Client Config from Secrets Manager

--mcp-config accepts either a local JSONC file path or secretsmanager://SECRET_ID. For Secrets Manager, store the raw contents of mcp-config.jsonc as the secret string, not as the environment-variable JSON map used by SECRET_MANAGER_SECRET_ID.

aws secretsmanager create-secret \
  --name ragent/mcp-config \
  --secret-string file://mcp-config.jsonc \
  --region us-east-1

RAGent query "your question" --mcp-config secretsmanager://ragent/mcp-config

This uses SECRET_MANAGER_REGION for the AWS region when set, otherwise us-east-1, and requires secretsmanager:GetSecretValue on the MCP config secret.

OpenTelemetry Observability

RAGent exposes distributed traces and usage metrics through the OpenTelemetry (OTel) Go SDK. Traces are emitted for Slack Bot message handling, MCP tool calls, and the shared hybrid search service, while counters and histograms capture request rates, error rates, and response times.

Enable OTel

Set the following environment variables (see .env.example for defaults):

  • OTEL_ENABLED: true to enable tracing and metrics (defaults to false).
  • OTEL_SERVICE_NAME: Logical service name (ragent by default).
  • OTEL_RESOURCE_ATTRIBUTES: Comma-separated key=value pairs such as service.namespace=ragent,environment=dev.
  • OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint URL (including scheme).
  • OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf (default) or grpc.
  • OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG: Configure sampling strategy (always_on, traceidratio, etc.).

When OTEL_ENABLED=false, RAGent registers no-op providers and carries no runtime overhead.

Example: Jaeger (local development)

# 1. Start Jaeger all-in-one
docker run --rm -it -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:1.58

# 2. Enable OTel before running RAGent
export OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

# 3. Run the Slack bot (or any other command)
go run main.go slack-bot
# Visit http://localhost:16686 to explore spans

Example: Prometheus via OpenTelemetry Collector

Use the OTel Collector to convert OTLP metrics into Prometheus format:

# collector.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

otelcol --config collector.yaml
export OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
go run main.go mcp-server
# Scrape metrics at http://localhost:9464/metrics

Example: AWS X-Ray with ADOT Collector

# 1. Run the AWS Distro for OpenTelemetry collector
docker run --rm -it -p 4317:4317 public.ecr.aws/aws-observability/aws-otel-collector:latest

# 2. Configure RAGent to send spans over gRPC
export OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.2

Metrics & Span Names

  • Spans
    • slackbot.process_message
    • mcpserver.hybrid_search
    • search.hybrid
  • Metrics
    • ragent.slack.requests.total, ragent.slack.errors.total, ragent.slack.response_time
    • ragent.mcp.requests.total, ragent.mcp.errors.total, `ragent.mcp.response_time