Vidilearn

Vidilearn is a local-first, AI-native universal knowledge ingestion and semantic hybrid retrieval engine. It operates completely offline with zero API costs, delivering sub-100ms hybrid searches over local documents, video transcripts, web pages, and RSS feeds.

✨ Features

  • Universal Ingestion: Supports YouTube video captions, PDFs, DOCX files, Markdown/text documents, RSS feeds, and local folder directories.
  • High-Speed Vector Search: Powered by sqlite-vec (compiled native C++ distance logic), querying 100K+ vector chunks in under 200ms.
  • Hybrid BM25 + Semantic Retrieval: Fuses virtual text match ranking (FTS5) with vector similarity (ANN) using Reciprocal Rank Fusion (RRF) and dynamic query weighting.
  • Neural Reranker: Optimizes results using a batched local cross-encoder model (ms-marco-MiniLM-L-6-v2) with startup session warm-up.
  • Local AI Synthesis: GeneratesCornell study notes, quizzes ( Obsidian/Anki TSV formats), and summaries (bullet, twitter-thread, blog, notes, podcast-recap) using local Ollama models.
  • Concurrency Protected: Protects Node event loop threads from thundering herd locks using memory-safe LRU caching and single-flight request coalescing.

📦 Installation

Install globally via npm:

npm install -g vidilearn

🚀 Quick Start

Ingest a document or YouTube video:

vidilearn ingest https://www.youtube.com/watch?v=sal78ACtGTc

Perform a hybrid search over ingested knowledge:

vidilearn search "agentic workflows design patterns" --hybrid

Generate Cornell study notes, flashcards, and quizzes:

vidilearn study https://www.youtube.com/watch?v=sal78ACtGTc

Analyze video transcript density for clips hooks:

vidilearn clips https://www.youtube.com/watch?v=sal78ACtGTc

🛠️ Architecture

graph TD
  A[YouTube / PDF / DOCX / Folder / RSS] -->|Ingest & Chunk| B[Embedding Pipeline: all-MiniLM-L6-v2]
  B -->|Normalized Vector BLOB| C[SQLite Database]
  C -->|Native C++ Indexing| D[vec_chunks table: sqlite-vec]
  C -->|Text Matching| E[chunks_fts table: FTS5]
  D -->|Semantic Matcher| F[RRF Hybrid Fusion]
  E -->|BM25 Matcher| F
  F -->|Top Candidates| G[Batched Neural Reranker: ms-marco-MiniLM-L-6-v2]
  G -->|Filtered & Ranked Results| H[CLI / MCP / AI Study Outputs]

📊 Scale Benchmarks (Real Measured Telemetry)

Tested on 100,000 synthetic chunks (~205 MB Database):

Metric Measured Value Target Status
Embedding Throughput 1433 chunks/min > 500 chunks/min ✅ PASSED
Search Latency 53.5ms < 100ms ✅ PASSED
First Search (Cold Boot) 336.9ms < 400ms ✅ PASSED
RAM Idle Footprint 64.0 MB < 300MB ✅ PASSED
RAG Precision Accuracy 100% (3/3) 100% ✅ PASSED

📋 Commands Reference

Command Description
vidilearn ingest <target> Ingest target document, RSS feed, local folder, or URL into memory
vidilearn search <query> Query database using RRF hybrid FTS5 and semantic vector search
vidilearn study <target> ExportCornell notes, flashcards, and quizzes to Anki TSV/Obsidian MD
vidilearn clips <url> Identify top pacing and hook clip timestamps with deep links
vidilearn summarize <target> Generate local summaries (blog, notes, twitter thread, podcast recap)
vidilearn graph Generate knowledge graph Mermaid flowcharts linking documents & entities
vidilearn doctor Check database schema, corruptions, duplicate records, and diagnostics
vidilearn audit Verify chunks hash duplicate detections
vidilearn evaluate Evaluate precision, recall, and cross-domain leakage
vidilearn trace <id> Trace chunk source text, document link, and metadata by UUID
vidilearn metrics Print physical database file sizes, chunk counts, and memory telemetry
vidilearn mcp-server Start stdio Model Context Protocol (MCP) server

🔒 Local-First Philosophy

Vidilearn runs 100% on your machine. It requires no external API keys, collects no user search history, and makes no network telemetry calls. All embeddings, text parsing, database index construction, and cross-encoder reranking operations execute locally inside the package runtime environment. For advanced AI reasoning or generation, it connects to your local Ollama instance, ensuring complete data privacy.