myKG — Knowledge Graph Extractor

Python 3.11+ License: MIT CI codecov Providers PyPI version Downloads GitHub Stars GitHub Issues Visitors Medium LinkedIn

myKG automatically generates a confidence-scored knowledge graph from a set of mixed documents — Markdown, plain text, PDF, Word, PowerPoint, Excel, HTML, and images — grounded in an induced RDFS/OWL ontology.

Contents

Features

MyKG builds trustworthy knowledge graphs through a self-evolving ontology that continuously adapts, maintains consistency, and assigns confidence scores to knowledge, keeping information grounded and reliable as it grows.

Ontology-Guided Extraction

  • Schema-guided knowledge graph generation — the extracted graph is always grounded in a formal RDFS/OWL schema: concept types, property names, domain/range constraints, and the is-a hierarchy are explicit and inspectable before any entity is extracted
  • Bring your own ontology — supply a --base-schema TTL file (RDFS or OWL) to lock in classes and properties from an existing formal ontology; the LLM expands it with domain-specific concepts but will not rename, remove, or contradict your authoritative vocabulary.Please note that this mechanism is contolled by the LLM, and may not be strictly enforced. Add --freeze-schema to skip LLM schema induction entirely and extract from the documents strictly against your ontology verbatim — no surprise types, no invented properties
  • SKOS thesaurus support — pass --thesaurus to load a SKOS vocabulary; skos:exactMatch terms are collapsed silently, skos:closeMatch terms trigger a warning — giving the schema merger richer synonym awareness than string matching alone
  • Verifiable TTL ontology — after Pass 1, the induced schema is exported as a valid RDFS/OWL Turtle file (intermediate/schema.ttl) that can be opened directly in ontology editors such as Protégé. The TTL is validated by rdflib (syntax + semantic checks: domain/range refer to declared classes, no conflicting ranges) before any extraction begins
  • Human-in-the-loop ontology design — pause after schema induction with --review, edit the schema, and resume extraction. Edit schema.json directly or refine schema.ttl in Protégé and feed it back with --freeze-schema
  • Incremental updates — append new files to an existing session, extracting only what changed. Optionally grow the schema from new documents while preserving existing concepts and properties
  • AI coding assistant friendly — designed for smooth use alongside AI coding assistants such as Claude Code; run extractions, inspect outputs, and iterate on your knowledge graph without leaving your coding environment; see Using mykg with Claude Code
  • Second brain for AI coding assistants — the Obsidian vault output turns your extracted knowledge graph into a directory of wikilinked Markdown notes that any AI coding assistant can read as project context; point Claude Code, Cursor, or Copilot at output/obsidian_vault/ and ask questions, trace relationships, and get answers grounded in your own documents
  • MCP server for desktop AI apps — run mykg mcp-serve to expose your knowledge graph via the Model Context Protocol; integrates with Claude Desktop, Cherry Studio, and any MCP-compatible client — 13 query tools let LLMs search entities, explore relationships, find paths, traverse the graph, and read wiki notes directly from your extracted knowledge; see MCP Server

Input

  • Mixed-format corpora — point mykg extract-graph at any directory; supported extensions are converted to Markdown automatically before ingest:

    Format Extensions Backend
    Markdown .md passthrough (consumed as-is)
    Plain text .txt renamed to .md in-process
    PDF, Word, PowerPoint, Excel, images .pdf .docx .doc .pptx .xlsx .png .jpg .jpeg MinerU in an ephemeral uv-managed Python 3.12 venv — nothing is installed into your active environment
    HTML .html .htm markdownify in-process; anchors and image tags stripped
    Websites, GitHub repos any URL Crawlee in an ephemeral uv venv — produces an mykg_web_fetch/ folder

    Anything outside the allowlist (e.g. .svg, .css, .php assets next to an HTML bundle) is logged and skipped, never silently dropped. The allowlist is configurable via preprocess.extensions in mykg_config.yaml.

    Incremental conversion — unchanged source files are skipped on re-run. Adding one PDF to a corpus and re-running only re-converts that PDF. Force a full re-conversion with mykg extract-graph --from-step preprocess.

Graph & Output

  • Provider-agnostic — works with Anthropic (Claude), OpenAI (GPT), Ollama (local), OpenRouter, or the claude CLI
  • Five output families — JSONL for Neo4j/NetworkX/RAG, Turtle RDF for OWL toolchains, NetworkX multi-format for graph analysis, Obsidian vault for linked personal knowledge management, and an optional Neo4j LOAD CSV bundle (plain-header CSVs + paste-and-run Cypher script for Neo4j Browser / cypher-shell)
  • Obsidian vault — second brain for AI coding assistants — every extracted entity becomes a wikilinked Markdown note in output/obsidian_vault/; open it in Obsidian to navigate the graph with backlinks and Graph View, or point your AI coding assistant (Claude Code, Cursor, Copilot) at the vault folder so it can answer questions, trace relationships, and reason over your knowledge base in natural language
  • Interactive HTML graph — node/edge filtering, search, hover popups; opens directly in a browser
  • Confidence scoring — every extracted attribute, node, and edge carries a 0.0–1.0 confidence score
  • Name normalization — surface-form variants ("Acme Corp", "ACME", "Acme Corporation") resolved to a single canonical node with aliases
  • Orphan-connection pass — reconnects isolated nodes via co-occurrence heuristic + LLM confirmation
  • Cross-session merge — combine two independently-produced graphs into one unified knowledge graph
  • Resumable pipeline — every stage persists intermediate state; re-enter at any step after a crash or edit
  • Session isolation — each run is fully self-contained; inputs, intermediate state, outputs, and logs co-located
  • Query knowledge graph — natural-language queries directly against the extracted graph via AI coding assistants such as Claude Code.

Command line

All Commands

Command Purpose
mykg init Interactive setup wizard — writes mykg_config.yaml and .env.mykg
mykg extract-graph Run the two-pass extraction pipeline over a directory
mykg approve-schema Write schema_approved.flag to unblock the human_review gate after editing schema.json (used with --review, see Human Review Gate)
mykg walkthrough Regenerate walkthrough.md for an existing session
mykg merge-graphs Merge two independently-produced sessions into one unified graph
mykg parse-docs Standalone MinerU/markdownify document-to-Markdown conversion
mykg fetch-web Crawl a website or clone a GitHub repo into an extract-graph-ready folder
mykg mcp-serve Start the MCP server exposing read-only graph query tools 
mykg extract-graph my_notes/        # any directory: .md, .txt, .pdf, .docx, .html, images

It uses a two-pass LLM pipeline: Pass 1 induces a global RDFS/OWL schema from your document corpus; Pass 2 extracts typed entity and relationship instances per file against that schema. Non-Markdown inputs (.txt .pdf .docx .doc .pptx .xlsx .png .jpg .jpeg .html .htm) are converted to Markdown automatically before extraction. The result is exported to multiple formats: JSONL for property-graph consumers such as Neo4j, Turtle RDF for OWL toolchains, seven NetworkX formats for graph analysis and visualization, an Obsidian vault — a second brain of wikilinked Markdown notes your AI coding assistant (Claude Code, Cursor, Copilot) can read and reason over directly — and optionally a Neo4j LOAD CSV bundle with a paste-and-run Cypher script for one-step import into Neo4j Browser or cypher-shell.

Quick Start

Requires Python 3.11+ (developed on macOS; automated CI runs the test suite on Ubuntu and Windows), and one of: an Anthropic/OpenAI/OpenRouter API key, Ollama running locally, or the claude CLI.

Install from PyPI

Install mykg, then run the interactive setup wizard — it asks for your provider, model, and API key and writes mykg_config.yaml and .env.mykg in one step.

pip install mykg
mykg init

Then extract a knowledge graph from your notes:

mykg extract-graph my_notes/

Open mykg_sessions/<timestamp>/output/knowledge_graph.html in your browser to explore the result.

Install from source

Install uv, clone the repo, sync dependencies, run the setup wizard, then extract.

git clone https://github.com/SenolIsci/mykg && cd mykg
uv sync && uv run mykg init --force

Then extract a knowledge graph from your notes:

uv run mykg extract-graph my_notes/

For Ollama (local inference, no API key needed), pull a model and select the ollama-local profile when mykg init prompts you.

ollama pull llama3.3
mykg init
mykg extract-graph my_notes/

Articles & Tutorials

Walkthroughs and case studies on Medium:

Configuration

All configuration lives in a single mykg_config.yaml file discovered automatically from the working directory (or any parent). There are no hardcoded defaults in the code — the YAML is the sole source of truth.

mykg init           # interactive: choose provider, model, paste API key
                    # writes mykg_config.yaml and .env.mykg in one step
mykg init --force   # overwrite an existing config
mykg init --profile openrouter-free --model google/llama-4-maverick --api-key sk-or-...  # non-interactive

The wizard walks you through three prompts:

  1. Profile — choose your LLM provider (OpenRouter, Anthropic, OpenAI, Ollama, Claude CLI, or Agent / Claude Code skill)
  2. Model — accept the default or type any model slug for that provider (skipped in agent mode — the host Claude Code session is the LLM)
  3. API key — paste your key (skipped for Ollama, Claude CLI, and agent mode)

LLM Providers

Provider Profile name API key env var Notes
Anthropic (Claude) anthropic-claude ANTHROPIC_API_KEY Recommended for quality
OpenAI openai OPENAI_API_KEY
Ollama ollama-local — Local inference, no key needed
OpenRouter openrouter-free OPENROUTER_API_KEY Access many models via one key
Claude CLI claude-cli — Uses claude -p subprocess; serial only
Agent (Claude Code skill) agent-claude-code — LLM answers come from a Claude Code skill via filesystem inbox/outbox — see docs/agent-mode.md

Switch provider by setting profile: at the top of mykg_config.yaml.

Token Budgets

Each profile's llm: and pipeline: blocks carry a chain of token-budget values sized for that model's context window:

  • llm.context_window — the model's total context limit
  • llm.max_output_tokens — the output cap reserved for each LLM response
  • pipeline.pass1.batch_token_target and pipeline.pass2.concat_batch_token_target / batch_token_target — input budget per LLM call, sized to (context_window − max_output_tokens) × 0.95
  • pipeline.chunking.window_tokens / overlap_tokens — chunk size and overlap for splitting large files, sized to roughly batch_token_target / 4 and window_tokens × 0.10

The shipped values are tuned per profile (e.g. claude-cli/anthropic-claude assume a 200K context window, openrouter-free/ollama-local assume 64K). If you switch to a different model — especially on ollama-local or openrouter-free — check that model's actual context window and rescale these values, otherwise window_tokens + max_output_tokens may exceed what the model can actually handle, causing truncated or failed responses.

Use the bundled context-calculator tool to recompute the chain for a new model:

# Compute a full token-budget chain from context window + output cap:
context-calculator --context 64000 --max-output 32000

# Or measure your actual corpus and suggest values for the active profile:
context-calculator --from-config --input-dir my_notes/

--from-config reads the active profile from mykg_config.yaml, measures token counts across your input files, and writes suggested values to mykg_config_candidate.yaml for review before copying them into mykg_config.yaml.

Diagnosing Truncated/Rejected LLM Calls

On JSON parse or validation errors in Pass 1/2, check run.log for a mykg.llm.retry warning just above the failure — always logged, regardless of logging.llm_log:

  • output truncated (finish_reason=...) — hit max_output_tokens. Lower pipeline.chunking.window_tokens or raise llm.max_output_tokens, then recompute with context-calculator (see Token Budgets).
  • context length exceeded, request rejected: ... — input didn't fit the context window. Lower batch_token_target (pipeline.pass1/pipeline.pass2).

Pass 2 Prep Mode (pass2.prep_mode)

Controls how source files are packed into Pass 2 LLM calls (set per profile in mykg_config.yaml):

  • batch_chunks (default) — packs chunks across files into token-bounded batches, ignoring file boundaries; best throughput and extraction density.
  • concat — merges whole small files into directory-grouped batches (one LLM call per window) for cross-file context.
  • per_file — one file per extraction unit; cleanest provenance (every entity traces to one source file).

See docs/architecture.md for the full comparison.

Hitting API Rate Limits (HTTP 429)

If you see repeated 429 errors during pass1, pass2, or the orphan-connection pass, your account's requests-per-minute limit is lower than the number of concurrent calls mykg is making. Each profile sets these independently under pipeline::

  • pass1.max_workers — concurrent schema-induction batch calls
  • pass2.max_workers — concurrent per-file extraction calls
  • orphan_pass.max_workers — concurrent orphan-connection calls

Lower these (e.g. from 8 down to 2–4) in the active profile to reduce concurrent requests. This is especially likely on openrouter-free (free-tier models have very low per-minute caps) and on lower-tier anthropic-claude/openai accounts. llm.retry_429_max / llm.retry_429_base_delay control automatic backoff on a 429, but a persistent 429 is a signal to reduce max_workers, not just retry harder. claude-cli is unaffected — it is serial by design (max_workers: 1); it doesn't hit API rate limits since there's no API call. agent-claude-code is not API rate-limited either (no API key involved), but it is not serial — its default profile sets pass1/pass2/orphan_pass max_workers > 1 (configurable, like any other profile), since the skill dispatches multiple subagents per wave.

API Keys

myKG reads API keys from environment variables. Set them by exporting directly or by creating a .env.mykg file in your project directory (loaded automatically on startup).

Option A — export in your shell:

export ANTHROPIC_API_KEY=sk-ant-...

Option B — create a .env.mykg file:

# .env.mykg
ANTHROPIC_API_KEY=sk-ant-...

For source installs you can also copy sample.env.mykg to .env.mykg as a starting template.

Extract Pipeline

Reads a directory of mixed format files and produces a typed knowledge graph in three output formats. The pipeline runs 12 sequential steps; all intermediate state is persisted so any step can be re-entered without repeating upstream work.

Running

mykg extract-graph <input_dir> [OPTIONS]
# source installs: uv run mykg extract-graph <input_dir> [OPTIONS]

<input_dir> is any directory containing your source files. Subdirectories are included recursively. Only files matching the configured extensions are copied into the session:

  • .md — always included (the pipeline's native format)
  • All extensions listed under preprocess.extensions in mykg_config.yaml (.pdf, .docx, .doc, .pptx, .xlsx, .png, .jpg, .jpeg, .html, .htm, .txt by default)

Everything else (.py, .json, .yaml, lock files, etc.) is ignored. Hidden directories (.venv, .git, etc.) and the sessions folder are also excluded automatically, so you can safely point extract-graph at the project root or any parent directory.

Options

Option Description
--session NAME Resume an existing session by folder name
--from-step NAME Delete a step's outputs and re-run from that point
--review Pause after Pass 1 for manual schema review
--append Skip Pass 1; re-run only on new/modified files
--append-with-grow-schema Like --append, but runs a locked Pass 1 over changed files to expand the schema
--pass1-schema-induction-only Run every step before Pass 2 (through schema_flatten), then stop — inspect/edit the schema before extracting
--pass2-kg-extraction-only Skip schema induction (requires an existing schema) and extract the full corpus through validate_graph. Unlike --from-step pass2, always re-derives flattened_schema.json first, so a hand-edited schema is picked up
--workers N Parallel workers for Pass 2
--confidence-agg mean|max Confidence aggregation when deduplicating
--base-schema PATH Locked TBox TTL file (locked classes/properties cannot be changed by the LLM)
--freeze-schema Use --base-schema verbatim: skip Pass 1 LLM induction entirely
--thesaurus PATH SKOS TTL thesaurus for synonym resolution in schema merge
--obsidian-vault Force Obsidian vault export for this run (overrides config)
--neo4j-csv Force Neo4j LOAD CSV bundle export for this run (overrides config)
--log-file PATH Write logs here (relative paths placed inside the session folder)
--verbose / -v Enable DEBUG-level logging

Examples

# New run — auto-creates a timestamped session
mykg extract-graph my_notes/

# Resume a session with 4 parallel Pass 2 workers
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --workers 4

# Pause for schema review after Pass 1
mykg extract-graph my_notes/ --review
# → edit mykg_sessions/<name>/intermediate/schema.json
mykg approve-schema --session 2026-05-17T18-31-07
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --review

# Re-run from assembly onward (reuses existing extractions)
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --from-step assemble

# Lock a base ontology so the LLM won't rename its classes
mykg extract-graph my_notes/ --base-schema ontology/core.ttl

# Induce and inspect the schema first, extract in a second invocation
mykg extract-graph my_notes/ --pass1-schema-induction-only
# → review/edit mykg_sessions/<name>/intermediate/schema.json
mykg extract-graph my_notes/ --session <name> --pass2-kg-extraction-only

Sessions

Every run automatically creates an isolated session folder:

mykg_sessions/
  2026-05-17T18-31-07/
    input/           ← archived copy of all input Markdown files
    intermediate/    ← all intermediate pipeline state
    output/          ← final outputs (JSONL, TTL, HTML, NetworkX)
    run.log          ← log file
    walkthrough.md   ← post-run report

Sessions are the primary unit of resumability. Pass --session <name> to resume from the last completed step. Pass --from-step <step> to force-restart from a specific point.

The sessions root is configurable via pipeline.paths.sessions_dir (default: mykg_sessions/ in the current directory).

Pipeline Steps

The pipeline runs 12 steps in sequence. All intermediate state is written to disk so any step can be re-entered without repeating upstream work.

# Step LLM Key outputs
1 preprocess — preprocess.done, preprocess_manifest.json, files under input/_preprocessed/ (routes non-md inputs to MinerU, markdownify, or rename; no-op for pure Markdown corpora)
2 ingest — file_manifest.json
3 pass1 ✓ (3 calls) schema.json, schema.ttl, schema_history/, pass1_batch_selection.json, pass1_batch_proposals/
4 schema_validate — schema_validate.done
5 human_review — schema_approved.flag (only with --review)
6 schema_flatten — flattened_schema.json
7 pass2 ✓ raw_extractions.json, chunk_node_index.json
8 normalize_names ✓ name_normalization.json
9 assemble — edge_metadata.json, nodes.json, merge_log.json
10 orphan_score — orphan_candidates.json
11 orphan_connect ✓ orphan_connections.json, orphan_log.json
12 validate_graph — nodes.jsonl, edges.jsonl, knowledge_graph.ttl, knowledge_graph.html, networkx_output/, obsidian_vault/, neo4j_csv/ (optional)

Pass 1 internally runs four sequential stages: parallel batch induction → algorithmic merge → harmonization LLM call → quality review LLM call. pass1.max_schema_proposals (default 50) caps the number of batches dispatched to the LLM; set to -1 to dispatch all batches on large corpora.

Outputs

Property Graph (JSONL)

nodes.jsonl — one JSON line per entity:

{
  "id": "person-alice",
  "type": "Person",
  "confidence": 0.94,
  "source_files": ["team.md"],
  "attributes": {
    "name":  {"value": "Alice",          "confidence": 1.0},
    "email": {"value": "[email protected]", "confidence": 0.88}
  },
  "aliases": ["Alice Smith", "A. Smith"]
}

edges.jsonl — one JSON line per relationship:

{
  "id": "works_at-abc123",
  "type": "works_at",
  "from": "person-alice",
  "to": "org-acme-corp",
  "confidence": 0.96,
  "method": "llm_extraction",
  "attributes": {
    "role":       {"value": "Engineer", "confidence": 0.91},
    "start_date": {"value": null,       "confidence": 0.0}
  }
}

Missing attributes are never dropped — they are represented as {"value": null, "confidence": 0.0}.

The method field distinguishes edges extracted by Pass 2 (llm_extraction) from edges inferred by the orphan pass (orphan_inferred).

RDF / OWL (Turtle)

knowledge_graph.ttl — pure RDFS/OWL triples, no edge metadata:

@prefix ex: <http://mykg.local/schema/> .
@prefix :   <http://mykg.local/data/> .

ex:Person  a rdfs:Class .
ex:works_at  rdfs:domain ex:Person ;  rdfs:range ex:Organization .

:person-alice  a ex:Person ;  rdfs:label "Alice" .
:person-alice  ex:works_at  :org-acme-corp .

Load in Protégé, query with SPARQL (Fuseki, GraphDB), or reason with HermiT/Pellet.

Interactive HTML

knowledge_graph.html — self-contained D3.js force-directed graph. Open in any browser, no server required. Supports:

  • Filter nodes and edges by type
  • Filter by confidence threshold
  • Search by name
  • Hover popups with full attribute values
  • Resizable sidebar

NetworkX Formats (networkx_output/)

File Format Best for
knowledge_graph.graphml GraphML yEd, Gephi, Cytoscape
knowledge_graph.gexf GEXF Gephi native (rich metadata)
knowledge_graph.json JSON node-link D3.js, Sigma.js, web apps
knowledge_graph.gml GML Human-readable inspection
knowledge_graph.net Pajek Network analysis
edges_nx.txt Edge list Text pipelines
adjacency.txt Adjacency list Topology consumers

Node/edge attributes are exported as attr_<name>_value / attr_<name>_confidence scalar pairs for GML compatibility.

Obsidian Vault (obsidian_vault/)

One .md note per extracted entity, grouped into subdirectories by concept type. Each note has YAML frontmatter (id, type, confidence, sources), an attributes section, outgoing and incoming wikilink relationship sections, and a source files list. An index.md at the vault root summarizes node counts per type with links to every entity.

Open output/obsidian_vault/ as a vault in Obsidian to get Graph View, backlink navigation, and full-text search across the extracted entities.

Neo4j LOAD CSV Bundle (neo4j_csv/)

Optional, off by default. Enable with --neo4j-csv on the command line, or set pipeline.export.neo4j_csv_enabled: true in mykg_config.yaml.

When enabled, step_validate_graph writes a self-contained Neo4j import bundle next to the other outputs:

output/neo4j_csv/
  nodes_<Label>.csv             ← one per concept type (Person, Organization, …)
  relationships_<TYPE>.csv      ← one per property (WORKS_AT, KNOWS, …)
  import_browser.cypher         ← paste-and-run for Neo4j Browser
  import_shell.cypher           ← for `cypher-shell -f`
  README.md                     ← bundle-local quick-reference

Plain CSV headers (id,name,name_confidence,_node_confidence,_parents,_source_files,...) — no :ID / :LABEL decorations, so the same files work for both Neo4j and any other CSV-aware tool.

Two ways to import the bundle:

# Flow A — Neo4j Browser
# 1. Copy *.csv into your DBMS's import/ directory
# 2. Paste the contents of import_browser.cypher and press play

# Flow B — cypher-shell (set dbms.security.allow_csv_import_from_file_urls=true first)
cypher-shell -u neo4j -p <pw> -f output/neo4j_csv/import_shell.cypher

Both scripts use idempotent MERGE against a _MykgNode uniqueness constraint, so re-running updates the graph in place. Requires Neo4j 5+. No Python driver, no plugin, no APOC — the scripts use only core Cypher.

See Neo4j LOAD CSV Export below for configuration details and the standalone CLI fallback.

Re-running from a Specific Step

Use --from-step to delete a step's outputs and all downstream outputs, then re-run from that point.

SESSION=2026-05-17T18-31-07

# Re-run from Pass 2 (reuse the existing schema)
mykg extract-graph my_notes/ --session $SESSION --from-step pass2

# Re-run only assembly + export (reuse raw extractions)
mykg extract-graph my_notes/ --session $SESSION --from-step assemble

# Re-run both orphan stages
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_score

# Orphan LLM pass only — full clean sweep
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_connect_fullsweep

# Orphan LLM pass only — additive (preserves prior confirmed edges)
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_connect_incremental

# Re-run only merge/harmonize/quality-review (reuse existing Pass 1 batch proposals)
mykg extract-graph my_notes/ --session $SESSION --from-step merge_proposals

Five re-entry patterns:

Pattern When to use Command
A — Schema changed Wrong concept types, missing properties Edit schema.json → approve-schema → --from-step pass1
A (merge-only) Want to re-run merge/harmonize with a different --thesaurus/--base-schema, without re-paying for Pass 1's schema-induction LLM calls --from-step merge_proposals (requires pass1_batch_proposals/ from a prior run; unlike plain --from-step pass1, does not delete it)
B — Extraction errors LLM missed entities or invented edge types Edit shard in raw_extractions_shards/ → --from-step pass2
C — Assembly errors Bad dedup decisions in merge_log.json Edit raw_extractions.json → --from-step assemble
D — Orphan pass Wrong candidates or confirmations --from-step orphan_score or orphan_connect_fullsweep

Orphan-Connection Pass

After assembly, nodes with zero edges are "orphans" — present in the graph but unreachable by traversal. The orphan pass reconnects them in two stages:

Stage 1 — orphan_score (no LLM): Uses chunk_node_index.json to find nodes that co-occur in the same source chunk as each orphan. Candidates are scored by co-occurrence frequency and filtered by schema type compatibility. Written to orphan_candidates.json.

Stage 2 — orphan_connect (LLM): One LLM call per source chunk. The prompt includes the full chunk text, all orphan IDs from that chunk, co-occurring connected nodes, and all schema properties. Confirmed edges carry "method": "orphan_inferred" and are merged directly into edge_metadata.json.

Unconnectable orphans (no resolvable source chunk) are logged as orphan_unconnectable advisory events in orphan_log.json.

Configure via pipeline.orphan_pass.* in mykg_config.yaml. Disable entirely with pipeline.orphan_pass.enabled: false.

Advanced Options

Human Review Gate (--review)

Pause after Pass 1 to inspect and edit the induced schema before Pass 2 runs:

mykg extract-graph my_notes/ --review
# → pipeline halts; edit mykg_sessions/<name>/intermediate/schema.json
mykg approve-schema --session <name>
mykg extract-graph my_notes/ --session <name> --review   # resumes from Pass 2

Locked Base Schema (--base-schema)

Lock certain classes and properties so the LLM cannot rename, remove, or restructure them:

mykg extract-graph my_notes/ --base-schema ontology/base.ttl

Both RDFS and OWL ontologies are accepted — owl:Class, owl:ObjectProperty, and owl:DatatypeProperty are recognized alongside their RDFS equivalents, so you can use an OWL ontology exported from Protégé directly without conversion. Locked entries can still receive additional attributes proposed by the LLM. Near-duplicate LLM proposals are collapsed into the locked entry with a warning.

Frozen Schema (--freeze-schema)

Use the base schema verbatim — skip Pass 1 LLM induction entirely. The LLM extracts instances against exactly the classes and properties you provide, with no additions:

mykg extract-graph my_notes/ --base-schema ontology/base.ttl --freeze-schema

This is useful when you have a complete ontology and want strict extraction — no LLM-invented types, no surprise properties. Only the concepts and relationships in your TTL will appear in the output. Requires --base-schema. Mutually exclusive with --append and --append-with-grow-schema.

Comparison with --base-schema alone:

Behavior --base-schema --base-schema --freeze-schema
Pass 1 LLM calls 3 (batch + harmonize + quality review) 0 (skipped)
Schema Locked entries + LLM-induced additions Exactly the TTL file
New concept types LLM can add them None
New properties LLM can add them None

SKOS Thesaurus (--thesaurus)

Resolve near-duplicate concept names during schema merge using a SKOS vocabulary:

mykg extract-graph my_notes/ --thesaurus ontology/terms.skos.ttl
  • skos:exactMatch → silent collapse
  • skos:closeMatch → collapse with warning in merge_log.json
  • skos:broader / skos:narrower → advisory hints only

Website / Repo Fetching (mykg fetch-web)

Crawl a website, or shallow-clone a GitHub repo, into a folder that's a ready-made extract-graph input:

# Crawl a website (same-domain, robots.txt-respecting)
mykg fetch-web https://example.com
mykg extract-graph ./mykg_web_fetch/example.com/

# Shallow-clone a GitHub repo (git clone --depth 1, no Crawlee/venv)
mykg fetch-web https://github.com/SenolIsci/mykg
mykg extract-graph ./mykg_web_fetch/github.com_SenolIsci_mykg/input/

# Fetch multiple seeds (one URL per line; mix of sites and GitHub repos)
mykg fetch-web --url-list urls.txt --output ./mykg_web_fetch/batch/
  • GitHub URLs (https://github.com/<owner>/<repo>) are detected automatically and routed to git clone — no Crawlee, no venv.
  • Everything else is crawled with Crawlee inside an ephemeral uv venv (same pattern as the MinerU venv used by preprocess), respecting robots.txt, --max-pages, --max-depth, and a configurable request delay/concurrency.
  • Resumable — fetch_manifest.json records a SHA-256 per page; re-running skips unchanged pages. --force re-fetches everything.
  • Output dir defaults to ./<fetch.output_dir>/<seed-domain>/ (configurable via fetch.output_dir in mykg_config.yaml, default mykg_web_fetch); --output overrides it and is required with --url-list.
  • Downloaded assets (PDFs, images, etc.) are filtered through the same preprocess.extensions allowlist used by extract-graph, so anything Crawlee saves is something preprocess already knows how to convert.

All knobs live under fetch: in mykg_config.yaml — see docs/architecture.md for the full crawl/clone sequence diagrams. Run mykg fetch-web --help for the complete flag list.

From Claude Code, the /mykg skill handles fetch requests in plain English — no flags to remember:

/mykg fetch https://example.com and extract
/mykg download the repo: https://github.com/SenolIsci/mykg
/mykg fetch these urls: <url1> <url2> ... and extract

All work: the skill picks the right fetch-web invocation (single page, GitHub clone, or --url-list batch with an auto-generated temp file for inline URLs), runs it, and — for the "and extract" intents — chains straight into extract-graph on the fetched output (one fresh session per seed for multi-seed fetches), confirming with you before the LLM-bearing extraction step.

Standalone Document Conversion (mykg parse-docs)

extract-graph already converts non-Markdown inputs (PDF, DOCX, images, …) to Markdown automatically via the preprocess step — on both the initial run and on --append. You only need parse-docs when you want to convert documents on their own, without running a pipeline or creating a session: inspecting MinerU output, building a Markdown corpus to commit, or feeding another tool.

# Convert a single file
mykg parse-docs --input report.pdf --output ./md/

# Convert every non-.md file under a directory (recursive; structure preserved)
mykg parse-docs --input raw_docs/ --output ./md/

# Convert only specific files (relative to --input; repeatable)
mykg parse-docs --input raw_docs/ --output ./md/ --file a.pdf --file sub/b.docx
  • MinerU in an ephemeral venv — conversion runs MinerU inside a throwaway uv-managed venv that is built per invocation and deleted on exit; nothing is installed into mykg's own interpreter. The multi-GB install is paid once per call and reused across every file in that call.
  • Extension allowlist — candidate files are filtered through preprocess.extensions from mykg_config.yaml (the same allowlist extract-graph uses). .html/.htm are always hard-skipped (MinerU cannot convert HTML — use extract-graph, which routes HTML through markdownify). Pass --no-filter to send every non-.md file to MinerU regardless of suffix.
  • **Large cor