Dense Mem

Dense-Mem is a standalone HTTP MCP memory server using Streamable HTTP. It stages exact evidence, derives semantic state through validated server policy, and returns active evidence contexts with graph-shaped Relationship handles. PostgreSQL is the durable authority for knowledge, lifecycle, provenance, search, authorization, and audit; Redis is coordination only. A single-node deployment may use process-local coordination; a multi-instance deployment requires Redis or an equivalent distributed coordination implementation.

The host LLM owns conversation and judgment. Dense-Mem owns durable evidence, owner authorization, lifecycle events, support eligibility, and bounded recall. The external memory automation contract is MCP at /mcp; browser routes are first-party interfaces, not an alternative public automation API.

Dense-Mem is part of the research preprint Governed Enterprise AI Memory Beyond RAG: From Vector Retrieval to Permissioned Knowledge Graphs.

Try the Hosted Demo

Create a temporary isolated team at https://demo-dense-mem.markhuang.ai to test disposable data before self-hosting.

Why Dense-Mem

• Evidence is exact, durable, and append-only. A lifecycle action changes its effective state without deleting provenance or trace lineage.
• Entity and typed Value are semantic nodes. Owner-alias-owned Relationships become active graph edges only when their evidence support is eligible.
• Provider output is a proposal. Closed-schema validation and deterministic server policy decide durable state.
• Default recall excludes candidates and Hypotheses and returns evidence only when its active Relationship support path is eligible for the requested time.
• Team visibility and owner mutation authority are distinct. An author can change only their own evidence or owned semantic records.

Authentication resolves one immutable actor as team + identity + membership + permanent owner alias + optional credential. An SSO browser session uses the selected membership's permanent owner alias and has no direct credential. An API-key request carries a credential whose stable ID is also its permanent owner alias. Team, identity, membership, and credential fields never let a client choose or replace the semantic owner.

60-Second Quickstart

Download the local compose example and environment template, configure the required secrets, and start Dense-Mem:

mkdir dense-mem-local
cd dense-mem-local

curl -fsSLo docker-compose.yml \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/docker-compose.base.yml
curl -fsSLo .env.example \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/.env.example

cp .env.example .env
# Fill in POSTGRES_PASSWORD, CONTROL_PORTAL_TOKEN, and AI_API_KEY.
${EDITOR:-vi} .env

docker compose up -d

The base stack uses PostgreSQL with pgvector as the durable authority. Current releases reject every NEO4J_* setting. If an installation still has a legacy Neo4j corpus, run the guided migration with v2.1.2 first, then upgrade with those variables unset. Neo4j is migration input, not a runtime fallback. The local ports are:

MCP:            http://127.0.0.1:8080/mcp
User portal:    http://127.0.0.1:8080/ui
Control portal: http://127.0.0.1:8090/

Open the control portal with CONTROL_PORTAL_TOKEN, then create a team and its first credential/API key. For control-plane automation, use the same private API:

control_token="<CONTROL_PORTAL_TOKEN from .env>"

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"primary-memory"}'

curl -fsS -X POST http://127.0.0.1:8090/control/api/teams/<team-id>/credentials \
  -H "Authorization: Bearer ${control_token}" \
  -H "Content-Type: application/json" \
  -d '{"name":"default credential"}'

The release image contains one project executable, /app/server. It applies pending PostgreSQL migrations under a database session lock before serving, so the Compose stack does not need a separate migration container. Multiple server replicas that share one writable primary serialize this startup step. Keep rolling-deployment migrations backward compatible with the previous app version; independent databases must each be migrated by a server connected to that database. Administration stays on the private control portal/API, while dreaming and automatic conflict review run as server background workers.

The image healthcheck allows the default 30-minute migration window and becomes active after its first success. If POSTGRES_MIGRATION_TIMEOUT_SECONDS is set above 1800, override the deployment healthcheck start period to at least the same duration.

Release candidates use vX.Y.Z-rc.N and demo-vX.Y.Z-rc.N. Stable releases use vX.Y.Z, latest, and demo-vX.Y.Z; there is no rolling demo tag.

The server requires complete embedding and verifier configuration at startup: AI_API_URL, AI_API_KEY, AI_API_EMBEDDING_MODEL, AI_API_EMBEDDING_DIMENSIONS, and AI_VERIFIER_MODEL. The compose examples provide OpenAI defaults for embeddings; choose the chat models explicitly in .env.

Verifier and assessor calls send temperature: 0 by default. Set AI_VERIFIER_DISABLE_TEMPERATURE=true to omit the field for providers or models that reject temperature.

Fully Local Setup (Ollama)

Any OpenAI-compatible endpoint can provide embeddings and verification. With Ollama running on the Docker host:

ollama pull nomic-embed-text
ollama pull llama3.1:8b

AI_API_URL=http://host.docker.internal:11434/v1
AI_API_KEY=ollama
AI_API_EMBEDDING_MODEL=nomic-embed-text
AI_API_EMBEDDING_DIMENSIONS=768
AI_VERIFIER_MODEL=llama3.1:8b
AI_VERIFIER_TIMEOUT_SECONDS=300

Use host.docker.internal, not 127.0.0.1, because the server calls the provider from the compose network. AI_API_KEY must remain non-empty because startup validation requires a complete provider configuration.

• Set AI_VERIFIER_MODEL to a model that exists on the selected chat endpoint. Startup validates the model configuration before the service accepts memory writes. A 7B-8B class model works for local smoke tests; larger models can exceed the default 60-second timeout while they load. Retryable processing stays within the durable placement-attempt budget and becomes terminal after that budget is exhausted.

Evidence Lifecycle

remember durably stages exact evidence and returns a submission_id; provider calls and processing happen after acknowledgement. Poll get_submission_status with that ID for the owner-scoped processing and search state. The status projection omits placement questions, provider output, and internal run IDs.

Callers submit logical Entity, predicate, and Value proposals rather than text offsets. remember does not accept span, surface, or relationship supports fields. Each Relationship instead lists the zero-based evidence_indices that support it, and those lists must collectively cover every evidence item in the submission. The assessor locates the exact evidence ranges; closed-schema validation and deterministic server policy decide whether those ranges are safe to commit.

To replace a specific current evidence item you own, put its UUID in the new item's supersedes_evidence_ids. Direct targeting is separate from advancing a source revision with previous_source_revision; do not combine them.

{
  "evidence": [
    {
      "content": "Dense-Mem now uses PostgreSQL as its only deployment target.",
      "source_type": "manual",
      "supersedes_evidence_ids": ["<owned-current-evidence-uuid>"],
      "idempotency_key": "deployment-target-correction-20260729"
    }
  ],
  "relationships": [
    {
      "ref": "deployment-target",
      "subject": {
        "name": "Dense-Mem",
        "entity_kind": "project"
      },
      "predicate": {
        "proposed_key": "uses_as_only_deployment_target"
      },
      "object": {
        "entity": {
          "name": "PostgreSQL",
          "entity_kind": "product"
        }
      },
      "polarity": "+",
      "modality": "statement",
      "evidence_indices": [0]
    }
  ],
  "idempotency_key": "deployment-target-correction-batch-20260729"
}

The target is retired atomically when the replacement is accepted for intake, even if later placement is rejected or quarantined. This preserves the exact correction decision instead of silently leaving stale evidence effective.

If the assessor cannot ground a proposal with enough confidence, or another semantic policy requires review, the whole submission moves to awaiting_review with zero partial semantic commit. get_submission_status returns bounded semantic_hold.issues and a semantic_hold.replacement instruction. Resubmit one complete corrected evidence-and-Relationships batch with replaces_submission_id set to the held submission_id; partial replacement is not supported.

To retract evidence without a replacement, call retract_evidence with owned current IDs, a bounded reason, and an idempotency key:

{
  "evidence_ids": ["<owned-current-evidence-uuid>"],
  "reason": "The source was withdrawn.",
  "idempotency_key": "withdrawn-source-20260729"
}

Both operations append lifecycle events. They never physically delete evidence or trace lineage. Current recall excludes retired evidence, while a historical known_at view before the event can still show what the system knew then.

correct_relationship replaces a specific active Relationship owned by the caller's permanent owner alias. It does not rewrite or delete the original record. The caller supplies the current Relationship version, its exact effective evidence spans, a bounded reason, and only the endpoints or predicate that need correction:

{
  "action": "submit",
  "relationship_id": "<owned-active-relationship-uuid>",
  "expected_version": 1,
  "patch": {
    "object_entity": {
      "entity_id": "<correct-same-team-entity-uuid>"
    }
  },
  "supports": [
    { "evidence_id": "<supporting-evidence-uuid>", "start": 0, "end": 38 }
  ],
  "reason": "The object was resolved to the wrong Entity.",
  "idempotency_key": "relationship-correction-20260808"
}

On acceptance, Dense-Mem atomically supersedes the original Relationship, creates or reuses the active successor, copies the effective support lineage, and appends the correction event and corrects cross-reference. A different owner in the same team may read team-visible memory but cannot correct the author's Relationship. Ambiguous Entity names require one owner confirmation; the original remains active until that confirmation succeeds.

Recall and Graph State

recall_memory is evidence-first but support-path gated. Its results[] contain evidence contexts only after final hydration proves an active, query-relevant Relationship support path remains eligible for the requested valid_at and known_at view. Related Relationships, communities, and Hypotheses are separate bounded fields; candidates and Hypotheses are not default memory results.

remember evidence (+ optional Entity/Relationship proposals)
        |
        v
durable staging -> validated placement -> active eligible Relationships
        |                                      |
        +-- lifecycle event -------------------+
                                               |
                                               v
                         support-gated evidence recall and trace lineage

MCP Tool Catalog

Discover the current closed-schema catalog with MCP tools/list; callers do not select a contract version. The server applies the same scope, feature, and visibility checks to tools/call.

The production release binary is compiled without the evaluation build tag, so no environment variable or control-panel setting can register evaluation tools in a live release. The evaluation target adds only the three harness tools above. eval_get_manifest, eval_get_knowledge_item, eval_list_recall_feedback_events, eval_get_recall_feedback_event, and eval_score_retrieval_case are removed because the current harness does not use them.

When recall feedback is enabled and the feedback snapshot is stored, recall_memory.suggested_actions points to submit_recall_session_feedback with the matching recall ID. When effective team Dreaming is enabled and recall returns Hypotheses, it also points to resolve_dream_feedback: confirm true or false only with independent evidence, and leave uncertain Hypotheses unresolved.

For local evaluation, the committed compose example builds the evaluation target and loads the ignored repository-root .env by default:

docker compose -p densemem_eval \
  -f examples/docker-compose.evaluation.yml up -d --build

go run ./cmd/eval-seedgen \
  --preset local_eval_100 \
  --out tests/eval/seeds/local_eval_100 \
  --suite tests/eval/suites/local_eval_100.jsonl

The local_eval_100 CLI preset emits the versioned local_eval_100_v2 seed identity with 100 corpus rows and 25 scored cases. It is a smoke check for the evaluation image and harness plumbing, not a replacement for the approved deterministic 1k release gate. Use IMPORT_CONCURRENCY=5 for this smoke; the full evaluation remains configurable up to the harness limit of 10.

Memory-pack export emits the current dense-mem.memory-pack.v2.4 artifact. Import and candidate-discovery workflows are not part of the public contract.

Supported HTTP Surfaces

There is no supported public REST memory API. Do not automate browser routes or depend on retired /api/v1 paths.

Telemetry Overlay

Prometheus telemetry is optional and off by default. To collect HTTP, embedding, verifier, assessor, recall feedback, Remember, conflict-review, cost, and Relationship lifecycle telemetry for the first-party dashboards, start the base stack with the overlay:

curl -fsSLo prometheus.yml \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/prometheus.yml
curl -fsSLo docker-compose.telemetry.yml \
  https://raw.githubusercontent.com/markhuangai/dense-mem/main/examples/docker-compose.telemetry.yml

export TELEMETRY_SCRAPE_TOKEN="$(openssl rand -hex 32)"
docker compose -f docker-compose.yml -f docker-compose.telemetry.yml up -d

The overlay starts Prometheus on 127.0.0.1:9090 and scopes dashboard queries to TELEMETRY_PROMETHEUS_JOB=dense-mem. Dashboard snapshots report whether each item is ready, inactive, unavailable, or unsupported. A valid zero is shown as zero; missing provider usage or pricing stays unavailable. Partial source failures keep successful cards and charts visible. System, team, and profile scopes apply the same visibility rules as the underlying data. Free-text recall-feedback comments stay in bounded investigation records; Prometheus receives only bounded labels. Conflict queue state gauges are emitted by each instance, so multi-instance dashboards should use max by (team_id, status) (or the equivalent label set), while event counters retain normal sum and rate semantics.

Responsibility Boundary

Data Egress and Consistency

Dense-Mem can send evidence text, proposal context, and recall queries to the configured embedding and verifier providers. Self-hosted providers keep that traffic within your boundary; hosted providers do not. Embeddings are derived, versioned state and cannot overwrite newer sources. Startup checks prevent mixing incompatible embedding models or dimensions.

Documentation

License

Apache-2.0