GDPR Forget-Me Agent Skill for OpenSearch

An OpenSearch Agent Skill that fulfils GDPR "right to be forgotten" requests, including the data that keyword-based PII tools miss: people who are identifiable without their name ever appearing.

Vendor neutral by design. No API keys, no cloud services, no proprietary dependencies. The embedding model is a local pretrained model deployed inside your cluster through ML Commons, so text never leaves your boundary. It runs on any OpenSearch distribution (self-managed, Amazon OpenSearch Service, or Serverless) and falls back to BM25 when no model is available.

Want to run it? DEMO-AGENT.md drives the skill by prompt, the way it is meant to be used. DEMO.md is the same workflow at the command line, from starting OpenSearch to a verified erasure with an audit certificate.

The problem

When an erasure request arrives, some of the data is easy to find. Where the person is named or their email appears, a regex matches it directly, and the skill's direct-identifier pass does exactly that.

The hard part is the personal data scattered through logs, traces, incident reviews, and tickets where the person is only described:

"the solo senior frontend engineer on-call during the #4091 outage who resigned at the end of March"

No name, no employee ID, no direct identifier. GDPR Recital 26 is explicit that someone is personal data whenever they are identifiable indirectly, "by reference to one or more factors specific to their identity." Direct scanning alone leaves this behind. This is the gap the skill closes: it runs the direct pass, then adds hybrid search plus agent reasoning to find the records where the person is only described.

What this skill does

Turns any Agent-Skills-compatible IDE (Claude Code, Cursor, Kiro, Copilot, Windsurf, Gemini CLI, Codex) into a privacy-engineering agent that:

1. Discovers candidates two ways: a direct-identifier pass (name, email, employee id, phone, IP) that searches both the text and the fields recording who a document is about, and hybrid BM25 plus neural/vector search for name-free descriptions. Searching text alone found 3% of one subject's footprint in a real email corpus; the rest was in from, to and cc.
2. Disambiguates each candidate by reasoning about whether it uniquely identifies the subject, scoring confidence and extracting exact identifying snippets. A precision_mode threshold controls precision versus recall.
3. Previews the exact documents and DSL before anything is written. The default leans to recall (high_recall, threshold 0.60), because leaving someone in the index after an erasure request is the worse failure and every run is reviewed by a human before anything changes.
4. Remediates with redact_in_place (replace only the identifying snippets with [GDPR_REDACTED], preserving the rest of the record) or hard_delete.
5. Verifies: the generated script carries read-back commands.
6. Records every run in a local, hash-chained erasure certificate, the evidence GDPR Art. 5(2) and Art. 30 require you to produce.

The skill never writes to OpenSearch. It emits a reviewable curl script that a human inspects and runs.

Quickstart

Requires uv, and Docker for the local demo.

# Install into your agent (Claude Code / Cursor / Kiro / ...)
npx skills add philterd/opensearch-agent-right-to-be-forgotten

# ...or clone and point your agent at this directory.

Then ask your agent, for example:

"We got a GDPR erasure request. Scrub the senior frontend engineer who owned Checkout, was sole on-call during incident #4091, and resigned end of March 2024, from logs-application-*. Redact, don't delete."

Is the indirect pass worth running on your data?

The direct pass works wherever identifiers appear literally. The indirect pass only earns its keep where documents describe people, and corpora differ by two orders of magnitude in whether they do. assess samples an index and says which it is, before you trust a result rather than after:

uv run python scripts/forget_me.py assess --index "logs-application-*"

It reports descriptive references per document against corpora we measured (21.0 in US court opinions, 0.20 in Enron email), how many of those sit in running prose rather than recipient lists, and whether any field could serve to check an answer against. A role noun is not a description, so it prints the phrases it matched: read those before trusting the band.

Datasets

seed-demo is deterministic and offline, and its ground truth lives in the seed script. Because the agent is also the evaluator, the corpus is built so it cannot see that answer key: document ids are opaque digests rather than labels like sub-1, and seed-demo writes the key to gdpr-eval/demo-ground-truth.json instead of printing it.

Building ground truth for indirect matches is the hard part, since whether a name-free description identifies someone depends on facts the index does not contain. EVALUATION.md sets out a method that derives labels from data the corpus already holds, how it applies to Enron, and what the resulting numbers do and do not mean. The roster command implements its first stage against mail-enron: it sweeps the address headers into a per-person record of address, display-name variants, Exchange login, active window, and message count, then reports coverage and records whether the available attributes can support the later stages. It reads only headers, never message, so the field discovery searches stays held out of the label source.

The readable names are in X-From, X-To, and X-cc, not in From, To, and Cc, which hold bare addresses. Reading only the latter puts display-name coverage at about 2% and the roster reports a corpus that cannot support the later stages; reading the X- headers puts it near 90%. They are paired positionally with the address lists, and the names are dropped for a message whose two headers disagree on length rather than risk attaching one person's name to another's address. The roster names real people, so it is written to gdpr-eval/ (gitignored) and only aggregate metrics are printed.

mask-corpus implements the second stage. For one subject it builds an alias set (full name, surname, given name, initials, login forms, every address that person used), removes every variant from the searched text, and writes the result to a separate index under opaque ids. mail-enron is left untouched, and only the masked text and its timestamp are carried across, so no header or mailbox name reaches a search over the masked copy. The positives go to a label file on disk rather than to the output.

Masking and labelling use different halves of that alias set. Everything gets masked, because over-masking only costs residual context. A document is only labelled a positive when it holds a variant no other person in the roster produces. A given name fails that test as soon as the corpus holds a second person with the same one, and on a ten-custodian sample of this corpus seven people share the given name of one subject: labelling on the full alias set made 92% of that subject's positives documents about somebody else.

Masking is then audited, and the audit is a gate rather than a warning: if one variant survives, the document is still trivially retrievable and every number computed from it would be fiction, so the run exits non-zero and the label set is marked unscorable. audit-mask re-runs that check on its own.

Masking removes each variant rather than replacing it with a marker. A visible marker would appear in exactly the documents that contained a variant, which is the definition of a positive, so the corpus would carry its own answer key. The gate fails any run whose marker is confined to the positives. This is separate from the erasure workflow, where [GDPR_REDACTED] marks what was removed on purpose.

subjects picks who to run this on, ranking by mentions in running prose rather than by message count and rejecting surnames that are ordinary words. score-discovery then scores retrieval and score-judgment scores the agent's judgment and its redaction spans. Scoring reports the three stages separately, because they fail independently and one end-to-end number hides which one broke.

The gate also fails closed on an alias set with no name in it. An audit can only look for the variants it was given, so a subject whose roster entry carries no display name would otherwise pass while the corpus still prints their name on every page. Alongside that, the report counts documents matching a name variant ignoring word boundaries. That number over-reports by construction and never fails a run, but a high count next to zero surviving variants is the signature of a masking pattern whose boundary rule is skipping something. Note that masking manufactures the indirect case, since a sentence written without a name would have been phrased differently from one with the name removed. Results from this corpus are a proxy for naturally occurring indirect reference, not a sample of it.

seed-enron is the realism check: nothing in it was written to be found. Role-reference language appears in only around 1% of Enron messages, and most instances describe a generic role or name the person elsewhere in the same message, so seed-demo remains the clearer demonstration of indirect identification while Enron exercises real direct identifiers at scale.

Provenance of the Enron corpus

The data is not redistributed with this skill, and .gitignore blocks a local copy from being committed. seed-enron fetches from CMU's distribution at run time. CMU grants no license; it distributes the corpus "as a resource for researchers who are interested in improving current email tools," and asks users to "be sensitive to the privacy of the people involved (and remember that many of these people were certainly not involved in any of the actions which precipitated the investigation)." Most of these people are private individuals whose email became public because of an investigation into other people's conduct.

That is also why it is the right test case. This corpus is personal data that has long outlived any lawful basis for processing and remains fully indexed and searchable more than two decades on. CMU's own page records that some messages were deleted "as part of a redaction effort due to requests from affected employees". Erasure requests against this dataset are not hypothetical.

Configuration

Safety

The skill never writes to the cluster. export-curl refuses any index matching a --legal-hold pattern, every generated command targets documents by exact (index, _id) rather than a blind query, and each run writes a hash-chained erasure certificate without touching the cluster. See the Safety rules in SKILL.md.

This is not legal advice

gdpr-forget-me is an illustrative example of how one might automate "forgetting" a user's personal data in OpenSearch: the discovery, review, redaction or deletion, verification, and audit mechanics. It is not legal advice, not a certified compliance product, and not a guarantee of GDPR, CCPA, or any other regulatory compliance. Detection is probabilistic.

Whether a given erasure is required, permitted, or complete, and which data, retention obligations, and legal holds apply, is a legal determination. Consult your Data Protection Officer and legal counsel, validate the behavior against your own obligations and data, and treat the demo dataset and outputs as a worked example rather than a turnkey solution. You are responsible for anything you run against your own clusters.