HERALD

The capabilities discovery layer for the agentic web.

One config. One command. One binary.

HERALD is an open-source framework + CLI that emits the standard discovery files agents need to read your site and discover its agent-interaction capabilities (including how to pay for access, when applicable). One config object drives all of it. Each file is an independent open standard; pick the layers you want and HERALD generates only those.

Declare directives for the agentic web

Every row above is a declaration, not a runtime. The discovery layer is published as agents.txt (plain text) and agents.json (structured capabilities), both generated from a single agentsjson.config.js. HERALD emits only what that config declares. The Chains row reflects any CAIP-2 network listed in x402.treasury; the Tokens & rails row reflects whatever you set under payments.* (Stripe SPT additionally covers card networks and Solana USDC).

What this does

Your site was built for human eyes. An agent that lands on it is effectively blind: it can't see your navigation, doesn't know which pages matter, and has no way to know what it is allowed to do. HERALD turns one config object into the discovery files that answer all of that, so any agent can read your site and act on it correctly.

One config object in. A complete, self-consistent discovery layer out. That is what turns an ordinary website into one that agents can actually use.

The files it generates / serves

/robots.txt                              : RFC 9309 access control                       [default, --skip-robots]
/llms.txt                                : llmstxt.org spec, from sitemap/Firecrawl       [optional, --skip-llms]
/sitemap.xml                             : sitemaps.org 0.9, when you supply the URL list [conditional; see below]
/agents.txt                              : agents.txt spec, plain-text capability layer   [optional, --skip-agents]
/agents.json                             : agents.txt spec, structured JSON companion     [optional, --skip-agents]
/.well-known/security.txt                : RFC 9116 vulnerability disclosure              [optional, --skip-security]
/.well-known/api-catalog                 : RFC 9727 linkset+json API directory            [optional, --skip-discovery]
/.well-known/mcp/server-card.json        : SEP-2127 MCP server card                       [optional, --skip-discovery]
/.well-known/agent-skills/index.json     : agentskills.io Discovery v0.2.0 index          [optional, --skip-discovery]
/openapi.json                            : OpenAPI 3.1 with x-payment-info per path       [optional, --skip-discovery]
_headers / vercel.json                   : §4.5 CORS + Link headers for the above set     [default, --skip-headers]

Each file is its own open standard. HERALD is the build/serve tooling for them. Pick the subset that matches the site: a robots.txt-only generator for static sites, add llms.txt for content briefing, go all the way to agents.txt + agents.json for capability discovery, or include the four ecosystem discovery surfaces under --discovery for full readability by agent-readiness scanners (Cloudflare's isitagentready.com, agentic-API auditors, MCP Registry probes). Every file follows the same honest-declarations rule: a surface is emitted only when its source block exists in agentsjson.config.js, and the matching _headers / Link: entries appear only when the file does.

Standards this builds on

agents.txt (with companion agents.json) is a lightweight, machine-readable capability declaration layer for websites in the agentic web: a protocol-agnostic discovery file that publicly announces what agent-interaction capabilities a site supports, without embedding the implementation details of any specific protocol.

HERALD implements the spec but does not own it. The spec lives at agents-txt.com under CC0. Anyone may implement it without restriction. The HERALD reference implementation is Apache 2.0.

Core design principles:

• Minimal & human-readable (agents.txt): plain text (UTF-8, RFC 3629), easy to serve and understand at a glance
• Rich & machine-first (agents.json): structured JSON (UTF-8 per RFC 8259) optimized for autonomous agents
• Standard-aligned companions (llms.txt / llms-full.txt): UTF-8 Markdown (RFC 3629) per the llmstxt.org spec; robots.txt UTF-8 plain text per RFC 9309; sitemap.xml UTF-8 with XML declaration per sitemaps.org
• Protocol & framework agnostic: declares that a site supports a protocol (x402, MPP, agent-auth, MCP, A2A, etc.) without prescribing how that protocol works
• Non-duplicative: implementation details, schemas, pricing, endpoints, and credentials live in the protocol's own mechanisms (402 responses, /.well-known/agent-configuration, AgentCard, MCP connection, etc.)
• Extensible: new capability blocks can be added without breaking existing parsers. Experimental identifiers (x-mypay, x-myauth) are accepted everywhere parsers see registered ones, giving new protocols a runway before formal registration

It is deliberately not a configuration file, not a full API spec, and not tied to any vendor. It is the neutral discovery layer for the entire agentic ecosystem.

Where these files fit

HERALD emits the four files that make up the agent-readiness stack:

Layer 1: ACCESS CONTROL     /robots.txt   (RFC 9309)         "You may enter my house"
Layer 2: PAGE INVENTORY     /sitemap.xml  (sitemaps.org)     "Here's how to navigate my house"
Layer 3: CONTENT BRIEFING   /llms.txt     (llmstxt.org)      "Here's what's inside my house"
Layer 4: AGENT CAPABILITIES /agents.txt   (agents.txt spec)  "Here's what you can do inside my house"

agents.txt (with companion agents.json) is the newest piece, an open standard for declaring agent-interaction capabilities (payments, auth, MCP, skills) without prescribing any specific protocol. HERALD exists to make adopting it trivial; the spec itself lives at agents-txt.com.

[!NOTE] HERALD declares payment capabilities. It does not wire the payment endpoints themselves.

When you set payments in agentsjson.config.js, HERALD emits the matching blocks in agents.txt and agents.json so agents can discover that your site supports x402, mpp, ap2, etc. and pre-screen pricing and chains. The actual 402 handler, signature verification, and on-chain settlement live outside HERALD; you supply them with your own middleware or a separate package.

Install

npm install -D @agentstxtdev/herald      # install as dev dependency
herald init                              # interactive setup → writes agentsjson.config.js
herald emit                          # writes discovery files to ./public

@agentstxtdev/herald-core is a transitive dependency pulled in automatically. You never install it directly.

CLI flags

# Strict-conformant minimal set (agents.txt spec §4.6):
herald emit --minimal                 # only agents.txt + agents.json + _headers

# Positive selectors (emit only these files):
herald emit --agents                  # only agents.txt + agents.json
herald emit --robots --llms           # only robots.txt + llms.txt
herald emit --robots                  # only robots.txt
herald emit --sitemap                 # only sitemap.xml
herald emit --llms-full               # only llms-full.txt

# Negative selectors (emit everything except):
herald emit --skip-agents             # skip agents.txt + agents.json
herald emit --skip-llms-full          # skip the expensive Firecrawl scrape

--minimal emits exactly the three files agents.txt spec §4.6 requires for strict conformance: the canonical /agents.txt, the structured /agents.json companion, and the platform _headers (or vercel.json) that satisfy §4.5 serving requirements. Use it on sites that want to be agents.txt-conformant without the ecosystem polish (llms.txt, sitemap.xml, security.txt, the RFC 9727 / SEP-2127 / agentskills.io / Payment Discovery surfaces). Other positive selectors take precedence if combined; --minimal is then ignored with a warning.

Generated robots.txt example

robots.txt is the Layer 1 access control file for your site. The format is defined by the Robots Exclusion Protocol (RFC 9309) and is honored by every well-behaved crawler. It declares which user agents may visit which paths, and it is the right place to draw the line between visitors you welcome and ones you do not.

Beyond the RFC, HERALD's generator does three things on top of a plain robots.txt. It explicitly allows the major search engine crawlers (Googlebot, Bingbot, and similar) so your SEO is unaffected. It blocks the well-known free AI training scrapers (GPTBot, ClaudeBot, CCBot, Google-Extended) when crawlers.blockFreeAiScrapers is enabled, since those crawls produce no value for the site owner. And it appends the Sitemap: and Content-Signal: directives that downstream tools rely on for sitemap discovery and for stating AI-usage preferences. The default wildcard block also Allow: /agents.txt and Allow: /llms.txt, which both grants explicit access and exposes those files to any crawler reading robots.txt (no separate discovery directive is needed; agents.txt is fixed at the canonical path).

The generator also merges intelligently with an existing robots.txt file. Anything below the # ── Existing rules (preserved) ── marker is kept verbatim across regenerations, so any project-specific rules you have authored survive every herald emit run.

# robots.txt
# Standard: https://www.rfc-editor.org/rfc/rfc9309

# Search engine crawlers
User-agent: Googlebot
User-agent: Bingbot
Allow: /

# Free AI training scrapers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Default
User-agent: *
Allow: /llms.txt
Allow: /agents.txt
Allow: /

Sitemap: https://mysite.com/sitemap.xml
Content-Signal: search=yes, ai-train=no, ai-input=no

Reading each section

The generated file is intentionally low on inline commentary so it stays easy to scan and audit. Here is what each block means in practice:

• Free AI training scrapers section. These UAs are listed under a single Disallow: /. The intent is to keep free AI training crawls off your origin while leaving a structured path for paying agents to negotiate access through agents.txt and the x402 / MPP flows. The block is a soft signal (UA strings are advisory and trivially spoofable); the load-bearing enforcement for paid access lives in the 402 handler on your gated routes, not in robots.txt.
• Paid agentic agents section (emitted only when crawlers.additionalAllowList is set, or when a canonical paid-crawler UA exists — currently neither is true by default). When present, this block names UAs that should be Allow-ed through to the rest of your stack, where they can hit agents.txt, discover the payments block, and negotiate access via x402 or MPP. There is no canonical ecosystem-wide UA for this class yet, so the section is suppressed by default; adopters who run their own crawler and want sites to recognize it can use crawlers.additionalAllowList: ['MyCrawlerBot'] in their config.
• Default wildcard block. Search engine and AI-scraper UAs above this block override it for those clients (RFC 9309 specificity). The wildcard exists so any other crawler reading robots.txt is explicitly told /llms.txt and /agents.txt are reachable, which doubles as discovery for those two files — no separate Agents-Txt: directive is needed because spec §4.3 fixes agents.txt at the canonical path.

Sitemap: is the long-standing widely-supported extension that points at your URL inventory; it appears whenever the content driver produces an authoritative URL list (static, manual, or firecrawl with --sitemap). Content-Signal: follows the IETF AIPREF draft (CC0) and lets you state AI-usage preferences in a machine-readable way alongside the access rules above.

sitemap.xml emission policy

HERALD only emits sitemap.xml when it has authoritative URLs to put in it. The default policy keys off content.driver:

If your framework already generates a sitemap (Next.js app/sitemap.ts, @astrojs/sitemap, Hugo, Jekyll, 11ty), keep using it; pass --skip-sitemap and robots.txt will still reference your framework-emitted file via the Sitemap: directive.

Generated llms.txt example

llms.txt is the Layer 3 content briefing for your site: an LLM-optimized index that follows the llmstxt.org spec. It tells an agent what your site is and points at the pages worth reading, in a structured plain-text format. Format is fixed: an H1 with the site name, an optional > blockquote summary, then ## Section headings each containing a bullet list of [Title](https://github.com/agents-txt/HERALD/blob/main/url): description lines. A trailing ## Optional section flags pages an agent can safely ignore on a first pass.

The page list itself comes from content.driver in your agentsjson.config.js. The driver decides where the URLs originate (your existing sitemap.xml, a Firecrawl crawl, an explicit list of pages, or fully curated sections), and @agentstxtdev/herald-core renders them into the format above. Payment terms, authentication, MCP endpoints, and skill packages do not belong in llms.txt; those live one layer up in agents.txt / agents.json.

# My Site

> A site accessible to AI agents.

## Docs
- [Getting Started](https://mysite.com/docs/getting-started): Quick start guide for new users.
- [API Reference](https://mysite.com/docs/api): Full API documentation with examples.

## Blog
- [How x402 Works](https://mysite.com/blog/x402): Deep dive into HTTP-native payments.

## Optional
- [Archive](https://mysite.com/archive): Older posts kept for reference.

For richer per-page descriptions and the expanded llms-full.txt companion (where the markdown body of each page is inlined under its heading), use the firecrawl content driver, covered next.

Firecrawl integration (richer llms.txt)

Not sponsored or affiliated. Firecrawl is one of the supported content drivers.

Instead of parsing sitemap.xml, use Firecrawl (free tier available) to crawl your site and generate a content-aware llms.txt:

content: {
  driver: {
    type: 'firecrawl',
    siteUrl: 'https://mysite.com',
    apiKey: process.env.FIRECRAWL_API_KEY,

    // optional: all map options from Firecrawl v2:
    limit: 5000,                    // default 5000, max 100000
    search: 'pricing',              // order results by relevance to a query
    sitemap: 'include',             // 'include' (default) | 'skip' | 'only'
    includeSubdomains: true,        // default true
    ignoreQueryParameters: true,    // default true; drops ?utm=... etc.
  },
},

Uses the Firecrawl /v2/map endpoint, which returns each URL with its title and description in one response. A single API call populates the entire llms.txt (no per-page scraping).

Get a free API key at firecrawl.dev (no credit card for free tier).

llms-full.txt: expanded companion with inlined page content

The llmstxt.org spec describes "expanded" forms (llms-ctx.txt, llms-ctx-full.txt) where each linked page's markdown content is inlined under its heading, so an LLM can ingest the whole site as one document. The community has converged on /llms-full.txt as the served filename. That's what agents look for, and that's what HERALD emits.

By default llms-full.txt is built from the same URL list as llms.txt. The optional content.fullTxt.driver lets you point at a different URL list, useful when your llms.txt indexes the marketing site but you want llms-full.txt to ingest the docs subdomain:

content: {
  // /llms.txt: concise index of the marketing site
  driver: {
    type: 'sitemap',
    sitemapUrl: 'https://mysite.com/sitemap.xml',
  },

  // /llms-full.txt: pages from the docs subdomain, with content scraped to markdown
  fullTxt: {
    driver: {
      type: 'firecrawl',
      siteUrl: 'https://docs.mysite.com',
      apiKey: process.env.FIRECRAWL_API_KEY,
    },
  },
},

The spec doesn't restrict URLs in llms.txt to a single origin, so cross-domain fullTxt sources are spec-compatible.

Behavior per source driver type:

• firecrawl: pages are scraped via /v2/scrape (5 concurrent requests, markdown format, main-content only) and inlined under each heading. This is the recommended setup.
• sitemap / static / manual: emits the file with link + description per page but no scraped body content (we only have URLs to work with). Add a Firecrawl source if you want actual content inlined.

Omit the fullTxt block to skip llms-full.txt generation entirely.

The @agentstxtdev/herald CLI and agentsjson.config.js

HERALD is driven by a single file at your project root: agentsjson.config.js. It's the source of truth for every discovery file HERALD emits. The CLI creates, validates, and re-renders from it.

Three commands

Per-file flags for emit:

Positive selectors (pass one or more to emit only those files; otherwise everything applicable to the config is emitted):

• --robots: emit robots.txt
• --llms: emit llms.txt
• --llms-full: emit llms-full.txt (requires content.fullTxt in the config)
• --agents: emit agents.txt and agents.json
• --sitemap: emit sitemap.xml (also forces emission for the firecrawl driver; warns + skips for the sitemap driver since that would be circular)
• --headers: emit the §4.5 headers config for the detected hosting platform (_headers for Cloudflare/Netlify, vercel.json for Vercel; --platform <name> overrides detection). See Serving headers below for the details.

Negative selectors (subtract from whatever set is selected):

• --skip-robots: skip robots.txt (useful when your framework or CDN owns it)
• --skip-llms: skip llms.txt
• --skip-llms-full: skip llms-full.txt (keep llms.txt; useful when you only want to refresh the index)
• --skip-agents: skip agents.txt and agents.json
• --skip-sitemap: never emit sitemap.xml, even for static / manual
• --skip-headers: skip the §4.5 headers config file

See herald emit --help for the full list.

agentsjson.config.js: the file you create

You don't manually write this from scratch. Run herald init in your project root and the wizard writes it for you. (herald emit then reads this file to generate the discovery files; it never writes the config.) The file shape:

// agentsjson.config.js  (lives at your project root)
export default {
  // Site metadata — required. Drives robots.txt, llms.txt, agents.txt, agents.json
  site: {
    name: 'My Blog',
    url: 'https://myblog.com',
    description: 'Technical writing about distributed systems.',
  },

  // Where llms.txt's page list comes from. Pick one driver:
  //   sitemap   — read your existing sitemap.xml
  //   firecrawl — crawl the live site (richer titles, auto-grouping; needs FIRECRAWL_API_KEY)
  //   static    — hand-curated sections, no crawl
  //   manual    — supply sections[] with full control
  content: {
    driver: {
      type: 'static',
      pages: [],
      sections: [
        {
          name: 'Docs',
          pages: [
            { url: 'https://myblog.com/intro', title: 'Intro',   description: 'Project overview.' },
            { url: 'https://myblog.com/api',   title: 'API ref', description: 'Endpoint reference.' },
          ],
        },
      ],
      // Switch to firecrawl for an auto-crawled page list:
      // type: 'firecrawl',
      // siteUrl: 'https://myblog.com',
      // apiKey: process.env.FIRECRAWL_API_KEY,
    },
  },

  // robots.txt rules
  crawlers: {
    blockFreeAiScrapers: true,   // GPTBot, ClaudeBot, CCBot, Google-Extended → Disallow
    allowSearchEngines: true,
    allowPaidAgents: true,
  },

  // Optional: payment capability declaration (advertised in agents.txt / agents.json).
  // HERALD does not wire the 402 handler; bring your own middleware.
  payments: {
    protocols: ['x402', 'mpp', 'ap2'],
    x402: {
      treasury: {
        evmAddress: process.env.EVM_ADDRESS,
        evmChains: ['eip155:8453'],
        solanaAddress: process.env.SOLANA_ADDRESS,
        solanaNetwork: 'mainnet-beta',
      },
      pricing: { amount: '0.01', token: 'USDC' },
    },
    mpp: {
      tempoRecipient: process.env.TREASURY_TEMPO,
      pricing: { amount: '0.01', token: 'USDC' },
    },
    // AP2 mandate layer (ap2-protocol.org). Announces support; the mandate
    // exchange (CheckoutMandate / PaymentMandate) happens during checkout.
    ap2: {
      presentations: ['sd-jwt-vc'],
      spec: 'https://ap2-protocol.org',
    },
    // Optional: OpenAPI discovery surface emitted at /openapi.json per the
    // Payment Discovery draft (paymentauth.org). One entry per payable path,
    // each carrying x-payment-info offers an agent reads before issuing the
    // request. Independent of the protocols[] gate above: this file
    // advertises protocol capability, not credential presence.
    openapi: {
      paths: {
        '/api/premium': {
          summary: 'Premium API endpoint.',
          offers: [
            { intent: 'charge', method: 'tempo',  amount: '10000', currency: '0x20c00...', description: 'USDC.e on Tempo' },
            { intent: 'charge', method: 'stripe', amount: '1',     currency: 'usd',       description: 'Stripe card or Solana USDC' },
          ],
        },
      },
    },
  },

  // Optional: agent identity verification (agent-auth)
  authorization: {
    enabled: true,
    protocols: ['agent-auth'],
    identityRequired: false,
  },

  // Optional: MCP endpoint declaration
  mcp: {
    endpoints: {
      url: 'https://myblog.com/mcp',
      description: 'MCP server exposing blog content and search.',
    },
    // Optional: SEP-2127 server card emitted at /.well-known/mcp/server-card.json.
    // All three capability booleans are required by the auditor; set the ones
    // your MCP server actually exposes.
    serverCard: {
      name:    'myblog-mcp',
      version: '1.0.0',
      capabilities: { tools: true, resources: false, prompts: false },
    },
  },

  // Optional: agent-installable skill packages (agentskills.io).
  // The name / type / digest fields drive the v0.2.0 discovery index at
  // /.well-known/agent-skills/index.json. Without a digest the entry is
  // emitted in agents.txt / agents.json but skipped from the index.
  skills: {
    urls: {
      url: 'https://myblog.com/skills/my-skill/SKILL.md',
      name: 'my-skill',
      type: 'skill-md',
      digest: 'sha256:0123456789abcdef...',   // sha256sum public/skills/my-skill/SKILL.md
      description: 'Teaches agents how to search and navigate this blog.',
    },
  },

  // Optional: A2A AgentCard discovery (a2a-protocol.org)
  a2a: {
    cards: {
      url: 'https://myblog.com/.well-known/agent-card.json',
      description: 'Blog assistant agent card.',
    },
  },

  // Optional: UCP profile discovery (ucp.dev)
  ucp: {
    profiles: {
      url: 'https://myblog.com/.well-known/ucp',
      description: 'UCP profile for commerce capabilities.',
    },
  },

  // Optional: WebMCP page discovery (webmachinelearning.github.io/webmcp)
  webmcp: {
    pages: {
      url: 'https://myblog.com/app',
      description: 'Page that registers in-browser tools via navigator.modelContext.',
    },
  },
}

Experimental protocols (x- prefix). Both payments.protocols and authorization.protocols accept identifiers prefixed with x- (for example x-mypay, x-myauth) per agents.txt spec §3.1. The generator emits them verbatim into agents.txt and as empty per-protocol objects in agents.json (payments['x-mypay']: {}). This is the runway for advertising a new protocol before it lands in the spec, without forking herald.

The same file is consumed by herald emit, which reads it to write the static discovery files into --out. You write it once. There is no separate runtime config; nothing duplicates.

Where the file lives

• Static / Jamstack sites (Astro, Hugo, 11ty, Next.js export): at your project root, generated at build time by herald emit --out ./public.
• Server frameworks (Express, Hono, Next.js App Router): at your project root, generated at build time or on deploy. Serve the resulting files as static assets, or hand-roll a route that imports @agentstxtdev/herald-core to render them on demand.

Editor autocomplete via $schema

Every agents.json herald emits carries a $schema field pointing at the canonical JSON Schema hosted on agents-txt.com:

{
  "$schema": "https://agents-txt.com/schema/agents-json/v1.0.json",
  "version": "1.0",
  "standard": "https://agents-txt.com",
  "site": { ... }
}

JSON-aware editors (VS Code, JetBrains, Helix with the JSON LSP, jq --schema, anything that respects the $schema field) read the referenced document and provide inline validation plus autocomplete the moment an operator opens the file. Hand-edits stay honest: a typo in payments.mpp.methods, a missing required field, a non-https URL in mcp[].url all surface in the editor before the file is ever served.

The schema is derived from the Zod source in @agentstxtdev/herald-schema via z.toJSONSchema(). To consume it directly in a third-party validator:

import { AgentsJsonSchema } from '@agentstxtdev/herald-schema'

const result = AgentsJsonSchema.safeParse(await (await fetch(url)).json())
if (!result.success) console.error(result.error.issues)

Validation

Both init and emit run a Zod schema (CLI-only, doesn't bloat @agentstxtdev/herald-core). Errors print field-level paths so misconfiguration surfaces early:

❌ Failed to load config: Invalid agentsjson.config.js:
  • site.url: must be a valid URL e.g. https://mysite.com
  • payments.x402: treasury must include at least one of evmAddress or solanaAddress (after lenient validation)

Per-field lenient validation for optional wallet env vars. The format checks for evmAddress (40-char 0x hex), solanaAddress (32-char base58 minimum), and stripeSecretKey (sk_ prefix) are still strict, but a malformed optional field no longer aborts the whole generate. Instead, the value is treated as undefined and the CLI prints a one-line warning:

herald: ignoring malformed evmAddress (evmAddress must be a 40-char hex EVM address (0x...)); set EVM_ADDRESS to a valid 0x[40 hex] value or unset to skip EVM.

This means a typo in an unused wallet (EVM_ADDRESS=garbage in your .env when you only meant to wire up Solana) does not break the Solana side. The TreasuryConfigSchema.refine rule still fires after the lenient pass: if every wallet is dropped, x402 fails with treasury must include at least one of evmAddress or solanaAddress (after lenient validation), because x402 with no recipient is meaningless.

The emit step then runs the spec validators (RFC 9309 for robots.txt, llmstxt.org for llms.txt, agents.txt v1 for agents.txt/json, sitemaps.org 0.9 for sitemap.xml) on the output files and prints any compliance warnings, so a typo in your config can never silently produce a non-compliant file.

Serving headers (agents.txt spec §4.5)

The agents.txt spec mandates four response headers on /agents.txt and /agents.json: a Content-Type with charset (for agents.txt), Access-Control-Allow-Origin: * (so browser-context agents can read the files cross-origin), and a Cache-Control: public, max-age=3600 (SHOULD). Static-asset pipelines on most hosting platforms do not set these by default, so the headers have to be wired in some platform-specific way.

herald emit handles this for you. The CLI detects your hosting platform from project files and emits the right config:

A2A AgentCard paths included automatically. When a2a.cards is set in agentsjson.config.js, the generator emits matching header entries for each same-origin AgentCard path alongside the /agents.txt and /agents.json entries. The headers used are Content-Type: application/json, `