agent-machines

Persistent Workers. Replaceable machinery.

Agent Machines is the Worker system for persistent digital labor. Describe a responsibility or choose a trusted specialist, then keep its identity, memory, instructions, schedules, files, permissions, abilities, history, and evidence while runtimes, models, tools, and sandboxes change underneath it.

The routing wedge is “OpenRouter for agents and containers.” The larger product is a creation environment for long-running Workers: Lovable/Bolt/v0-shaped composition with ChatGPT-simple access to the first useful result. People do not want a bare sandbox or another disposable chat. They want a Worker that owns a job, remains inspectable, and is still there tomorrow.

Live site: https://www.agent-machines.com Source: https://github.com/Kevin-Liu-01/agent-machines

Table of contents

• What it is
• The product invariant
• The core idea: dual routing
• Control plane v2
• Browser Agent Console
• Deploy → Bootstrap → Attach → Talk
• The streaming gateway
• The harness
• Architecture
• Provider capability matrix
• Quick start
• CLI
• Web app
• Dashboard surfaces
• Repository layout
• Data boundaries and security
• Further reading

What it is

One account to route the machinery, compose the Worker, and access the result:

The intended interaction is four steps: describe or choose → connect → assign → supervise. Models, runtimes, sandboxes, terminal transports, scheduling, recovery, and migration disappear beneath that experience.

Two audiences:

1. Humans choose useful specialists or compose them from primitives, then watch, approve, inspect, and move the fleet.
2. Other agents drive the same API/CLI surface so a head Worker can provision, route, observe, and tear down subordinate Workers.

The product invariant

The Worker is durable. Everything underneath is replaceable.

Replaceable does not mean identical. Every provider and runtime declares what it can actually do; the control plane exposes only supported operations. The stable Worker abstraction preserves the user relationship while necessary infrastructure differences remain visible.

The core idea: dual routing

Most products lock you into one runtime or one cloud. Agent Machines routes both axes independently. Every substrate implements a single MachineProvider interface (provision / state / wake / sleep / destroy / exec / streamExec), so the rest of the system is provider-agnostic.

A credential gate blocks provisioning when the chosen runtime has no usable model upstream or the substrate has no key, so spin-up never fails silently downstream.

v2 lifecycle cutover. Hosted MachineProvider is now a compatibility facade over the mux provider adapters, so vendor SDK code has one home. The new control-plane kernel consumes those adapters through public and hosted runtime drivers. Every hosted lifecycle mutation now submits a durable operation before provider work. UserConfig / MachineRef remains as the dashboard's compatibility projection, while the operation journal uses atomic JSON locally or transactional Supabase in hosted deployments. The latter requires migration 009_control_plane_v2.sql. The exact ledger is docs/CONTROL-PLANE-V2.md.

Control plane v2: declarative Workers

src/control-plane is the new lifecycle kernel. Callers submit desired Worker state; a leased, idempotent reconciler owns provision, bootstrap, cold-start, sleep, live provider migration, scheduled dispatch, repair, and teardown. Requests and cron ticks may disappear without owning the Worker: unfinished operations remain in the journal and expired leases can be reclaimed.

import {
  AgentMachinesControlPlane,
  JsonFileControlPlaneStore,
  MuxWorkerRuntimeDriver,
  createMux,
} from "agent-machines";

const plane = new AgentMachinesControlPlane(
  new JsonFileControlPlaneStore(".agent-machines/control-plane.json"),
  new MuxWorkerRuntimeDriver(createMux()),
);

await plane.apply({
  id: "repo-coder",
  spec: {
    name: "Repo coder",
    runtime: "claude-code",
    sandbox: "e2b",
    migrationPolicy: "live",
    schedules: [],
  },
});

await plane.drain(); // a request, queue consumer, or cron reconciler can do this

The local package ships in-memory and atomic JSON journal adapters; the hosted app adds a tenant-scoped Supabase adapter with fenced leases and atomic RPCs. Worker detail exposes operation history and retry, and the internal cron tick is also the recovery consumer. See docs/CONTROL-PLANE-V2.md for the exact boundary.

Browser Agent Console (live CLI in the browser)

The headline capability and the hardest engineering problem in the repo: you operate the real agent CLI (Codex, Claude Code, Hermes, OpenClaw) from a browser tab, on a remote worker, with no local terminal and no tunnel.

The problem

The most capable agent tools ship as terminal programs (CLIs), not chat widgets. A live browser terminal normally requires a long-lived WebSocket PTY server in the middle, relaying every keystroke. Vercel added WebSocket Functions in June 2026, but a socket invocation is still bounded and cannot own durable terminal state. The session therefore stays on the worker while a pinned Function accelerates the live data plane.

• assume a local terminal (claude / codex on your laptop), excluding non-terminal users, or
• wrap the agent in a chat UI and lose the full TUI/CLI experience, or
• ship their own terminal locked to their own sandbox infra, or
• bolt on a relay/tunnel they then have to operate.

The inversion

Stop hosting the session in the API. Put the session on the worker; keep the control plane stateless.

• Session lives on the box. A persistent tmux session (amconsole) holds the PTY, the agent process, and scrollback. It survives serverless cold starts and function timeouts.
• Fast path (native-PTY substrates). One authenticated Vercel WebSocket Function pins one provider PTY and attaches it to amconsole; input, output, and resize stay on that connection.
• Portable fallback. HTTP tmux send-keys input plus offset-aware SSE tail -f output remains available anywhere exec exists.
• Session durability is unchanged. Socket reconnects reattach to worker-owned tmux; the Function never owns the shell or its scrollback.

exec is the only primitive each substrate must provide, so the same UI works across Dedalus, E2B, Sprites, and Vercel.

Browser (xterm.js)
  | input/output/resize <--> /api/dashboard/terminal/socket (pinned WebSocket)
  | fallback keystrokes --> /api/dashboard/terminal/input   (tmux send-keys -H)
  | fallback output    <--  /api/dashboard/terminal/stream  (SSE tail -f)
  v
Next.js control plane (Clerk auth, resolve machine + provider creds)
  v
provider.openPty (native fast path) or provider.exec / streamExec (fallback)
  v
Remote VM: tmux "amconsole" + pipe-pane --> /tmp/am-console.log
  v
Agent CLI running inside the pane (codex | claude | hermes | openclaw)

The closest predecessor is AWS CloudShell, a real browser terminal, but it is locked to the AWS ecosystem. This is CloudShell-shaped, substrate-agnostic, and wired to agent CLIs. Worker-owned tmux makes both the native-PTY WebSocket lane and the HTTP/SSE fallback reconnectable instead of tying session lifetime to an API process.

Performance notes

The interactive console is tuned to feel close to local:

• parallel xterm bundle load and tmux session attach,
• snapshot paint on connect (capture-pane), so the screen is never blank,
• requestAnimationFrame-batched writes to xterm,
• immediate flush for control keys and a zero-delay coalesce for raw printable input,
• a direct browser-to-worker WebSocket with one primary lane plus five preconnected failover lanes and deduplicated 12ms retry,
• correlated browser-to-PTY acknowledgements with a rolling 20-sample p95 SLO badge,
• an HTTP/SSE fallback with ordered input coalescing for non-native lanes,
• per-user getUserConfig cache (10s) and machine-state cache (3s),
• E2B sandbox connect reuse (45s) within a warm serverless instance,
• tmux pre-installed during bootstrap so the first attach never triggers a package install.

Production proof on 2026-08-14: the authenticated E2B dashboard reported a 41ms latest acknowledgement and 89ms p95 across 20 human inputs. A location-aware Sprite managed Service reached 10.8ms p50 across 100 paced inputs, with 87/100 below the 50ms target; periodic provider-proxy stalls produced 75.6ms p95 and 80.8ms max. The worker PTY write itself remained sub-millisecond. The UI therefore treats 50ms as a strict target and visibly reports breaches instead of claiming a hard internet latency guarantee.

Full write-up: knowledge/BROWSER-AGENT-CONSOLE.md. Engineering spec: web/docs/sandbox-terminal-gateway.md.

Deploy → Bootstrap → Attach → Talk

The primary dashboard flow is now runtime → sandbox → live console. The sandbox click submits one POST /api/dashboard/control-plane/workers intent; the hosted compatibility adapter owns these stages:

1. Declare. Create the durable Worker recipe: runtime, sandbox intent, model route, memory, and live-migration policy.
2. Provision + bootstrap. The adapter creates the provider machine, records its MachineRef, and schedules phase-aligned bootstrap (web/lib/bootstrap/runner.ts). Each phase tees to bootstrap.log and persists bootstrapState.
3. Attach. The browser opens the terminal page with ?launch=1, attaches the tmux console, and paints the pane snapshot.
4. Talk. The agent CLI auto-launches inside the pane and you interact line by line, including full-screen TUIs.

Bootstrap phases are split into CORE_BOOTSTRAP_PHASES (must succeed; gateway marked ready) and POST_GATEWAY_BOOTSTRAP_PHASES (best-effort, e.g. browser tooling) so a slow optional install can't block the agent from coming online.

The streaming gateway (capability-tiered)

Two output paths reuse the same SSE event contract (started · output · idle · error) so the UI is identical regardless of substrate capability.

• Interactive console (/api/dashboard/terminal/*): live PTY over tmux-over-exec, described above.
• One-shot exec (/api/dashboard/exec/stream) and bootstrap tail (/api/dashboard/bootstrap/stream): run a command, stream stdout/stderr while it runs.

Streaming prefers each provider's native primitive and only falls back to log-tail polling for substrates that physically cannot stream. The callback-to-generator adapter lives in web/lib/providers/stream-util.ts (bridgeExecStream).

The agent's HTTP chat gateway (Hermes :8642 / OpenClaw :18789) is now optional. The console path is exec-first and needs no public URL or Cloudflare tunnel; POST /api/chat degrades gracefully to "use the Terminal console" when a machine has no public gateway URL. Machine bearer tokens never become NEXT_PUBLIC_*.

The harness (registry-driven)

A worker is a runtime plus a composable harness. The app derives every count at runtime from the registries (web/lib/platform/harness.ts reads web/data/skills.json and web/data/mcps-catalog.json). The numbers written into this table are a snapshot of those registries, kept honest by src/lib/public-claims.test.ts, which fails if a published count and the registry disagree.

Skills follow the SKILL.md protocol: procedures saved to the machine compound over time and cannot be exported out of a stateless chat product.

Loadout vs registry: Loadout is what is already active on a machine (skills, MCPs, service routes). Registry is the install catalog: search, add to loadout, sync on deploy/reload.

Architecture

you
  | browser / CLI / API
  v
Declarative Worker API  ---------------  dashboard / SDK / CLI
  | operation journal + lifecycle reconciler
  v
WorkerRuntimeDriver  ------------------  MuxWorkerRuntimeDriver
  | hosted bridge: Clerk UserConfig + MachineRef during cutover
  v
MachineProvider  ----------------  E2B | Sprites | Vercel Sandbox | Dedalus
  | provision / state / wake / sleep / destroy / exec / streamExec
  v
persistent worker (provider home: /home/user | /home/sprite | /vercel/sandbox | /home/machine)
  |
  |-- tmux "amconsole"            interactive browser console (PTY over exec)
  |-- :8642 / :18789 gateway      optional HTTP chat (exec-first; no tunnel required)
  |-- ~/.agent-machines/          skills, mcps, chats, crons, sessions, logs, artifacts
  |-- <home>/agent-machines/      git checkout, used for knowledge reload
  v
model upstream (Vercel AI Gateway | OpenRouter | native OpenAI/Anthropic | custom)

Provider capability matrix

Every substrate implements MachineProvider; streaming tier depends on the SDK.

Native tiers relay output frame by frame with no extra exec calls. The poll fallback launches a detached command, tees combined output to a temp log, and polls new bytes until an exit-marker file appears.

Which lanes are proven, and on which surface. The multiplexer's own adapters (src/mux/providers/*) have run every agent on every substrate live. Latest full run, 2026-08-05: 12 of 16 cells pass -- all four harnesses on E2B, Sprites and Vercel Sandbox, none of four on Dedalus, every green cell asserted to return the exact sentinel text through the normalized event stream. Two Dedalus cells lost the run to one intermittent vendor defect (machine_not_found on a machine its own API reports as running) and two more passed the run but failed to tear down, which now counts as red. An earlier run on 2026-08-01 was 16 of 16 but gated on the exit code alone; docs/MUX-RESULTS.md says what each vintage proves. That run exercised the mux, not the table above: the hosted control plane's MachineProvider adapters are a second implementation of the same four vendors and are not covered by it. Convergence is item 0 of docs/ROADMAP.md.

Quick start

git clone https://github.com/Kevin-Liu-01/agent-machines
cd agent-machines
cp .env.example .env
npm install
npm run deploy   # CLI path (Hermes); or use /dashboard/setup for any provider

Requires Node ^20.19 or >= 22.12. Not merely ">= 20": require("agent-machines") resolves through the module-sync export condition, which only those Nodes match, and older ones cannot require() an ES module at all.

Agent Machines SDK

Create a user-scoped key in Dashboard → Settings → Developer API, then use the same provision → bootstrap → run flow from code:

export AGENT_MACHINES_URL=https://your-app.vercel.app
export AGENT_MACHINES_API_KEY=am_live_...

import { AgentMachines } from "agent-machines";

const am = new AgentMachines();
const agent = await am.create({
  agent: "codex",
  sandbox: "e2b",
  model: "openai/gpt-5.2",   // codex speaks the OpenAI Responses API
});
const result = await agent.run("Inspect this repository and fix the failing test.");
console.log(result.text);

model is optional and defaults to the agent's own upstream: codex gets an OpenAI id, claude-code an Anthropic one, and the gateway runtimes (hermes, openclaw) get the account default. Pairing an agent with a model its upstream cannot serve -- agent: "codex" with model: "anthropic/..." -- is refused by create() instead of provisioning a machine that 404s on its first turn. See docs/UPSTREAMS.md for the measured per-upstream wire formats and the model-id namespacing rule.

The key is displayed once and stored only as a SHA-256 hash. Rotating it invalidates the previous key immediately. Set bootstrap: false on the client only when you intend to bootstrap the machine yourself.

The multiplexer (no control plane required)

The same package ships a direct-to-substrate multiplexer: it talks to E2B, Sprites, Vercel Sandbox and Dedalus itself, installs the agent harness, and streams normalized events. No hosted control plane, no API key of ours -- just your provider keys. See docs/MUX.md for the architecture and docs/MUX-RESULTS.md for measured latencies.

agent-machines.json in your project root:

{
  "keys": { "anthropic": "env:ANTHROPIC_API_KEY" },
  "providers": { "e2b": "env:E2B_API_KEY", "sprites": "env:SPRITES_TOKEN" },
  "sandboxes": { "primary": "e2b", "backups": ["sprites"] },
  "agents": { "default": "claude-code" }
}

import { createMux } from "agent-machines";

const mux = createMux();

const machine = await mux.create({
  agent: "claude-code",
  sandbox: "auto",                                  // route, don't pin
  constraints: { pty: "native", maxRuntimeMs: 3_600_000 },
  optimize: "cost",                                 // opt-in, off by default
});

for await (const event of machine.run("review this repo", { runKey: "review-42" })) {
  if (event.type === "text") process.stdout.write(event.delta);
}

const pty = await machine.pty();   // real terminal, native PTY where available
console.log(machine.attempts);     // why it landed where it landed

// Application-level live handoff: warm target, drain managed runs, final
// stable state delta, verify, then atomically repoint the name. Processes
// restart from durable state; this is not a cross-provider RAM transplant.
await mux.migrate("coder", { to: "sprites", mode: "live" });

sandbox: "auto" walks primary -> backups through five stages, in this order:

1. Credentials. Lanes the config cannot authenticate are dropped, with the missing variables named. Fail closed.
2. Constraints. Lanes that cannot satisfy constraints are dropped, and the attempt records the dimension that failed (constraint: "pty"). An unprovable vendor fact reads as unknown and rejects rather than being hoped for.
3. Price, only with optimize: "cost": cheapest modeled total first, and a lane whose vendor publishes no rate sorts last rather than being read as free.
4. Learned selection, only for auto with no optimize: lanes are ordered by a score over this machine's own run traces -- task success first, then cost per successful result, then time to first output.
5. Health. A rolling window per substrate puts a lane that is failing right now at the back.

Only the first two stages ever remove a lane. The last three return permutations, because an incident that opened every circuit -- or a policy that has learned to dislike every lane -- must not make create() impossible. Provisioning errors fail over to the next lane, and every decision lands in machine.attempts with its reason, health state, modeled price and learned score.

What stage 4 is and is not: it is a deterministic scorer over the local JSONL trace store, shrunk toward a prior by sample count so one lucky run cannot outrank a long record, and it only ever reorders the lanes that survived stages 1 and 2. It is not a bandit (no exploration), it does not override a pinned sandbox or an explicit optimize, its evidence is local to one host, and the hosted dashboard has none of it.

Failover is placement-time only. A run that dies mid-stream comes back with truncated: true and is not replayed. runKey is an idempotency key, not a retry: a second run() with the same key returns the stored result instead of executing the agent twice. Each run also appends one trace record to a day shard in ~/.agent-machines/traces/ -- placement attempts, time to first event, and cost kept split into modeled sandbox compute and harness-reported model spend, with the total present only when both halves are known.

CLI

npm run deploy             # provision + bootstrap Hermes
npm run deploy:openclaw    # provision + bootstrap OpenClaw
npm run chat -- "message"  # chat with the active machine's gateway
npm run status             # active machine state
npm run logs               # tail gateway logs
npm run shell              # exec a shell command on the machine
npm run wake / sleep / destroy -- --yes
npm run reload             # git-pull the repo on the VM and re-sync knowledge
npm run doctor             # environment + machine health checks
npm run benchmark          # cross-substrate boot/exec/IO benchmarks

Multiplexer commands (direct to substrate, no control plane):

npm run mux -- run --agent claude-code "review my repo"  # streamed one-shot
npm run mux -- term --agent codex --name coder           # interactive agent PTY
npm run mux -- shell --name coder                        # raw PTY on the sandbox
npm run mux -- ls                                        # named machines
npm run mux -- migrate --name coder --to sprites --live  # drain + final delta + cutover
npm run mux -- rm --name coder                           # destroy a named machine

Read-only reporting, which works on a fresh install with no keys and no traces:

npm run mux -- routes                    # the five routing stages, and what each did
npm run mux -- routes --needs '{"pty":"native"}' --agent codex
npm run mux -- routes --optimize cost --json
npm run mux -- stats --since 24h         # task success, time to first output,
                                         # cost per successful result, truncation rate
npm run mux -- health                    # circuit state, samples and cooldown per substrate

stats reports those four numbers per harness@substrate lane; routes adds the learned score and the sample count behind it when the policy ran. A number nobody measured renders as unknown -- never 0, never a dash -- because zero is the best possible value for a cost, so a zero-filled unknown would report the lane nobody can price as the cheapest one available.

Live-test every harness on every credentialed substrate:

npx tsx scripts/mux-live-test.ts

Web app

cd web
cp .env.local.example .env.local
npm install
npm run dev

Open http://localhost:3210.

Configure Clerk for authenticated routes:

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=...
CLERK_SECRET_KEY=...

Production note: use Clerk production keys (pk_live_… / sk_live_…) on the deployed domain. Development keys carry strict rate limits and store metadata on a separate instance.

Key routes

Command palette (⌘K) jumps across machines, registry, loadout, and console routes.

Dashboard surfaces

Beyond provision-and-chat, the control plane is a fleet operations desk:

Supabase is required for durable metrics, usage, and activity. Without it, the app falls back to Clerk metadata only. See web/.env.local.example.

Repository layout

agent-machines/
  src/           SDK + CLI + mux + declarative control plane
    control-plane/*                    intent, operations, stores, reconciler, mux driver
  web/           Next.js control plane (site + dashboard + provider adapters)
    app/api/dashboard/terminal/*     Browser Agent Console
    app/api/dashboard/registry/*     unified install catalog search
    app/api/internal/cron/tick       scheduler + metrics collector (Vercel Cron)
    lib/providers/*                  MachineProvider (e2b | sprites | dedalus | vercel)
    lib/dashboard/registry/*         MCP registry, skills.sh, npm, bundled adapters
    lib/bootstrap/runner.ts          browser bootstrap (phase recipes)
    lib/dashboard/terminal-session.ts  tmux-over-exec session logic
    lib/metrics/collector.ts         usage + machine metrics → Supabase
    docs/WHITEPAPER.md               public technical whitepaper
    docs/sandbox-terminal-gateway.md engineering spec
    docs/README.md                   internal doc index
  knowledge/     skills, mcps, VISION.md, AGENTS.md, BROWSER-AGENT-CONSOLE.md
  mcp/           cursor-bridge MCP server

Data boundaries and security

• Clerk private metadata stores provider API keys, the Cursor key, gateway bearer tokens, and the full UserConfig.
• Clerk public metadata only exposes redacted setup state and machine summaries.
• All agent state (runtime, app data, skills, sessions, crons, config) lives under <provider home>/.agent-machines/.
• The VM repo checkout (<provider home>/agent-machines/) is used only for knowledge reloads.
• Machine gateway bearers are server-only and never shipped to the client as NEXT_PUBLIC_*.
• App-Router error boundaries (app/dashboard/error.tsx, app/global-error.tsx) keep a single component crash from white-screening the dashboard.

Further reading

License

MIT.