record-and-replay

Agent-agnostic, open-source record & replay skill. Let the user demonstrate a workflow — in the browser or on the desktop — while an AI coding agent records it, then turns the demonstration into a reusable agent skill.

Inspired by OpenAI Codex's macOS-only Record & Replay plugin, rebuilt on open tooling so it works across platforms and across agents: Claude Code, Codex CLI, opencode, and any agent that can run shell commands and read SKILL.md-style skills.

Platform support

How it works

Browser mode (default — richest evidence): launches a headed browser via Playwright (your installed Google Chrome if available, bundled Chromium otherwise) and records every user action into events.jsonl — each with multiple selector candidates (testId → role+name → id → text → css) — plus Playwright trace chunks (DOM snapshots + screenshots, 60 s checkpoints).

Desktop mode: a single-process recorder (pynput) captures global mouse/keyboard events, active-window changes and a half-resolution JPEG at each click into desktop-events.jsonl + screens/ — the same philosophy as Codex's native Record & Replay (an event/context stream, not video) at ~0-3% CPU.

The agent then reads the recording (compact markdown summaries via the bundled summarizers), treats it as evidence of intent — not a pixel-by-pixel script — and generates a reusable skill with semantic locators and verification steps.

Install

git clone https://github.com/ugarchance/record-and-replay-skill ~/.agents/skills/record-and-replay
cd ~/.agents/skills/record-and-replay
node setup.mjs      # npm install + (no Chrome? download Chromium) + self-test
node install.mjs    # link into agent skill dirs (symlink on macOS/Linux, junction on Windows)

install.mjs links the skill into ~/.claude/skills/, ~/.codex/skills/ and ~/.config/opencode/skills/ for every agent that is installed. For any other agent supporting the SKILL.md format, link the folder into its skill directory.

Desktop mode's Python env (.venv-desktop: pynput+mss+pillow, ~50 MB) is created by setup.mjs automatically when uv is installed.

Usage

Tell your agent "watch me do this and make it a skill" — the skill file guides it. Manually:

# Browser (macOS/Linux; Windows: Start-Process node …)
nohup node scripts/recorder.mjs --minutes 30 --name my-task &
node scripts/summarize.mjs ~/.agents/recordings/my-task

# Desktop
node scripts/desktop-record.mjs start my-task 30
node scripts/desktop-record.mjs pause  my-task   # closes segment recording-partN.db
node scripts/desktop-record.mjs resume my-task
node scripts/desktop-record.mjs stop   my-task
python3 scripts/desktop-summarize.py ~/.agents/recordings/my-task

Recording controls

• Floating control window (bottom-right, separate from the recorded pages — like Codex's native Recording Controls panel): live timer, Pause/Resume, Stop, and Discard (two-tap confirm; deletes all evidence, session ends as discarded). Reopens itself if closed by accident. Pass --badge to use an in-page badge instead.
• While paused, nothing is captured — events and trace both stop.
• Agent/file control: touch <outDir>/PAUSE (delete to resume), touch <outDir>/STOP. Stays in sync with the window buttons.
• Auto-stop at the time limit; closing the browser window also ends the session (avoid Cmd+Q — the last minute of the trace may be lost; everything else survives thanks to 60 s trace checkpoints).
• Desktop pause/resume is segment-based (suspending the capture process could silently kill the macOS event tap) — the summarizer merges segments into one timeline.

Performance defaults (lightweight by design)

• Browser trace defaults to light: full DOM snapshots per action, no continuous screencast (--trace full re-enables screenshots, --trace off disables tracing entirely).
• Desktop capture is a single process with no continuous screen reading and no video — measured at ~0-3% CPU while recording. (Continuous full-res video capture was tried and rejected: the raw frame pipeline can overwhelm a machine.)
• Typing is debounced, scroll/selection sampled, the event stream is rate-limited — a 30-minute recording stays in the tens of MB, not GBs.
• Stopping is signal-free on every platform (STOP-file watch) — recordings always finalize cleanly, including on Windows.

Design notes

• Two-layer evidence (browser): events.jsonl (semantic actions, multi-candidate selectors) + trace-NNN.zip (per-action DOM snapshot & screenshot; open with npx playwright show-trace trace-*.zip).
• Persistent browser profile at ~/.agents/recordings/.browser-profile keeps logins across recordings and replays. ⚠ It holds live session cookies — treat it as plaintext credentials; never commit or share it.
• Sensitive data (browser): password/OTP/card fields (English and Turkish field names) arrive pre-masked; paste content is never logged; secret-looking selections/copies are masked; plain-character keydowns aren't logged at all. Masking is heuristic — trace chunks are not masked; treat recordings as sensitive.
• Desktop recordings have NO masking — every keystroke and screenshot is raw. The summarizer warns the agent explicitly.
• Page-JS hardening: the recording binding is nonce-guarded and removed from window; the event stream is rate-limited and capped (no fake-event injection, no disk-fill).
• One active recording per mode (lock files); restarting with the same name archives the previous evidence as .bak instead of appending.
• Replaying: generated skills use semantic Playwright locators (browser) or an accessibility-tree computer-use layer such as cua-driver (desktop) — never raw coordinates unless there is no better target.

Files

License

MIT