Open Record Replay

What is Record & Replay?

An MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated SKILL.md can then be given to an agent so the agent can perform the workflow later.

• Record — Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference.
• Compile — An LLM transforms the event log and scene descriptions into reusable SKILL.json and human-readable SKILL.md files.
• Use — The server installs the generated SKILL.md into the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance.

Demonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill.

Demo

https://github.com/user-attachments/assets/5ec93a7a-f285-4265-99a7-4ea60349b0ee

Installation

Prerequisites

• Python 3.10+
• uv package manager
• A VideoDB API key

1. Clone and install

git clone https://github.com/video-db/open-record-replay.git
cd open-record-replay
uv sync

2. Set your API key

Create a .env file in the project root:

VIDEODB_API_KEY=your_VIDEODB_API_KEY

3. Configure your MCP client

Add to your MCP config (claude_desktop_config.json, VS Code MCP settings, etc.):

{
  "mcpServers": {
    "videodb-record-replay": {
      "command": "uv",
      "args": ["run", "python", "server.py"],
      "cwd": "/path/to/open-record-replay"
    }
  }
}

4. Restart your client

Five tools and two resources will appear. You're ready to record.

macOS — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first:

uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, enable the terminal process in System Settings > Privacy & Security > Accessibility and Input Monitoring, then rerun.

Windows — Uses UI Automation. No additional setup required beyond the standard install.

Linux — Uses AT-SPI. Ensure at-spi2-core is installed and your desktop environment has accessibility enabled.

Usage

Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured.

record_skill_tool("my-workflow", lead_in_seconds=5)
    → Agent tells the operator recording is active
    → Operator switches to the target app and performs the workflow
    → Operator returns to the MCP client and says "stop"
    → Agent calls stop_recording_tool(trim_end_seconds=10)
    → Agent calls compile_skill_tool(video_id, "my-workflow")
    → Agent verifies/reports global_skill_md_path for future use

The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With lead_in_seconds=5, the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds.

Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, trim_end_seconds=10 ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window.

If VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call compile_skill_tool with video_id="" or video_id="none" to compile from events alone.

How It Works

Human performs workflow
        |
        v
+------------------------------------------------------------------+
| Native AX/UIA/AT-SPI hooks -> events.jsonl                       |
| VideoDB Capture SDK -> video_id, when capture succeeds           |
+------------------------------------------------------------------+
        |
        v
+------------------------------------------------------------------+
| Compiler                                                         |
| events.jsonl + optional scene summaries/transcript -> SKILL.json |
| SKILL.json -> SKILL.md                                           |
+------------------------------------------------------------------+
        |
        v
+------------------------------------------------------------------+
| Agent skill install                                              |
| ~/.mcp-videodb/skills/<name>/SKILL.md                            |
| ~/.codex/skills/<name>/SKILL.md, unless overridden               |
+------------------------------------------------------------------+

The accessibility hooks provide the action log. VideoDB scene indexing adds visual context when screen capture is available. The compiler combines those signals into skill files that a future agent can use to perform the workflow.

Features

Tools

Resources

Skill Output

Compiled skills land in ~/.mcp-videodb/skills/<name>/:

Every generated SKILL.json includes an execution_strategy — web_browser, desktop_app, hybrid, terminal, file_system, or unknown — so the agent consuming the skill knows which tool path to prefer. Every SKILL.md includes execution guidance, continuous improvement guidance, and agent tool-priority guidance.

After SKILL.md is created, compile_skill_tool also installs it into the agent's global skills directory, ~/.codex/skills/<name>/SKILL.md by default, and returns global_skill_md_path plus an agent_instruction reminding the agent to verify or report the global install. Agents should ensure this global install step has happened so the skill is available in future runs. Set CODEX_HOME to change the base Codex directory, or set AGENT_GLOBAL_SKILLS_ROOT to override the global skills directory directly.

Architecture

open-record-replay/
├── server.py                 # FastMCP entry point, tool and resource definitions
├── state.py                  # Shared server state singleton
├── config.py                 # Constants, .env loading
├── registry.py               # Skill CRUD + versioning
│
├── capture/
│   ├── recorder.py           # Records native accessibility events + optional VideoDB capture
│   ├── ax_client.py          # JSONL IPC wrapper for native accessibility companion
│   ├── capture_client.py     # VideoDB Capture SDK wrapper
│   └── native/
│       ├── ax_hook_win32.py   # Windows: UI Automation + keyboard polling + TCP IPC
│       ├── ax_hook_darwin.py  # macOS: Accessibility API + pynput + pipe IPC
│       └── ax_hook_linux.py   # Linux: AT-SPI + pynput + pipe IPC
│
├── compiler/
│   ├── compiler.py           # LLM compilation: index scenes → match events → prompt → normalize
│   ├── prompts.py            # LLM system prompt for structured skill generation
│   ├── md_generator.py       # Converts SKILL.json to agent-readable SKILL.md
│   ├── tool_manifest.py      # Surface-to-tool mapping for replay guidance
│   └── recommended_tools.json
│
├── schema/
│   └── skill.schema.json     # JSON Schema (draft-07) for SKILL.json validation
│
├── scripts/
│   └── smoke_macos_hook.py   # macOS AX hook permission helper

Troubleshooting

• Verify VIDEODB_API_KEY is set in .env and is valid
• Run request_capture_permissions_tool and approve any permission prompts
• On macOS, check Screen Recording and Accessibility permissions
• Check that no other application is using the accessibility hook

• Ensure the recording has meaningful UI interactions (not just idle time)
• Try events-only compilation (video_id="") if video indexing is slow
• The LLM may need another generation attempt — the compiler is configured for up to 2 attempts

# Reset permissions and try again
uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, manually enable the terminal in System Settings > Privacy & Security > Accessibility and Input Monitoring.

• Ensure the app being recorded has UI Automation support (most modern apps do)

Community & Support

• Docs: docs.videodb.io
• Issues: GitHub Issues
• Discord: Join the VideoDB community
• API Key: console.videodb.io