Watch Skill

Give every AI agent eyes for video—and a way to inspect its own work.

Watch Skill is a local-first video intelligence layer for agents. It turns videos, live streams, meetings, and screen recordings into searchable, timestamped evidence. Agents can understand what happened, retain it across sessions, and cite the exact moment behind an answer.

When the video is the agent's own browser or desktop session, THE LOOP closes the feedback cycle: record the work, evaluate it against plain-language criteria, guide the fix, and verify the result. The same engine is available through skills, 23 MCP tools, a CLI, REST, and native framework adapters.

Watch. Remember. Fix. Verify.

One video layer, three agent capabilities

Start in 60 seconds

Claude Code

/plugin marketplace add oxbshw/watch-skill
/plugin install watch-skill@watch-skill

Run /watch-skill:setup-watch-skill once after installation. It installs the engine, checks the binary dependencies, registers the MCP server, and offers to configure a vision provider.

macOS and Linux

curl -fsSL https://raw.githubusercontent.com/oxbshw/watch-skill/main/scripts/install.sh | sh

Windows

powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/oxbshw/watch-skill/main/scripts/install.ps1 | iex"

Then watch a video and ask a follow-up:

watch-skill watch "https://youtu.be/..." "Summarize the important moments."
watch-skill ask <video_id> "When does the demo first fail?"
watch-skill search "pricing decision"       # search every indexed video
watch-skill serve                           # MCP over stdio

Transcription, OCR, and search can run locally without an API key. For visual Q&A, use Gemini, Anthropic, OpenAI, OpenRouter, or a local Ollama model. See Getting started for manual installation and Configuration for provider and privacy settings.

Why use it

• Evidence instead of frame dumps. Scene detection and perceptual deduplication spend the frame budget on distinct moments. Answers include timestamps, confidence, and the evidence used to support them.
• Persistent video memory. Analyze once, ask again without downloading or transcribing the same video. Hybrid full-text and vector retrieval works within one video or across the entire library.
• Local-first processing. Original-language captions are preferred, local Whisper is the default fallback, and cloud speech-to-text is opt-in. An Ollama configuration keeps the complete pipeline on the machine.
• Flow verification. THE LOOP records an agent's browser, screen, or window; checks the result against plain-language criteria; and produces before/after proof after a fix.
• Corrections that persist. report_mistake stores a local lesson, applies it to related questions, and turns it into a replayable evaluation.
• Measured cost controls. Text-first answers, semantic caching, configurable token budgets, and explicit cheapest, quality_first, and offline_only policies keep the trade-offs visible.
• Multilingual retrieval. Script-aware OCR routing, Arabic normalization, CJK substring matching, and multilingual embeddings support questions across languages.

The repository includes reproducible cost and perception benchmarks. Product claims in this README link to the relevant implementation notes or testable example rather than relying on unqualified marketing numbers.

Works with your agent

The setup command detects supported clients and updates their configuration with a backup. Manual guides are available for every entry below.

Native tools are also available for LangChain/LangGraph, CrewAI, OpenAI Agents SDK, LlamaIndex, and AutoGen; any other framework can use REST or MCP.

The full compatibility matrix separates machine-tested, machine-configured, and documentation-verified integrations. If your agent is missing, the adapter template provides a short contribution path.

Common workflows

Build a searchable video library

watch-skill batch ./recordings --limit 50
watch-skill library overview
watch-skill library ask "What did the team decide about authentication?"

library ask synthesizes evidence across videos and retains per-video timestamp provenance. The library example demonstrates a question whose answer is distributed across four clips.

Verify an agent's browser work

watch-skill loop start \
  --source "browser:http://127.0.0.1:3000" \
  --criteria "Checkout completes and the total is always a valid currency amount"

The loop captures the full interaction, critiques failures, and records proof after the agent applies a fix. Example 14 includes a transient $NaN bug that an end-state screenshot misses.

Export an offline report

watch-skill viewer <video_id> --out video-report.html

The generated page contains its frames, transcript, OCR, cached answers, and cited evidence. It has no external runtime dependencies and can be opened without a server.

Examples

The examples progress from a first watch to agent integration, cross-video memory, and self-verification.

See the example catalog for prerequisites, expected output, and a recommended path through all 16 examples.

Architecture

All interfaces call the same Python core. Skills and agent adapters decide when to use Watch Skill; acquisition, perception, transcription, indexing, answering, and verification remain in src/watch_skill.

flowchart LR
    A["Agents and frameworks"] --> S["Skills · MCP · CLI · REST"]
    S --> AC["Acquire"]
    AC --> P["Scenes · OCR · transcript"]
    P --> I[("Persistent index")]
    I --> Q["Answers · extraction · library"]
    I --> L["Lessons and evaluations"]
    V["Browser · screen · stream capture"] --> C["Loop critic"]
    C --> I

Read Architecture for the data model, provider boundaries, and extension points.

Documentation

Development

git clone https://github.com/oxbshw/watch-skill
cd watch-skill
uv sync --extra all
uv run pytest
uv run ruff check .

See CONTRIBUTING.md for test tiers, documentation standards, and the agent-adapter checklist. Security and privacy reports are covered by SECURITY.md.

Watch Skill is available under the MIT License.