简体中文

What this is

Embodied Eval Automation is an Agent Skill–style workflow and a skills-only Codex plugin for collecting batch episodes from a policy, VLA, world model, or hybrid model against a robot benchmark or simulator.

It does not ship model weights or benchmark code. It teaches an agent how to:

  • confirm the local and remote workspaces before writing;
  • obtain narrowly scoped access without collecting secrets in chat;
  • inventory and reuse existing repositories, environments, checkpoints, and datasets;
  • pin official sources and explain the model/benchmark before execution;
  • distinguish paper, repository, checkpoint, wrapper, adapter, writer, validator, and visualizer capability evidence;
  • compare inference, closed-loop, prediction, collection, and training-readiness modes before execution;
  • validate every required output, one closed-loop episode, a result-blind pilot, and only then an approved study;
  • preserve native model output while deriving comparable episode representations;
  • generate truthful replay/video or a recorded waiver plus user-readable behavior analysis;
  • monitor long jobs, resume safely, verify transfers, and prune only approved data;
  • reconcile repairs, benchmark failures, operational failures, manifests, reports, visualizations, and a reproducibility package.

Why it exists

Batch embodied-AI runs often fail after consuming substantial GPU time because a checkpoint was not pinned, an existing benchmark environment was modified, native predictions were discarded, a transfer was assumed complete, or “success” meant only that a process stayed alive.

This project turns that work into a gated state machine with explicit evidence at every boundary.

flowchart LR
  A["G0 Access and inventory"] --> B["G1 Source and reuse lock"]
  B --> B2["Capability matrix and informed mode selection"]
  B2 --> C["G2-G4 Environment, runtime, side effects"]
  C --> D["G5 Every required output"]
  D --> E["G6 One rollout, three views, replay"]
  E --> F["G7 Result-blind pilot and study choice"]
  F --> G["G8 Goal-driven study or explicit skip"]
  G --> H["G9 Audit, explanation, delivery"]

Quick start

Invoke the skill with a request like:

Use $embodied-eval-automation to collect episodes from <model> on <benchmark>. I have a GPU server but have not decided how to authenticate. First confirm the work directories, permissions, official revisions, reusable assets, target episode set, storage thresholds, and completion criteria. Do not download, install, upload, delete, or start paid compute until I approve those actions.

The skill begins with a short intake. It will not ask you to paste passwords, tokens, private keys, or cloud credentials into the conversation.

Installation

Codex or another Agent Skills–compatible agent

For a stable, reproducible install, ask Codex to use its built-in skill-installer with the tagged skill URL:

Install https://github.com/Yinzhanqing/embodied-eval-automation/tree/v0.2.0/skills/embodied-eval-automation with skill-installer.

The equivalent installer arguments are:

python <path-to-install-skill-from-github.py> --repo Yinzhanqing/embodied-eval-automation --path skills/embodied-eval-automation --ref v0.2.0

The version tag pins the installed behavior. Contributors who intentionally want the newest unreleased changes may replace v0.2.0 with main.

For manual installation, copy the directory:

skills/embodied-eval-automation/

into the agent’s skill directory. Restart or begin a new agent conversation after installation so the skill is discovered.

Run-contract compatibility

  • v0.2.0 creates embodied-run-contract/v2 contracts by default.
  • Existing v1 contracts are recognized only to provide explicit migration guidance.
  • A v1 contract never gains implicit capability, media, study, training, or permission authority.
  • Migrate and review the resulting v2 contract before continuing a v1 run with v0.2.0.

Skills-only Codex plugin

The repository root includes .codex-plugin/plugin.json. Clone the repository into your local plugin source, add it to a local marketplace with Codex’s plugin-creator, then start a new conversation with the plugin enabled. The same package is structured for eventual submission to the public plugin directory.

See Publishing and release checklist before publishing.

What the skill asks for

The intake is intentionally permission-aware:

  1. Model, benchmark, online/offline rollout mode, and target episode set.
  2. Local workspace and remote data-root choices.
  3. Connection method: existing shell, SSH, provider CLI, or managed connector.
  4. Read-only inspection boundary.
  5. Separate approval for installs, downloads, uploads, paid compute, deletion, and Git writes.
  6. Disk/GPU budgets, pause thresholds, transfer destination, and retention policy.
  7. GitHub and Hugging Face access only when needed, using interactive login or existing credential stores.

Outputs

A successful run is expected to produce:

  • host and asset inventory;
  • pinned source and artifact manifests;
  • reuse compatibility matrix;
  • model and benchmark technical report;
  • capability activation matrix, mode cards, and informed selection;
  • run-contract v2, study plan, expected episode ID set, and claim boundaries;
  • model, benchmark, and episode adapters;
  • native/current/candidate episode representations from the same rollout;
  • result-blind pilot, schema, pairing, resource, transfer, repair, and final audit reports;
  • replay/video or waiver, human-readable behavior analysis, static fallback, and a reproducibility package.

The default comparison contract keeps:

  • pair_key independent of model identity;
  • T+1 observations for T executed transitions;
  • every real model call in policy_queries;
  • raw action chunks separate from executed actions;
  • model predictions separate from environment observations;
  • capability and quality flags instead of fabricated values.

Security model

  • Secrets stay in the user’s terminal, credential manager, SSH agent, or provider credential store.
  • Host identity is verified before unattended SSH.
  • Gated-model license acceptance remains a user action.
  • Install, upload, deletion, paid compute, Git writes, and external messaging are distinct approvals.
  • Remote data is pruned only after download, hash verification, archive inspection, audit, path-boundary checks, and explicit authorization.

Read SECURITY.md and PRIVACY.md before using the skill on institutional or confidential infrastructure.

Repository layout

.
├── .codex-plugin/plugin.json
├── skills/embodied-eval-automation/
│   ├── SKILL.md
│   ├── agents/openai.yaml
│   ├── references/
│   ├── scripts/
│   └── assets/templates/
├── examples/
├── tests/
├── docs/
└── .github/

Validation

The validation and audit scripts have no third-party Python runtime dependencies. Replay video encoding is optional and uses FFmpeg only when requested and available.

python skills/embodied-eval-automation/scripts/validate_repository.py .
python -m unittest discover -s tests -v

Scope and limitations

  • This is a workflow skill, not a universal benchmark adapter implementation.
  • Each new model + benchmark pair still requires evidence-backed adapters.
  • It does not bypass gated licenses, institutional policy, cloud billing controls, or benchmark terms.
  • It never treats a full benchmark run as mandatory; the approved scope is the completion boundary.
  • It assesses fine-tuning readiness but does not silently turn an evaluation approval into training.

Contributing

Contributions for new providers, benchmarks, model families, schema validators, and failure playbooks are welcome. Start with CONTRIBUTING.md.

License

MIT. See LICENSE.