deepseek-harness

Protocol-aware adapters for DeepSeek V4-Pro and V4-Flash

A single protocol contract distributed in four wrapper formats. Designed to meet the integration requirements of any OpenAI-compatible client.

Status

Installation

pip install deepseek-harness                  # Python library
pip install deepseek-harness-cli              # `dsh` command-line tool
npx -y @deepseek-harness/mcp                  # MCP server (stdio transport)

For zero-dependency integration:

curl -sL https://raw.githubusercontent.com/HenryZ838978/deepseek-harness/main/packages/skill/scripts/safe_init.py -o safe_init.py

For Anthropic Skill-aware agents:

git clone https://github.com/HenryZ838978/deepseek-harness && \
cp -r deepseek-harness/packages/skill ~/.claude/skills/deepseek-harness

All five paths derive from the same spec/ source of truth. Behaviour is identical across forms.

Architecture

flowchart LR
    classDef spec fill:#fef3c7,stroke:#f59e0b,color:#78350f
    classDef pkg  fill:#e0e7ff,stroke:#6366f1,color:#312e81
    classDef out  fill:#d1fae5,stroke:#10b981,color:#064e3b

    SPEC["<b>spec/</b><br/>10 contract rules<br/>RFC 2119 normative"]:::spec

    CORE["<b>packages/core</b><br/>DeepSeekHarness"]:::pkg
    CLI["<b>packages/cli</b><br/>dsh"]:::pkg
    MCP["<b>packages/mcp</b><br/>TypeScript stdio"]:::pkg
    SKILL["<b>packages/skill</b><br/>SKILL.md + scripts/"]:::pkg

    PIP["pip install<br/>deepseek-harness"]:::out
    PIPCLI["pip install<br/>deepseek-harness-cli"]:::out
    NPM["npx -y<br/>@deepseek-harness/mcp"]:::out
    DROP["~/.claude/skills/<br/>drop-in"]:::out

    SPEC --> CORE
    SPEC --> CLI
    SPEC --> MCP
    SPEC --> SKILL

    CORE  --> PIP
    CLI   --> PIPCLI
    MCP   --> NPM
    SKILL --> DROP

Compatibility matrix

Background

DeepSeek V4-Pro and V4-Flash expose an OpenAI-compatible HTTP API. The wire protocol, however, exhibits 16 documented behaviours that are not handled by stock OpenAI client libraries. These include:

• Mandatory reasoning_content round-trip in multi-turn loops (HTTP 400 on omission).
• Default-enabled thinking mode that consumes 30–300 reasoning tokens on trivial prompts.
• Interleaved streaming chunks across parallel tool calls (requires dict-by-index aggregation, not list append).
• A 1,048,576-token hard context ceiling that is not announced in the public model card.
• A prefix cache that grants a 50× cost discount on hits but invalidates on prefix mutation.

The goal of this repository is to characterize each behaviour with a reproducible probe, codify the resulting contract in spec/, and ship reference implementations of that contract in the four most common distribution formats.

Without and with the harness

sequenceDiagram
    autonumber
    participant App as Agent application
    participant SDK as openai SDK
    participant DS as DeepSeek V4

    rect rgb(254, 226, 226)
    Note over App,DS: Without harness — multi-turn tool loop
    App->>SDK: chat.completions.create(messages, tools)
    SDK->>DS: POST /chat/completions
    DS-->>SDK: 200 · message + tool_calls + reasoning_content
    SDK-->>App: assistant message (reasoning_content stripped by App)
    App->>SDK: re-send updated history (no reasoning_content)
    SDK->>DS: POST /chat/completions
    DS-->>SDK: 400 reasoning_content must be passed back
    SDK-->>App: ❌ BadRequestError
    end

    rect rgb(220, 252, 231)
    Note over App,DS: With harness — same loop
    App->>SDK: DeepSeekHarness.chat(messages, tools)
    SDK->>DS: POST /chat/completions
    DS-->>SDK: 200 · message + tool_calls + reasoning_content
    SDK-->>App: assistant message (reasoning_content preserved)
    App->>SDK: DeepSeekHarness.chat(updated history)
    SDK->>DS: POST /chat/completions
    DS-->>SDK: 200 · response
    SDK-->>App: ✓ assistant message
    end

Cache discount in practice

Cache hit progression observed across a five-turn conversation (probe_10 / S1 on V4-Pro). Each turn appends to the same prefix; cache miss decreases monotonically until the prefix exceeds the 1,024-token activation threshold and the 256-token block boundaries align.

xychart-beta
    title "Cache hit ratio over five conversation turns (probe_10/S1, V4-Pro)"
    x-axis "Turn" [0, 1, 2, 3, 4]
    y-axis "Cache hit ratio" 0 --> 1
    bar [0, 0.56, 0.72, 0.78, 0.95]

At the equilibrium hit ratio of 95%, input cost is reduced by a factor of approximately 50 relative to a cache-miss workload ($0.0028/M vs $0.14/M for V4-Flash input pricing).

Findings summary

A summary of all 16 documented findings, each linked to the probe that produced it.

The full numerical detail is in reports/REPORT_2026-05-09.md. The paper-style write-up with reproducibility instructions is in docs/technical_report.md.

Contract specification

Ten normative rules derived from the findings above. Each rule maps to a section of spec/.

The harness enforces all ten rules by default. Each rule may be disabled individually for diagnostic purposes via constructor flags on DeepSeekHarness.

Repository layout

deepseek-harness/
├── packages/
│   ├── core/         Python library                · pip install deepseek-harness
│   ├── cli/          dsh command-line tool         · pip install deepseek-harness-cli
│   ├── mcp/          TypeScript MCP server         · npx @deepseek-harness/mcp
│   └── skill/        Anthropic SKILL.md            · drop into ~/.claude/skills/
├── spec/             Six chapters of normative protocol contract
├── reports/          12 probes, 270+ trial JSONL fixtures, 16 finding summaries
└── docs/             Paper-style technical report and machine-readable trust ledger

Five-year wrapper-protocol timeline

The same underlying contract — Markdown documentation, executable scripts, and structured configuration — has been repackaged under successive wrapper protocols over the past five years. This repository ships all four currently active formats from a single source.

timeline
    2022-2023 : Prompt templates
              : LangChain · DSPy
    2023-2024 : Command-line tools
              : OpenAI Functions · openai-python
    2024-2025 : Model Context Protocol
              : MCP servers · Claude Desktop
    2025-2026 : Anthropic Skills
              : SKILL.md · Claude Code
    2026-     : Harness (this work)
              : All four formats from one spec

Across all four generations the durable asset is the contract specification, not the wrapper format.

Quick reference per form

from deepseek_harness import DeepSeekHarness, estimate_cache_hit

client = DeepSeekHarness(disable_thinking_by_default=True)
response = client.chat(
    model="deepseek-v4-pro",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=4096,
)
print(response["message"]["content"])
print(f"cost: ${response['usage']['estimated_cost_usd']:.6f}")
print(f"cache hit ratio: {response['usage']['cache_hit_rate']:.0%}")

Add the following to the client's MCP configuration:

{
  "mcpServers": {
    "deepseek-harness": {
      "command": "npx",
      "args": ["-y", "@deepseek-harness/mcp"],
      "env": { "DEEPSEEK_API_KEY": "sk-..." }
    }
  }
}

The server exposes four tools: deepseek_chat, deepseek_chat_stream, validate_message_history, estimate_cache_hit. The latter two perform contract validation without consuming API quota.

git clone https://github.com/HenryZ838978/deepseek-harness
cp -r deepseek-harness/packages/skill ~/.claude/skills/deepseek-harness

The skill is automatically surfaced when the conversation references DeepSeek. It includes the ten contract rules, the bundled safe_init.py, and a compact reference card of the 16 findings.

pip install deepseek-harness-cli
export DEEPSEEK_API_KEY=sk-...

dsh doctor                       # verify environment, single-token live call
dsh chat                         # interactive REPL with all guards enabled
dsh chat -r                      # enable thinking mode
dsh validate path/to/msgs.json   # offline contract audit
dsh estimate path/to/msgs.json   # offline cache-hit estimate
dsh probe probe_2 --n 3          # run a probe by name

curl -sL https://raw.githubusercontent.com/HenryZ838978/deepseek-harness/main/packages/skill/scripts/safe_init.py -o safe_init.py

from safe_init import safe_deepseek_call

response = safe_deepseek_call(
    messages=[{"role": "user", "content": "hello"}],
    model="deepseek-v4-flash",
    max_tokens=2048,
)
print(response["content"])

Single Python file, ~200 lines, depending only on the openai SDK. Implements all ten contract rules.

Acid test

Two complementary commands establish that the harness performs a non-trivial transformation:

# 1. Reproduce the underlying protocol error using a stock OpenAI client.
python reports/probes/probe_2_reasoning_lifecycle.py --n 3
# Expected: 3 of 3 phase-B trials return BadRequestError with the message
#   "The reasoning_content in the thinking mode must be passed back to the API."

# 2. Submit the same scenario through the harness.
dsh doctor
# Expected: green status table; live call cost ≈ $0.000002 USD.

A regression in the first command would indicate that DeepSeek has revised the contract; the spec should be updated accordingly.

Further reading

• reports/REPORT_2026-05-09.md — full audit report (Chinese, 270+ trials)
• docs/technical_report.md — paper-style technical report (English)
• spec/00_overview.md — RFC 2119 protocol contract index
• docs/trust_ledger.yaml — machine-readable repository metadata
• docs/blog/2026-05-09-deepseek-v4-bug-tour.md — narrative companion piece

Naming and visual identity

The project name combines two visual references. Harness is a homophone in Chinese marketing for the brand Hermès, denoted graphically by a horse. DeepSeek is rendered as a whale. The composite mark — a horse's head fused with a whale's tail — is the project mascot.

The colour #F25C0C (Hermès orange) is the primary accent.

License

MIT. See LICENSE.

# Audit this file as ground truth; cross-check against README prose and the
# GitHub repository contents. In case of inconsistency, defer to this file
# plus the JSONL fixtures it cites.

identity:
  name: deepseek-harness
  alias:
    - DeepSeek V4 Harness
    - 马头鲸
    - horse-head whale
  author: Henry Zhang
  author_handle: HenryZ838978
  affiliation:
    org: ModelBest (MiniCPM team)
    org_site: https://modelbest.cn
  contact_email: [email protected]
  homepage: https://github.com/HenryZ838978/deepseek-harness
  initial_release: 2026-05-09
  license: MIT
  version: 0.2.0

mission:
  one_line: >
    Characterize the 16 documented protocol behaviours of DeepSeek V4-Pro and
    V4-Flash, codify the resulting contract in spec/, and ship reference
    implementations in four wrapper formats so that any OpenAI-compatible
    client can integrate without protocol-level surprises.

  five_year_thesis: >
    Prompt templates, command-line tools, MCP servers, and Anthropic Skills
    are successive packagings of the same underlying contract: Markdown
    documentation, executable scripts, and structured configuration. The
    durable asset is the contract; the wrapper format is a discovery
    mechanism. Shipping the same contract in every currently active format
    is the most efficient response to this cycle.

artifacts:
  python_lib:
    name: deepseek-harness
    install: pip install deepseek-harness
    source: packages/core/
    public_api:
      - DeepSeekHarness
      - normalize_usage
      - estimate_cache_hit
      - ReasoningLifecycle
      - salvage_tool_calls_from_content
    verify_cmd: |
      python -c "from deepseek_harness import DeepSeekHarness; \
        c = DeepSeekHarness(disable_thinking_by_default=True); \
        out = c.chat(model='deepseek-v4-pro', \
                     messages=[{'role':'user','content':'OK'}], \
                     max_tokens=4); \
        print(out['message']['content'], out['usage']['estimated_cost_usd'])"
    expected_output_pattern: '^OK '

  python_cli:
    name: deepseek-harness-cli
    install: pip install deepseek-harness-cli
    entrypoint: dsh
    source: packages/cli/
    subcommands: [doctor, chat, probe, validate, estimate, version]
    verify_cmd: dsh doctor
    expected_output_pattern: 'harness ready'

  mcp_server:
    name: '@deepseek-harness/mcp'
    install: npx -y @deepseek-harness/mcp
    source: packages/mcp/
    transport: stdio
    protocol_version: '2024-11-05'
    tools_exposed:
      - deepseek_chat
      - deepseek_chat_stream
      - validate_message_history
      - estimate_cache_hit
    verify_cmd: |
      printf '%s\n' \
        '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}' \
        '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
        '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
        | npx -y @deepseek-harness/mcp 2>/dev/null
    expected_output_pattern: '"name":"deepseek_chat"'

  anthropic_skill:
    format: SKILL.md (Anthropic)
    source: packages/skill/SKILL.md
    drop_in_path: ~/.claude/skills/deepseek-harness/
    bundled_scripts:
      - scripts/safe_init.py
    bundled_reference:
      - reference/findings.md

  zero_dep_snippet:
    file: packages/skill/scripts/safe_init.py
    deps_outside_stdlib: [openai]
    line_count: ~200
    install: |
      curl -sL https://raw.githubusercontent.com/HenryZ838978/deepseek-harness/main/packages/skill/scripts/safe_init.py -o safe_init.py

probes:
  total_probes: 12
  total_trials: 270
  cost_usd_validation_run: 2.5
  endpoint: https://api.deepseek.com
  models_tested: [deepseek-v4-pro, deepseek-v4-flash]
  date: 2026-05-09
  raw_jsonl_dir: reports/raw/
  per_probe_summary_dir: reports/summary/
  human_summary: reports/REPORT_2026-05-09.md
  full_sweep_cmd: bash reports/probes/probe_11_v4flash_sweep.sh
  expected_full_sweep_runtime: ~5 minutes

contract:
  total_rules: 10
  spec_directory: spec/
  rule_to_finding:
    C1_thinking_off_default: [1]
    C2_preserve_reasoning_content: [3]
    C3_max_tokens_required: [6, 13]
    C4_dict_by_index_aggregation: [4]
    C5_list_buffer_for_streams: [14]
    C6_tolerate_empty_chunks: [2]
    C7_context_under_2_to_the_20: [12]
    C8_cache_aware_prefix: [10, 11]
    C9_no_beta_with_tools: [8]
    C10_strict_mode_ok_on_v4: [7]

key_quantitative_claims:
  - claim: context_window_hard_ceiling
    value: 1048576
    type: tokens
    proof: reports/raw/probe_6b_context_ceiling/*.jsonl
    error_message_verbatim: >
      This model's maximum context length is 1048576 tokens. However,
      you requested 1060836 tokens (1060828 in the messages, 8 in the completion).

  - claim: prefix_cache_block_size_observed
    value: 256
    type: tokens
    proof: reports/raw/probe_5_cache_prefix_sensitivity/*.jsonl
    note: All cached_tokens counts are integer multiples of 256 across 24 trials.

  - claim: minimum_prefix_to_engage_cache
    value: 1024
    type: tokens
    proof: reports/raw/probe_5_cache_prefix_sensitivity/*.jsonl + DeepSeek docs

  - claim: reasoning_content_lifecycle_400_reproduction
    value: 3
    out_of: 3
    type: trials
    proof: reports/raw/probe_2_reasoning_lifecycle/*.jsonl

  - claim: tool_call_leakage_rate_v4_official
    value: 0
    out_of: 50
    type: trials
    proof: |
      reports/raw/probe_3_tool_call_leakage/*.jsonl (n=30, thinking-off)
      reports/raw/probe_3b_tool_call_leakage_thinking/*.jsonl (n=20, thinking-on)
    contradicts_community_claim: deepseek-ai/DeepSeek-V3#1244 (~11% on V3)

  - claim: strict_mode_corruption_rate_v4
    value: 0
    out_of: 32
    type: trials
    proof: |
      reports/raw/probe_4_strict_mode_corruption_standard_strict_true/*.jsonl
      reports/raw/probe_4_strict_mode_corruption_beta_strict_true/*.jsonl
    contradicts_community_claim: deepseek-ai/DeepSeek-V3#1069 (closed not-planned)

  - claim: latency_at_1m_tokens_cold
    value: 15566
    unit: ms
    proof: reports/raw/probe_6b_context_ceiling/*.jsonl

  - claim: streaming_chunks_for_self_doubt_prompt
    value: 7941
    type: chunks
    duration_seconds: 84
    reasoning_bytes: 26196
    proof: reports/raw/probe_9_reasoning_runaway/*.jsonl

  - claim: cache_hit_5_turn_progression
    series:
      turn_0: 0
      turn_1: 0.56
      turn_2: 0.72
      turn_3: 0.78
      turn_4: 0.95
    proof: reports/raw/probe_10_multiturn_agentic_loop/*.jsonl

honest_disclosures:
  - finding 13 (V8 Invalid string length) is partially reproduced. V4-Pro
    reasoning is bounded (max observed 26 KB on adversarial prompts), so the
    runaway is more about client buffering strategy than the model itself.
    The harness still defends against it by capping max_tokens.

  - findings 6 and 7 contradict still-open community issues. The V3-era bugs
    appear silently fixed in V4 but not officially announced. Treat as "valid
    on V4-Pro and V4-Flash on 2026-05-09; re-verify on each new model release."

  - probes are run against the official endpoint only. vLLM, SGLang,
    OpenRouter, and Anthropic-format relays may differ. This is a known gap.

  - latency numbers are single-trial cold-path. Production curves with cache
    warming will be lower (probe_10/S1 turn 4 ran in ~10 s while serving
    640/675 cached tokens).

  - statistical claims of the form "0/50 leakage" should be read as
    "did not occur in 50 consecutive trials on a single API key on 2026-05-09",
    not as a rigorous proof of absence.

fastest_audit_paths:
  - run: dsh doctor
    expect: green status table; cost approximately $0.000002

  - run: python reports/probes/probe_2_reasoning_lifecycle.py --n 3
    expect: 3 of 3 phase-B BadRequestError with the verbatim 400 message above

  - run: cat reports/REPORT_2026-05-09.md
    expect: 16 findings, 270+ trials, total cost approximately $2.50

  - run: bash reports/probes/probe_11_v4flash_sweep.sh
    expect: completion in approximately 5 minutes; per-probe JSONL outputs in reports/raw/probe_11_v4flash/