Evaluate the harness, not the demo.
August 16, 2026
Agent harnesses demo well and cost badly. The demo shows a task completing; the invoice shows what the task cost, how many times it retried, and which model served each attempt. This is a short checklist for the second question, worked against DeepSeek's newly published open-source harness so the criteria stay concrete.
Getting it running takes about five minutes
The quick path is npx @deepseek-ai/dsh web, which starts a local web UI on port 3080. Two steps before it becomes usable: enter a DeepSeek API key under Settings then Models — the route enables immediately, no restart — and activate a project directory with Choose workspace, since the session composer is unavailable until one is active. From source it is a clone, then install, build, and run, on Node 22.19 or newer with pnpm pinned by Corepack.
Once running, the agent reads and edits files, runs commands, delegates to subagents, and plans, prompting for confirmation according to the active permission policy.
Criterion 1: is model choice swappable?
Ask where the provider is named. In dsh the model service is a pluggable interface with several implementations behind it, and the agent loop consumes a provider-neutral stream without knowing which backend is live. One of those implementations replays a recorded session without paying for inference again, which is the sort of thing that only exists when the interface is real.
If the harness you already run names a provider inside application code, that is your answer, and it is the finding worth taking away from the exercise.
Criterion 2: is there a complete record?
Every session in dsh is an append-only log of typed events, treated as the single source of truth for the interaction. Turn and step boundaries are explicit, and token usage travels attached to the model output that incurred it. Message history is derived from the log rather than stored beside it, so there is one record and it cannot disagree with itself.
The test for any harness: could you reconstruct last week's most expensive session, including which model served each call, from what the system already writes down?
Criterion 3: are there enforcement points?
Two, in this case. One before the model request is issued, whose decision is documented as authoritative. One before any tool body executes, backed by permission guards that can deny or abstain but never grant, and that treat an unanswerable approval prompt as a denial. That is fail-closed behaviour, and it is the property to look for rather than the specific names.
Criterion 4: does it run headless?
A harness that only works interactively cannot be put in CI, and a harness that cannot be put in CI cannot be measured on a schedule. dsh ships a headless one-shot profile and an automation server speaking JSON-RPC over stdio. It also ships a self-referential demo that inspects and modifies its own live plugin runtime, which is a fair way to confirm the plugin claim is structural rather than positioning.
Criterion 5: what does the config layering look like?
Composition here runs through profiles — named plugin compositions storing an ordered bundle list plus patches — resolving in four layers: profile bundles, then profile patches, then home-level patches, then command-line overlays, with later layers winning. Knowing that order is what turns "why is it using that model" from an investigation into a lookup.
What to do with the result
- Run the five criteria against the harness you already use in production. The gaps are your roadmap.
- Record the switching cost for your default model, in engineering days. Re-measure it each quarter.
- Put a step ceiling and a token budget in place before scaling agent usage, not after the first surprising invoice.
- Require that agent cost data come out of the system's own record rather than a parallel instrumentation effort.
Related
- Your agent harness decides how much lock-in you buy
- An agent audit trail finance can actually read
- Budget enforcement belongs inside the agent loop
- Open-weight model TCO
FAQ
How do you try DeepSeek Harness quickly?
Run npx @deepseek-ai/dsh web, which starts a local web UI on port 3080. Enter a DeepSeek API key under Settings then Models, and activate a project directory with Choose workspace. The session composer stays unavailable until a workspace is active.
What is a profile in an agent harness?
A profile is a named composition of plugins: an ordered list of bundles plus user patches. DeepSeek Harness ships three templates, covering the base runtime, the browser UI, and a headless one-shot runner.
What does developer preview mean for an evaluation?
It means the interfaces will change. Evaluate and prototype, pin the version you build against, and do not schedule production migration work off a preview API. Treat the evaluation as a benchmark for what you already run.
What should an agent harness evaluation actually test?
Whether model choice is swappable, whether there is a complete session record with token usage attached, whether enforcement points exist before model calls and tool execution, and whether headless and automation modes exist for CI use.