The harness is the product

When people compare coding agents, they usually compare models. Which is smarter, which is cheaper, which tops the leaderboard this week. It's a reasonable instinct, and on many tasks it's the wrong variable.

Over the last year, enough public results have accumulated to isolate the effect of the harness: the loop, tools, prompts and plumbing around the model. Hold the model fixed, change the harness, and the score moves. On terminal-heavy work it moves by tens of points.

This post walks through that data, and then through what we built into Cusp because of it.

Same model, different harnessCopy Anchor Link

Terminal-Bench measures agents on real tasks in a terminal: build this, fix that, get this service running. Several groups have run the same model through different harnesses and published the results. The table below quotes their numbers, with the harness names as the sources give them.

Sources: GPT-5.4 from the AHE paper; GPT-5.4-mini from the Terminal-Bench 2.1 announcement; Opus 4.6 and Haiku 4.5 from arXiv 2603.28052.

That is a 29.8-point spread with the model held fixed. What changed is how the model was asked to work.

The gains are largest for weaker modelsCopy Anchor Link

The most important number for us is a comparison inside the AHE paper. Its authors searched for a better harness, froze it, and applied it to other models. The frozen harness gave DeepSeek-v4-flash +10.1 points, and GPT-5.4 only +2.3 (source).

That pattern shows up repeatedly. Frontier models paper over a mediocre harness; they recover from awkward tools and missing context on their own. Smaller and open models can't, so they gain the most from a harness that meets them halfway.

Two further data points from outside Terminal-Bench point the same way. Both come from the projects that publish the benchmark:

• The Aider code-editing leaderboard scores each model per edit format and tracks how often its edits are well-formed; the same model's results change with the format (leaderboard).
• Hash-anchored line edits moved Grok Code Fast from 6.7% to 68.3% on oh-my-pi's edit benchmark, as reported by the project (source).

And one finding points the other way: a good neutral harness equalizes models. Terminus-KIRA reports near-identical Terminal-Bench scores for three vendors' models: GPT-5.3-Codex at 75.5%, Opus 4.6 at 75.7%, Gemini 3.1 Pro at 74.8% (source). With a good enough harness, the choice of model starts to become a question of price and preference as much as capability.

That is the bet Cusp is built on.

Where the spread is smallCopy Anchor Link

To be fair to the other side: on patch-style tasks like SWE-bench, the spread is much smaller. A 100-line bash-only agent, mini-swe-agent, keeps pace with the labs' own harnesses in published comparisons, and it's the harness the SWE-bench Pro and DeepSWE leaderboards use. When the task is "read these files and write a diff," frontier models don't need much help.

But for open models reached through a gateway, even similar pass rates hide a cost difference.

One study of Qwen 3.6 Plus and MiniMax M2.5 found pass rates within 0 to 8 points across Goose, OpenCode and OpenHands, but tokens per solved task varying 40×, from about 30k to about 1.3M (source).

The harness decided the bill.

What actually helpsCopy Anchor Link

Not every harness change helps. In the AHE work, a 79-line system prompt rewrite about "discipline" lowered the score by 2.3 points (source). The Meta-Harness authors report that all six of their prior prompt and completion-flow iterations regressed (source). Prompt wordsmithing is the least reliable lever.

What does help is mostly mechanical. Ranked roughly by how much the evidence supports it:

1. Give the model the tool formats it was trained on, through native tool calling.
2. Round-trip reasoning state. Dropping reasoning traces cost GPT-5-Codex 30% on one tool vendor's internal benchmark (source). On some providers, a missing reasoning block is an HTTP 400.
3. Make the shell work for real terminals: a non-interactive environment, a PTY for interactive programs, background processes, timeouts that don't kill long builds, bounded output.
4. Bootstrap the environment before the first turn. "Not installed" and "not in PATH" account for 24.1% of command failures on Terminal-Bench 2 (source). Collecting the basics up front saved two to four turns in the Meta-Harness work (source).
5. Gate on verification before stopping. A finish hook that forces a verification check was worth +2.2 points in AHE (source).
6. Keep the plan near the end of the context, where the model attends to it.
7. Keep the context cache-stable, and trim old tool output before summarizing.
8. Persist lessons across sessions.

What Cusp doesCopy Anchor Link

Here's how that list maps onto Cusp 0.1, including what isn't finished.

Per-family edit formats. Cusp keeps a profile for each model family and picks the edit tool accordingly. GPT-family models get apply_patch in the V4A format they were trained on, sent as OpenAI's native tool where available. Claude and Gemini get str_replace-style Edit and MultiEdit, with a fuzzy matching ladder that reports when it had to be lenient. Weaker and local models get EditLines: every line in Read output carries a short content hash, and the model edits by anchor instead of reproducing exact text. The canonical tool names stay the same for hooks and permission rules, so your configuration doesn't care which dialect the model speaks.

Reasoning round-trip. Anthropic thinking signatures, OpenAI encrypted reasoning, Gemini thought signatures, and the reasoning fields DeepSeek, Kimi and GLM expect are stored opaquely with each message and replayed byte-exact to the same model. Switching models mid-session sends a safe projection and records that it did.

A shell that works. Every command runs with stdin closed and pagers off, so commands don't hang waiting for input. The working directory persists inside the workspace. A command that outlives its timeout moves to the background instead of dying, and the agent is told when it exits. Process starts dev servers and waits for the port to open. Terminal drives REPLs, debuggers and anything else that needs a real PTY. Output over about 30,000 characters is cut to head and tail, with the full text kept as an artifact the agent can search.

Tolerance for imperfect models. Malformed JSON is repaired, types are coerced against the schema, and tool calls that open models embed in text are extracted, before a turn is allowed to fail. A loop detector watches for repeated tool cycles and repeated errors.

Diagnostics after every edit. After an edit, Cusp runs the type checker or language server and appends new errors to the tool result. The agent sees what it broke in the same step.

Cache-stable context. The system prompt and tool list stay byte-identical from turn to turn. Rarely used tools, and all MCP tools, are deferred and loaded on demand, appended after the core set so the cached prefix stays the same. Cusp sets each provider's cache key and places Anthropic's cache breakpoints itself.

Still in progress. Environment bootstrap at turn zero and a built-in verification gate before the agent stops are being built now. Until then, a Stop hook that runs your tests gives you the same gate today; the hooks docs have an example.

Measuring itCopy Anchor Link

Claims about harness quality should come with runs, not adjectives. We're integrating Cusp with Harbor, the framework behind Terminal-Bench, so it can be run the same way as every other agent on the board. We'll publish our Harbor runs with full configurations and trajectories, across frontier and open models, including the ones where we lose.

Two cautions we'll hold ourselves to. Infrastructure alone moves Terminal-Bench 2 scores by about 6 points (source), so we'll treat deltas under 3 points as noise. And benchmark runs will use a clean scaffold, without project instruction files that could leak answers.

Until those runs are out, we aren't quoting a number for Cusp. The data above is other people's, and it's the reason we built Cusp the way we did.