The Assumption That Native Pairings Are Safer
Faced with a choice between Claude Code, Codex, and various third-party agents, many people default to the "native" pairing — reasoning that a company's own tools must be optimized for its own models. That instinct is even stronger for solo founders and product people who can't afford to hire a developer. With no time to test each tool individually, they just pick the option that shares a brand name and move on.
But an AI coding tool isn't just a model. A control layer sits on top of it — deciding how tools get called, how prompts are structured, and what order tasks run in — and that's what turns a chat model into an agent that actually edits code. This layer is called the harness. The belief that native pairings perform better rests on the assumption that a company tunes its harness for its own model, so it should solve more tasks. A recent paper actually put that assumption to the test.
The Study
In a paper by Mohsen Arjmandi, the model is held constant while only the harness changes, to see how much that alone shifts performance. The same 80 tasks were run on claude-opus-4-8 using both its native harness (claude-agent-sdk) and a neutral harness (deepagents), and on gpt-5.5 using both the openai-codex SDK and deepagents.
The tasks weren't drawn from a public benchmark; they came from a private pool of 256 problems the researchers maintain, mixing tasks that work with existing code repositories and contest problems issued after the models' training cutoffs — a design meant to reduce the odds that a model had simply memorized the answer. Of the 800 planned runs, 792 were evaluated by an isolated grader.
What the Study Found
On average, neither harness came out ahead. On Opus 4.8, the native harness solved 48.8% of tasks versus 50.0% for the neutral harness — a 1.25-point gap in the neutral harness's favor, but the 95% confidence interval ran from -10.0 to +7.5, too wide to call it a real difference. On GPT-5.5, the native harness solved 55.6% versus 54.4% for the neutral harness — a 1.25-point edge for native this time — but again the confidence interval spanned zero.
Similar averages masked very different internals. Once the Opus 4.8 results were split by task type, they pointed in opposite directions.
The neutral harness won on repository tasks, the native harness won on contest tasks, and the two effects canceled out to push the average close to zero. A permutation test that randomly shuffled the labels put the p-value at 0.003.
Cost told a different story. Recalculating each run's usage at fixed list prices, the neutral harness cost 1.3 to 1.6 times more per solved task on Opus 4.8, and 1.2 times more on GPT-5.5. Accuracy and completion rate didn't move together, either: of the 81 runs canceled for hitting the time limit, 22 had already produced a patch that would have passed.
What This Means in Practice
If you start comparing AI coding tools by brand, you'll miss exactly the split this study surfaced. Start by writing down what kind of work you actually do. If you're mostly patching code in a live storefront or booking system, that's closer to a repository task — and on Opus 4.8, the native pairing lagged behind in that category. If you're frequently solving new algorithmic problems from scratch, that's closer to a contest task — and on the same model, the native pairing pulled well ahead. The task-type split was drawn after the fact, so it isn't an established rule, but the habit of asking what kind of task you actually have is worth carrying over regardless.
Check the bill, too. Even with identical solve rates, the cost per solved task varied by 1.2x to 1.6x. Don't just compare monthly subscription prices — run the same task through both tools once each and calculate the actual per-task cost from your usage logs.
Look at your time-limit settings as well. Some runs produced a correct answer but were canceled for hitting the deadline, so set a generous limit or check whether the tool can preserve intermediate results.
Caveats
The researchers flagged their own limitations. The split between repository and contest tasks was defined after looking at the data, so it needs a purpose-built follow-up study to confirm. The cost figures are estimates based on observed usage, and 58 runs on the Anthropic account left no usage record at all. Depending on how that missing share gets allocated, the Opus cost ratio swings from 0.7x to 2.3x, so the cost ranking isn't settled. This version is a correction of an August 2026 draft, whose cost figures rested on a measurement flaw in the original instrumentation; the runner, grader, reanalysis code, and aggregate results are public, though the underlying tasks themselves are not. Rather than importing these exact numbers when comparing coding tools, the safer move is to borrow the procedure — run your own tasks through both harnesses and see for yourself.
