2026-07-20
We Debug Our AI Harness on Weak Models on Purpose
How we use weak, cheap models to find bugs strong models hide - and what a nightly 26-task benchmark says about which ones you can trust with real work.
Written by
Arseny Kravchenko
Strong models are excellent at hiding product bugs. Give one a broken tool call or a confusing error, and it may still finish the task.
Weaker models are less forgiving. When the tools, prompts, or runtime are wrong, they fail quickly, which makes them useful for debugging. I think of it as the difference between testing an app on a maxed-out MacBook Pro and testing it on an old ThinkPad.
We run Archestra Chat (our built-in chat interface for working with agents and MCP tools) on weaker models on purpose, record the full trajectory, and investigate the failures they expose. This approach has found defects in file handling, sandbox tooling, provider schemas, and our agent runtime. Fixing those bugs helps every model - not just the weaker ones. Strong models no longer need to spend extra tokens and retries working around our mistakes. As a result our harness works reliably even with weaker models (not only models capable of compensating for its flaws).
The idea came from a customer who asked me a simple question: would Archestra Chat work with cheaper models? I couldn't answer. At the time, we were testing changes one at a time, adjusting tools and prompts, and checking whether the final answer looked right. Sometimes it worked. Sometimes it didn't. We had no repeatable way to measure it. I'm an ML person at heart, and when a system behaves like that, the first thing I build is a validation loop.
That loop became an end-to-end benchmark based on 26 real customer workflows. It starts the actual product, runs each task, and checks the answer against hidden ground truth. Every night, it repeats the same tasks across about ten models. Important to note: this is not an academic benchmark attempt. But it's an advanced integration test that answers two questions: where does our product fail, and how far down the price ladder can we go before the assistant stops doing useful work?
How we benchmark AI agents on real work, not party tricks
I approached this the way you'd approach any serious evaluation: the test has to be something the system can't cheat, and it has to look like reality. That leads to two design pillars.
First, the assistant submits its final answer through a dedicated channel. This channel checks the format, not whether the answer is correct. If the task expects a number, it rejects text. If it expects three candidates, it rejects two.
The answer is checked for correctness only after submission, using deterministic code, not another LLM. The grader and expected answer are hidden from the agent during the run. If the agent could read the answer key, it could game the test, and then the score would be a lie. It can't, so the score means something.
The boundary that makes the score mean something:
Loading diagram...
submit_result never tells the agent whether it was right - only whether the JSON parsed against the task's schema. The expected ground truth lives entirely on the right of that line and is never staged into the sandbox, so the agent has nothing to read off.Second, we don't record just pass or fail. We capture the assistant's entire visible trajectory: every message, tool call, file it touched, and artifact it produced. That turns “it failed” into “here's exactly where it failed.” It's the difference between a number that shames you and a report you can actually use. More on that shortly, because this became the most useful thing we built.
The tasks come from reality, not our imagination. Each one is based on something we actually saw: a real customer workflow or a real incident, lightly anonymized. We have a small set today, and it grows whenever reality hands us a new headache. Once a task goes in, it stays in, a permanent tripwire for that failure mode.
A few examples, to make this concrete:
- Document workflows. Approve invoices and shortlist CVs from mixed file formats while rejecting hidden prompt injections and inconsistent data.
- Incident triage. Find the cause of an outage in a zip of unsorted logs containing plausible red herrings.
- Memory across conversations. Create a file in one conversation, then retrieve and use it in a new one - this exercises the persistent files we shipped in July.
- Live facts. Fetch a repository's current star count, an asset price at a specified time, or the latest package version.
We've tightened them to match how a real employee actually talks: “Here are some invoices. Do the right thing.” That harder version is the one worth measuring.
We also keep a hard boundary around what belongs in the suite. Tasks should be difficult, but solvable with the tools Archestra Chat actually has: file access, web access where needed, the sandbox and skills, and the customer's connected systems. We're not measuring whether a model can do frontier physics, and we're not testing features we haven't built yet. The useful failure mode is: “The product should have been able to do this, but didn't.”
One implementation detail matters a lot: this is not a mock benchmark. It boots the product. The benchmark runs Chat against a real Archestra backend in headless mode. For each environment, it starts a fresh backend on a new port, migrates a fresh database, seeds the providers, skills, MCP servers, and agent surface, drives chat sessions through the product, grades the answers out of band, and tears the instance down.
The harness is public: it lives in the
ai-labs workspace of our monorepo, alongside the benchmark, task suite, and environment definitions.It runs every night in CI against that day's deployed build, across a roster of about ten models. The nightly series is where the numbers later in this post come from.
A single run, end to end:
Loading diagram...
Every box above serves the benchmark's one job: run each task-model combination, produce a graded result and a recorded trajectory, then tear everything down. Nothing here explains why a run failed. That comes later, in a separate tool.
Anyone can ship a demo that runs on a maxed-out MacBook
Here's the part that surprised me.
Think about software performance for a second. Any developer can write an app that feels smooth on the latest MacBook Pro - the machine is so fast it hides your sins. Lazy code, wasteful operations, sloppy logic: none of it shows, because the hardware brute-forces past it. Then you try to run that same app on an old IBM ThinkPad with the red TrackPoint and a spinning hard drive, and it falls apart instantly. Every shortcut you took is suddenly visible.
Premium AI models are the maxed-out MacBooks. They can bulldoze through bad plumbing - a malformed tool call, a confusing error message, a missing fallback - and make the product look healthier than it is. The model works around the bug, your benchmark comes back green, and you learn nothing.
So we did our debugging on the metaphorical old ThinkPad. We deliberately ran the benchmark against weak, cheap models, because they don't self-heal. When something in our plumbing is even slightly wrong, a weak model trips over it immediately and loudly - which means it shows us exactly what to fix. The weak models are a better smoke alarm precisely because they're worse at compensating for our mistakes.
This isn't just a clever debugging trick. Software that survives the old ThinkPad is the software that runs well everywhere. A model that compensates for our bugs is still paying for them in retries, tokens, latency, and fragility. Hardening Chat against the weakest models is how we stop taxing the strongest ones. The test method and the business goal turn out to be the same thing.
You can see that tax in the results table below: two models with similar performance can differ in price by two orders of magnitude.
The real bugs weak models found for us
If I'm honest, the benchmark's biggest contribution so far hasn't been the scoreboard. It's been a bug-finding machine, and most of those bugs were ours, not the models'.
I read the first recordings by hand. After around ten nearly identical failures, I gave up and spent a week building a separate analyzer on top of them. The benchmark grades and records each run. The analyzer then reads those recordings, summarizes what appears to have gone wrong, groups similar failures, and points engineers toward the relevant parts of the product.
The important word is appears. The analyzer produces hypotheses, not truth, so a human still reviews the trajectory, traces the source code, confirms the failure, and decides what to fix. That human step is what turns “the benchmark failed” into a real product improvement instead of an automated blame machine.
It's a map-reduce process over the recorded trajectories. It starts where the previous diagram ended, using the same trajectory files. It only accesses the live product as read-only source code to cross-check its findings against.
Loading diagram...
The hand-off between the two tools is a single artifact: the recorded trajectory. The benchmark never tries to identify the root cause, and the analyzer never runs the product. Keeping them separate lets a human use the analyzer's leads without trusting them blindly.
Here's a representative sample of issues we found and fixed through that loop in a single week, limited to defects you can understand without knowing our prompt stack:
| Area | What broke | What changed |
|---|---|---|
| File handling | Binary uploads could crash the LLM request even though the files were already available in the sandbox. | Keep those files as sandbox references instead of trying to inline unsupported MIME types. #5623 |
| Sandbox output | A command that printed binary data with NUL bytes could crash Postgres persistence and kill the conversation. | Strip NULs before saving sandbox output, and warn models not to stream raw binary to stdout. #5697 |
| Sandbox tools | Incident tasks often arrive as zip files, but the sandbox image did not include zip or unzip. | Add zip and unzip so Chat can inspect archives directly. #5707 |
| Agent runtime | Some rollouts repeated the same tool call hundreds of times until they ended with no answer. | Detect repeated tool-call loops and stop them after a fixed ceiling. #5756, #5786 |
| Provider backend | One provider returned a finish_reason string our API schema did not recognize, so trajectory playback failed. | Preserve arbitrary provider finish reasons instead of rejecting the interaction. #5780 |
| Benchmark harness | Shared benchmark lanes and database connectivity made real product failures hard to separate from harness noise. | Move the benchmark onto dedicated Postgres, isolate lanes by project, make file conflicts explicit, and stop transient auth hiccups from surfacing as fatal errors. #5749, #5787 |
| Sandbox engine | A rare sandbox engine panic could poison the backend process, while orphaned child processes kept hammering the engine. | Treat the panic as retryable, respawn the session, reap child processes, and add keepalive checks. #5797, #5801 |
These fixes came from running hundreds of rollouts through the same product surface, not from engineers manually clicking around Chat. Unit and integration tests remain necessary, but they rarely exercise files, tools, sandboxes, provider quirks, and persistence in one path, and do not bring enough variance to cover a non-deterministic system.
So which models held up? Open-weight vs. frontier results
The question I get when talking to our customers is: do we really have to budget premium models for every seat?
When export controls forced Anthropic to suspend access to Claude Fable 5 in June, we bet every executive who has ever rolled out AI had the same thought: what gets cut off next, and for whom? Access came back three weeks later, but that box doesn't close. You didn't have to use that model to see the risk - and now nobody gets to un-see it. Terms change, access gets revoked, regions get geofenced, prices move. Open-weight models - models you can run yourself instead of depending on one vendor's permission slip - are the obvious hedge.
But a hedge you can't trust isn't a hedge. So we test which cheaper, self-hosted models can handle real employee tasks. That's the benchmark's second axis: we run the same tasks on models ranging from premium APIs to small open-weight models.
The results match what we expected - and a benchmark that agrees with reality where you can already check it is one you can believe where you can't.
Every model below ran the same 26 tasks against the latest daily build. Small changes between builds can add some variance; we're not aiming for full determinism here.
For models on our daily roster, we report the average pass rate across seven consecutive daily runs on the same frozen task set, along with the typical variation between runs. Frontier models were run once on the same day and tasks, and are marked accordingly.
| Model | Open weights | Pass rate | Day-to-day swing, pp1 | Suite cost2 |
|---|---|---|---|---|
| Claude Sonnet 5 | 100% | single run | $19.95 | |
| Claude Opus 4.8 | 96% | single run | $27.60 | |
| GPT-5.6 (sol) | 96% | single run | $7.50 | |
| GPT-5.6 (terra) | 96% | single run | $3.93 | |
| Qwen3.7 Plus | ● | 94% | ± 2 | $0.57 |
| Sakana Fugu-Ultra | 92% | single run | $27.69 | |
| GLM-5.2 | ● | 91% | ± 7 | $1.23 |
| DeepSeek V4 Flash | ● | 88% | ± 7 | $0.34 |
| Claude Fable 53 | 88% | single run | $46.17 | |
| Xiaomi MiMo v2.5 | ● | 86% | ± 4 | $0.27 |
| Qwen3.6 27B | ● | 85% | ± 4 | $1.19 |
| GPT-5.6 (luna) | 81% | single run | $1.62 | |
| Qwen3.6 35B-A3B | ● | 71% | ± 10 | $0.82 |
| Claude Haiku 4.54 | 68% | ± 6 | $7.82 | |
| Gemma 4 31B | ● | 45% | ± 8 | $0.57 |
| GPT-5.4 nano | 12% | ± 6 | $0.37 | |
| GPT-5.4 mini | 9% | ± 3 | $0.90 |
Notes
1 Standard deviation across seven daily runs from July 3-9. On a 26-task suite, each task is worth about four percentage points. So ±7 means a typical run varies by roughly two tasks from the average. “Single run” means one measurement, not a stable average.
2 Cost of one complete 26-task run through OpenRouter.
3 Clean rerun. An earlier attempt that day failed at boot because of a harness error, so we excluded it.
4 Average across four daily runs from July 3-6, before the model left the daily roster. The cost is also averaged across those four runs.
Focus on the ranges, not the decimals. With only 26 tasks, one result can move the score by about four points. Small differences don't mean much.
Three important findings:
- The strongest open models overlap with the frontier tier. The cheapest model in that range costs $0.34 per run, compared with $20-28 for frontier models.
- Below roughly 70%, the wheels come off entirely.
- Price and capability don't follow the same order. One brand-name mid-tier model costs $7.82 per run but scores below five open models that cost between $0.27 and $1.23.
One outlier is worth calling out. Qwen3.6 35B-A3B scored between 58% and 88% across the week, despite running the same tasks each day. A single-run benchmark would have hidden that instability.
The takeaway for decision-makers: stop asking, “Which premium model should we buy for everyone?” Ask, “What's the cheapest model that can still handle our actual work?”
In our benchmark today, an open-weight model passes roughly 90% of the tasks for well under $1 per run. Frontier models cost $20-30 per run for the remaining few percentage points.
So no, you may not need a premium model for every seat. Our numbers are ours - your tasks will rank models differently. But the benchmark is public, so steal the good bits.
Where this is going
To be clear, this is still early. The suite has 26 tasks - enough to see broad performance ranges, but small enough that one task moves the score by four points. We'll keep adding tasks based on real customer problems.
The benchmark already runs in CI. Next, we want to make it harder to ignore: broader coverage, clear release gates, and better reports when something regresses. The harder problem is orchestration: can we help smaller models get closer to the larger ones without hiding problems behind retries and extra tokens? That needs proper research. We'll write about it when we have results.
Before this, we couldn't answer a basic question: does Chat actually work across different models? Now we have repeatable runs, failures broken down by model and task, and trajectories that lead us back to real bugs. That's what we need to make Chat reliable for someone who doesn't know - or care - which model is underneath.