Model-Agnostic by Design: How We Choose a Model With Benchmarks

A new frontier model ships on a Tuesday. By Wednesday someone has posted its leaderboard row in the team channel, and by Thursday the question is on my desk: should Resolve switch?

The easy answer is to read the top row, check the token price and flip the default. We don't, and our own benchmark data shows why.

Every agent in Frontier Investigation Agents is model-agnostic by design. That only helps if we know which model to put behind it. This post is how we decide.

Why a leaderboard row doesn't pick a model

Four structural reasons, each one visible in our data.

A row measures a model inside someone else's harness. On SREGym, Claude Opus 5 closed 16 of 21 faults inside a general coding agent and 18 of 20 inside Resolve. The tools, the verification pass and the prompt move the number as much as the weights do.

Token price isn't the cost of an incident. You pay for a resolved fault. A model that's cheap per token but fails half the time, or reads twice the context to get there, costs more per fix.

Diagnosis and mitigation are different skills. A model can restore service while naming the wrong component. The scoreboard may forgive that. The postmortem won't.

Benchmarks have their own defects. One fault in our cohort can't be mitigated by anyone, because of the grader. Unless you read the source, it looks like every model's failure.

Hold the harness still, change the model

So we fix everything except the model.

01Fix the harness

One agent, one prompt

Same tools, same verification pass, same approval gates for every model.

02Fix the suite

21 live faults

The SREGym-Lite cohort on a real Kubernetes cluster.

03Grade twice

Diagnosis and fix

An LLM judge grades the write-up, an oracle checks the live cluster.

04Price the outcome

Cost per resolved fault

What we were billed, divided by the faults that model closed.

SREGym is a live benchmark for SRE agents from the University of Illinois and the University of Toronto. It deploys a real application on Kubernetes, injects one fault, and grades in two stages. Diagnosis passes only if an LLM judge agrees the agent named the faulty component and characterised the fault. Mitigation passes only if a problem-specific oracle finds the application working again. End to end means both, on the same run.

We ran the SREGym-Lite cohort: 21 faults across four applications. One of them, duplicate_pvc_mounts_social_network, has a mitigation check that cannot pass, because the oracle reads Endpoints that Kubernetes never creates for the service the benchmark's own observer installs. We void that one stage for every model. Diagnosis counts over 21 faults, mitigation and end to end over 20.

Seven models, one harness

Same contract for every row: one cold run per fault, no memory between faults, noise injection off, the diagnosis judge fixed to Claude Sonnet 4.6 as in the SREGym paper.

Model choice alone moved the resolution rate from 40% to 100%. The harness sets the ceiling. The model decides how close you get to it.

What the table shows that a leaderboard can't

01Price

Per fix, not per token

Astra lists at twice the Opus 5 token price and costs a third as much per resolved fault.

02Tokens

Spend is not effort

The three lowest token counts per run were the top three by resolution.

03Coupling

Named it, fixed it

Astra and Opus 5 never patched a component they could not name.

04Failures

Cluster by fault

Five models missed the webhook fault the same way, by blaming the victim pod.

Cost per resolved fault is the number that matters. Cost per attempt rewards a model that fails fast. Astra lists at twice the Opus 5 token price and still costs about a third as much per fix, because it reads half the context and writes a sixth of the output per run. Sol costs more per resolution than either and resolves five fewer faults than Opus 5.

Token spend doesn't predict accuracy. Opus 5, Sol and Astra used the fewest tokens per run and took the top three spots. Sonnet 5 used 6.22M per run and came sixth. A model that burns context while losing is lost, not working harder.

Coupling is the one I watch most. GPT-5.6 Luna restored service in six runs while its write-up named the wrong component. That isn't a scoring quirk. An engineer reading that postmortem learns the wrong lesson, and the next incident starts from it. Astra and Opus 5 had zero such runs. Every other model did it between two and six times.

Failures cluster by fault, not by model. On the mutating-webhook fault, five models named the same victim pod instead of the webhook injecting the bad limits. The fault type decides how a failure looks. The model decides whether it happens.

Reading the intervals honestly

Twenty faults, one run each, makes for wide intervals. Every interval in the table overlaps its neighbour. Astra separates at 95% confidence from five of the other six models, Opus 5 from two, and neither separates from the other.

So we read the table as "Astra and Opus 5, then a pack". A clean sweep on one run is a result, not a ranking.

From numbers to defaults

01Default

Claude Opus 5 (high)

Second only to Astra, with diagnosis and fix coupled on every fault.

02Candidate

GPT-6 Astra (low)

Leads every stage. Re-run before it moves the default.

03Budget tier

GLM-5.3 (low)

Two thirds of the Opus 5 outcome at one sixth of the price.

When we published the campaign, Opus 5 at high effort was Resolve's default. It led every model except Astra, and its diagnosis and fix stayed coupled on every fault.

Astra at low effort is the candidate to replace it. It leads every stage, costs about a third as much per resolved fault, and reaches a fix almost four times faster. We're re-running it before we move the default, because one run at the top of a table isn't a decision.

GLM-5.3 at low effort stays available as a budget tier, for teams that want the incident loop at a sixth of the Opus 5 price and can accept a longer latency tail.

The campaign also changed our own harness. Our diagnosis prompt told earlier models to treat tracing as out of scope. Astra diagnosed that fault anyway. The others didn't, so we're fixing the prompt.

The same loop on every release

SREGym is one suite. Every release and every new model runs the same fixed suites before it reaches customers: SREGym for Resolve, real production bugs for Review, OWASP Juice Shop for Cyber. Each is graded the same way and compared with the release before. A new model earns its row the same way a harness change does.

What we're not claiming

• These runs are self-run, and they aren't official leaderboard submissions yet.
• One run per fault. The public leaderboard aggregates three.
• Noise injection was off. If the leaderboard rows ran with noise on, theirs was the harder condition.
• The Astra and DeepSeek prices are floors, because 10% and 19% of their tokens weren't priced.
• Sonnet 5 is understated. Several of its runs were lost to failures in our stack, not to the model.

The full method, including the leaderboard comparison and the per-run setup, is in Your On-Call Has Not Met Astra Yet.

What this means for your workspace

You don't have to take our default. On your own key you can pick the model per mode and module, within the models AWS Bedrock or Anthropic serve for BYOK. Not every model in the table above is on that list. Your Model, Your Key covers which ones are and how the controls work.

Try it

Run your own small version. Pick three incidents whose root causes you already know, and run each on two models:

"Investigate the incident from last Tuesday's database failover and tell me the root cause, the evidence, and what you ruled out."

"Show me which component you'd change to fix this, and why that component and not its neighbours."

Then compare against your postmortems. Check whether each model named the right component, and whether the fix it proposed targets that component. That second check is the one leaderboards skip.

Start a trial or book a walkthrough.

Related reading

• Your On-Call Has Not Met Astra Yet: the full SREGym evaluation
• Your Model, Your Key: BYOK and Model Choice
• The Harness: Why a Frontier Model Needs Context From Every Surface
• Frontier Investigation Agents