Your On-Call Has Not Met Astra Yet

We put seven AI models behind Resolve, the CloudThinker incident agent, and let each one work the same 21 live Kubernetes faults. The agent was the same, the sandbox was the same, and the prompt was the same. Only the model changed.

Five results came out of it.

• With GPT-6 Astra at low effort, Resolve diagnosed 21 of 21 faults and fixed 20 of 20. One of the 21 faults has a mitigation check that no model can pass, so we void that stage for every model and explain why below. On the public leaderboard for the same 21 faults, the best row today closes 17 of 21.
• With Claude Opus 5, Resolve closed 18 of 20. The leaderboard row that runs the same model inside a coding agent closes 16 of 21. Same model, more incidents closed.
• Astra and Opus 5 were the only models that never patched a component they could not name. Every other model did that between two and six times.
• Model choice moved the resolution rate from 40% to 100% inside one agent. The harness sets the ceiling. The model decides how close you get to it.
• Astra did it for $3.53 per resolved fault, about a third of Opus 5 at $9.53. That price is a floor, because 10% of Astra's tokens were not priced, and we say so below.

Everything below is self-run. We say where our setup matches the public leaderboard and where it does not.

The benchmark

SREGym is a live benchmark for SRE agents from the University of Illinois and the University of Toronto. It deploys a real application on a Kubernetes cluster, injects one fault, and grades the agent in two stages.

• Diagnosis. An LLM judge reads the agent's write-up against a checklist and passes it only if the agent named the faulty component and characterised the fault correctly.
• Mitigation. A problem-specific oracle checks the live cluster. It passes only if the application actually works again.

End to end is the intersection. Both stages must pass on the same run. An agent can name the cause and fail to fix it, or fix it while naming the wrong thing. Both happen, and both show up in the tables below.

We ran the SREGym-Lite cohort: 21 faults across four applications, the same 21 problem ids the public leaderboard uses.

One fault no model can close

One of the 21, duplicate_pvc_mounts_social_network, has a mitigation check that cannot pass. We read the benchmark source to find out why. The benchmark's own observer replaces the tracing service with an ExternalName service, and Kubernetes never creates Endpoints for that kind of service. The mitigation oracle then reads the Endpoints of the tracing service and fails with a 404 before it looks at the cluster. It does this whatever the agent did.

No model has ever passed this mitigation in our ledger, across 22 runs and eight models. Astra diagnosed it correctly twice, fixed the workload to two ready replicas, and still scored a fail. We have not yet filed the defect upstream.

So we void that one stage. Diagnosis on that fault still counts, because the judge can grade it. Mitigation and end to end do not, because the oracle cannot. Every CloudThinker number below uses 21 faults for diagnosis and 20 for mitigation and end to end. The leaderboard rows keep their published denominator of 21, because we do not have their per-fault results. If their agents hit the same wall, their rates are understated by up to one fault each.

Resolve against the public leaderboard

Model names are shown without the serving provider. Effort level is in parentheses where the source publishes it.

How to read the CloudThinker rows. They are self-run and they are not official leaderboard submissions. We ran the SREGym-Lite-0904 cohort (21 faults, SREGym commit f125857) on our own infrastructure with our benchmark kit 1.7.0, Opus 5 on 6 September 2026 and Astra on 9 September 2026. One cold run per fault, no memory carried between faults, noise injection off, diagnosis judged by Claude Sonnet 4.6 as in the SREGym paper. Mitigation and end to end are over the 20 gradable faults. Over all 21, the raw figures are Astra 100, 95.2, 95.2 and Opus 5 95.2, 85.7, 85.7. The leaderboard rows aggregate three runs per fault, and the leaderboard does not publish its judge model or its noise setting. Leaderboard values were fetched on 8 September 2026.

Three things stand out.

Astra clears the board. It passed every diagnosis and every gradable mitigation. That is one run of 20 faults, so the 95% interval on its rate runs from 84 to 100 and still overlaps the best public row at 81. It is a clean sweep, not a settled ranking.

Same model, more incidents closed. Resolve with Opus 5 closes 90%. The leaderboard's Claude Code row with Opus 5 closes 76%. The difference is coupling: when Resolve named the resource correctly, it fixed that resource. It had zero runs in the "fixed it, named it wrong" column. Codex with Sol passed 18 diagnoses and 18 mitigations but only 17 on the same fault.

Speed is the model's, not the harness's. With Opus 5, Resolve's time to mitigation is three times the leaderboard's. With Astra, on the same harness, the same verification pass and the same approval gates, it is 335 seconds, inside the leaderboard's range. Part of the Opus 5 gap is measurement: our TTD includes the judge's grading time, and our TTM includes the TTD. The rest is the model's pace. We would rather report a slow verified fix than a fast unverified one, but the number is the number.

Seven models, one harness

Now the same 21 faults, all seven models, all inside Resolve. This is the table we actually use to decide what to ship.

Same contract for every row: one cold run per fault, noise off, prompt mode fixed, judge Claude Sonnet 4.6. Diagnosis is over 21 faults, mitigation and end to end over the 20 gradable ones. Cost is what we were billed, divided by the faults that model resolved. Astra, Opus 5, Sol, and Sonnet 5 ran on three parallel lanes; the other three ran on five, so compare times only within the same group. The Astra and DeepSeek prices are floors because 10% and 19% of their tokens were not priced. The Luna price is recomputed from token counts at list rates because our tracing stopped pricing that model mid-campaign.

Every confidence interval overlaps its neighbour. With 20 faults and one run each, Astra separates at 95% confidence from five of the six other models, and Opus 5 separates from two. Neither separates from the other. Read the table as "Astra and Opus 5, then a pack", not as a ranking.

What the second table says that the first cannot

Cost per resolved fault is the number that matters. Cost per fault attempted rewards a model that fails fast. Astra costs $3.53 per resolved fault, about a third of Opus 5 at $9.53, and it resolves two more. Astra lists at twice the Opus 5 token price and still comes out cheaper, because it reads half the context and writes a sixth of the output per run. Sol costs more per resolution than either and resolves five fewer faults than Opus 5, so it is both worse and dearer. GLM-5.3 reaches two thirds of the Opus 5 outcome at one sixth of the price, with a fatter latency tail.

Token spend does not predict accuracy. Opus 5 at 1.93M tokens per run, Sol at 1.75M, and Astra at 2.13M are the three lowest by tokens and the top three by resolution. Sonnet 5 at 6.22M comes sixth. Sonnet 5 and DeepSeek also spend more tokens on the runs they fail than on the runs they resolve. A model that burns context while losing is lost, not working harder.

Diagnosis and mitigation are separate skills. Luna ties for third on mitigation and sits fourth on diagnosis. It fixes things it describes wrongly. Six of its runs restored the service while the write-up named the wrong component. That is an operational risk, not a scoring artifact: a human reading that postmortem learns the wrong lesson. Astra and Opus 5 had zero such runs. The only points Astra dropped anywhere were on scope: twice it named a slightly wider blast radius than the ground truth, and the judge still passed both.

Failures cluster by fault, not by model. When five models missed the mutating-webhook fault, all five named the same victim pod instead of the webhook that was injecting the bad limits. Astra named the webhook. Four faults that broke every earlier model, stale rotated secrets, a retry collapse under search load, an unschedulable port assignment, and a misconfigured rolling update, all passed on Astra's first run. The fault type decides how the failure looks. The model decides whether it fails.

What we changed because of this

Opus 5 at high effort is the default model for Resolve today. It leads every model except Astra, and its diagnosis and fix stayed coupled on every fault.

GPT-6 Astra at low effort is the candidate to replace it. It leads every stage, it costs about a third as much per resolved fault, and it reaches a fix almost four times faster. One run at the top of the table is a result, not a decision. We are re-running Astra before we move the default.

We are keeping GLM-5.3 at low effort available as a budget tier for teams that want the incident loop at a sixth of the Opus 5 price and can accept a longer tail.

We are filing the broken mitigation check upstream, and we are fixing our own diagnosis prompt, which told earlier models to treat tracing as out of scope. Astra diagnosed that fault anyway. The others did not.

What we are not claiming

• We are not on the official leaderboard. Acceptance requires full agent transcripts handed to the maintainers. We keep those transcripts and we are talking to the maintainers, but until a row is accepted the comparison above is ours.
• Our mitigation and end-to-end denominators are 20, not 21. We void one fault whose check cannot pass. Over all 21, Astra closes 95.2% and Opus 5 closes 85.7%, and the leaderboard rows are not corrected the same way.
• One run per fault. The leaderboard aggregates three. A clean sweep on one run carries a wide interval, and the interval overlaps the best public row.
• Our judge is the paper's judge, Claude Sonnet 4.6, but the leaderboard does not say what judged its rows. The Astra run reached that judge through a different provider route than the other six rows. Same judge model, two routes.
• Noise injection was off. If the leaderboard rows ran with noise on, they carried a harder condition than ours.
• Sonnet 5 is understated. Several of its runs were lost to failures in our stack, not to the model. Its true rate is higher by an amount we did not measure.

The full internal report, the per-fault grid for all seven models, and the ledger behind every number in this post are available on request.