Hypothesis First: Why Investigation Agents Rank Competing Explanations
An illustrative case. Checkout errors start at 14:02. There was a deploy at 13:55. Everyone in the channel already knows the answer: roll it back.
Sometimes that's right. Sometimes the rollback goes out, the errors keep coming, and twenty minutes later someone notices the database has been at its connection limit since 13:40. The deploy was a coincidence that happened to be on screen.
That failure has a name: anchoring. The first plausible explanation becomes the only one, and every later check gets read as support for it. It's the default way people debug under pressure, and it's the default way a language model will debug too, unless the method stops it.
So our Frontier Investigation Agents work hypothesis first. This post is about that reasoning method: why it exists, and how it shows up in a CloudThinker Resolve investigation.
Why the usual approach breaks
Human debugging is serial for a practical reason. Every check costs a dashboard, a query, a context switch. So you test the most likely cause first, and if it seems to fit, you stop.
The trouble is that "seems to fit" is a low bar. A recent deploy fits almost any incident, because something was deployed recently. Errors went up after the deploy. That's consistent with the deploy causing them. It's also consistent with half a dozen other things.
Runbooks don't help here. A runbook is a diagnosis someone made in advance. It matches the failure you expected. Anchoring is what happens on the failure you didn't.
A model can make it worse. It's fluent, so a plausible story comes out sounding finished. The fix is structural: make the model hold more than one story and force them to compete.
Explanations that predict different things
A hypothesis is only useful if it predicts something the others don't. That's the core move.
01Hypothesis A
The 13:55 deploy
Predicts errors only on new pods, starting at rollout time.
02Hypothesis B
Database connections
Predicts connection waits on every pod, old and new.
03Hypothesis C
Upstream dependency
Predicts errors only on calls to that dependency, from any version.
If the deploy is the cause, the errors should be on new pods and start at rollout. If the database is the cause, old and new pods should both be waiting on connections. If the upstream dependency is the cause, only calls to it should fail, whatever version made them.
Now one check, error rate split by pod version and by call target, can rule out two of the three. The investigation stops asking "does this fit?" and starts asking "which of these does the evidence contradict?"
How it works in an investigation
01Recall
Read past lessons
Incident memory arrives as leads, before any hypothesis is formed.
02Propose
Several explanations
Each one named, with what it predicts in the evidence.
03Test
Look for the miss
Read-only subagents check the predictions in parallel.
04Rule out
Record why
A failed prediction removes a hypothesis and becomes a finding.
05Converge
Verdict or Not found
What survives, with its causal graph, or an honest no.
Recall comes first. At the start of an investigation, CloudThinker reads recent, relevant entries from incident memory before hypothesis work begins. A past lesson goes in as a lead. It's compared against current evidence instead of being treated as proof that history is repeating.
Hypotheses are tested in parallel. In the Analysis phase, CloudThinker starts short-lived subagents, each checking one focused hypothesis against logs, metrics, traces, changes and dependencies. A typical investigation uses two to four, with a maximum of eight. They're read-only, so testing a hypothesis can't change the system that's being tested. The mechanics are in Inside an Investigation.
Parallel matters for more than speed. When hypotheses are checked one at a time, the first one tested gets the most attention and the benefit of the doubt. Run them side by side and each one has to survive the same evidence on its own.
What survives becomes the verdict. CloudThinker combines the subagents' findings into one root cause, with a causal graph from the initiating condition to the visible impact. If nothing survives, the outcome is Not found.
Ruled-out paths are findings
The Hypotheses section of every investigation lists the explanations tested, confirmed and ruled out. We treat the ruled-out list as part of the result.
Three reasons. First, it's how you check the agent's work. A verdict that says "the database" is easier to trust when you can see the deploy was considered and why it was dropped.
Second, it saves the next person. The on-call engineer joining at 03:00 doesn't re-check the deploy because the record already says it was checked, and against what.
Third, it's where wrong conclusions get caught. If a hypothesis was ruled out on thin evidence, you can see that, and re-investigate with what's missing.
01Anchored
First plausible cause
- Starts fromThe most recent change, or the last incident
- Looks forEvidence that agrees
- Stops whenSomething fits
- Leaves behindOne cause, no record of the rest
Fast, until it is wrong
02Hypothesis first
Competing explanations
- Starts fromSeveral explanations held open
- Looks forEvidence that tells them apart
- Stops whenOne survives, or none does
- Leaves behindThe cause and every path ruled out
Shows its work
Design decisions
Many explanations, one verdict. Holding hypotheses open is the method. Handing the engineer five of them is not an answer. The investigation has to converge, and the alternatives stay visible as supporting context.
Memory as a lead, never a diagnosis. The cheapest anchor is the last incident that looked like this one. Recall happens before hypotheses form, so past lessons widen the set of explanations instead of replacing the search. A lesson whose fix failed is kept as counter-evidence, not guidance.
Confidence next to evidence. The agent records a confidence score and the interface groups it as High, Medium or Low. It's context. The evidence and the ruled-out paths are what you review.
"Not found" is allowed. A method that must always produce a cause will invent one. We'd rather return Not found and show what was checked.
Honest limits
- The method only tests what the agent can reach. If the real cause sits in a system with no connection, no hypothesis about it can be confirmed, and the investigation is likely to end in Not found. That's why the harness matters.
- Hypotheses come from the evidence and the model's reasoning. If none of the proposed explanations is the real one, ruling them all out is still progress, but it isn't a root cause.
- Confidence is recorded by the agent about its own conclusion. Treat it as context, not proof.
- Recall is capped and ordered by recent updates, so an old but relevant lesson may not make it into the context.
Try it
Take an incident where the obvious cause turned out to be wrong, and see whether the agent gets anchored the same way:
"Investigate the checkout errors starting at 14:02. Don't assume the 13:55 deploy is the cause. Test it against database connections and upstream dependencies, and show me what each explanation predicts."
"For this incident, list every hypothesis you ruled out and the evidence that ruled it out."