Inside an Investigation: Hypotheses, Read-Only Subagents and a Verdict
An illustrative case. Error rates on a payments service climb at 14:10. Pulse folds a dozen alerts into one Cluster and routes it into an Incident. On Duty is on, so nobody clicks anything. Root-cause analysis starts on its own.
Twenty minutes later the on-call engineer opens the Incident. There's a root cause, a graph of how it happened, the explanations that were tested and dropped, and a proposed fix waiting for approval.
This post walks through what happened in between. It's the mechanics of one investigation in CloudThinker Resolve, the core of what we call Frontier Investigation Agents.
What usually happens instead
A human investigation is serial. You check the deploy, then the database, then the node, and each check costs a context switch across tools. The first plausible answer tends to win because you're tired and the next check is expensive.
The output is also thin. The postmortem records the cause you settled on. The three things you ruled out, and why, live in your head and leave with you.
We wanted an investigation that tests several explanations at once, can't damage what it's looking at, and writes down everything it checked.
How an investigation starts
There are two ways in. An eligible Incident that Pulse created can start root-cause analysis (RCA) automatically when On Duty is on. An Incident you log by hand waits until you click Investigate.
When several alerts share a likely cause, Resolve groups them under one parent incident and runs RCA there, so the correlated alerts are analyzed together. If one of them turns out to be its own problem, Promote to Standalone detaches it into its own incident with its own lifecycle.
Three phases
01Context
Read the case
The Incident, its Pulse signals, available topology and connected operational data.
02Analysis
Test hypotheses
Parallel, read-only subagents check logs, metrics, traces, changes and dependencies.
03Resolution
Record the verdict
A supported root cause or another outcome, with evidence and recommended actions.
Context. CloudThinker reads the Incident, the Pulse signals behind it, the topology it can see, and connected operational data. Before any hypothesis work begins, it also reads recent, relevant entries from incident memory, lessons written during past investigations.
Analysis. CloudThinker forms competing explanations and starts parallel subagents to test them against logs, metrics, traces, changes and dependencies. Each subagent is a short-lived helper that checks one focused hypothesis and reports back.
Resolution. CloudThinker records the supported root cause, or another outcome, and organizes the evidence and recommended actions.
Subagents: chosen per case, and read-only
The set of subagents isn't a fixed roster. CloudThinker picks the domains the evidence calls for. An illustrative split for the payments case: one subagent on the recent rollout, one on database connections, one on the upstream card processor.
A typical investigation uses two to four subagents, with a maximum of eight.
During an investigation, subagents are read-only. That's the rule we care about most. An agent that can restart a pod while it's still deciding why the pod failed can erase the evidence it needs. Reading and changing are separate steps with separate rules.
Reading the verdict
The investigation view keeps the conclusion next to its proof:
- Root cause: the confirmed explanation, when it was identified, and the agent-recorded confidence.
- How it happened: a causal graph from the initiating condition to the visible impact.
- Hypotheses: every explanation tested, confirmed and ruled out.
- Evidence: the logs, metrics, traces, changes, alerts and configuration observations behind it.
- Timeline: the ordered investigation steps and important operational events.
- Affected services: the services and dependencies involved.
- Remediation: recommended next actions, including any runbook that needs approval.
The causal graph matters more than it looks. "Database connections exhausted" is a symptom. The graph has to connect it back to what started it, or the fix will treat the wrong node.
Confidence, in three bands
01High
0.8 or above
Still read the evidence before you approve a change.
02Medium
0.5 to below 0.8
Check the ruled-out paths as closely as the confirmed one.
03Low
Below 0.5
Treat the verdict as a lead, and look at what evidence is missing.
The score is something the agent records about its own conclusion. We show it, grouped into bands, because hiding it would be worse. But it sits next to the evidence, never instead of it. Read what the subagents found before you approve anything.
How an investigation ends
01Identified
Root cause identified
The evidence supports a root cause, and remediation can proceed.
02Not real
False alarm
The reported condition was not an incident.
03Unsupported
Not found
The available evidence did not support a root cause.
04Human verified
Resolved
Remediation was verified with a person in the loop.
05Agent verified
Auto-resolved
The agent verified the remediation with nobody in the loop.
Not found is a real outcome, and we'd rather return it than a guess. A confident wrong answer sends an on-call engineer down the wrong path at the worst time.
Two states look like endings but aren't. On hold pauses the investigation when CloudThinker needs information, access or a decision, and the Incident moves to Needs your decision. Acknowledged means a responder owns the Incident but analysis hasn't started.
The Investigation queue groups everything by what needs attention: AI is handling, Needs your decision, Not started, Resolved and Dismissed.
From verdict to fix
A proposed action follows the effect set on its runbook command: Allow, Require Approval or Deny. A command with no matching effect defaults to Require Approval. Everything else runs within Auto Mode.
Each remediation suggestion carries its own decision. Mark as Done records what your team did. Dismiss records why it doesn't apply. A settled suggestion keeps its state and note, and a later analysis session never overwrites it.
When the case changes
New evidence arrives after the verdict. Maybe a second region starts failing. You can re-investigate, and Resolve settings decide how readily it does so on its own: Conservative, Balanced or Sensitive.
Every session updates the same investigation record. The timeline shows what each session added. If the conclusion changes, it's recorded as a root-cause revision, not a second report. One incident, one record, with its history intact.
Honest limits
- The agent can only test hypotheses against what it can reach. A missing connection is a blind spot, and an investigation with blind spots is more likely to end in Not found.
- Confidence is self-reported by the agent. It's useful context and nothing more.
- Incidents with a Not found result don't stay in the visible queue. If you want to learn from them, look for them deliberately.
- Not every investigation reaches a fix. A runbook may not match, and that's covered in the loop every module runs.
Try it
Log an incident you already understand and compare the hypotheses list with what you remember checking. Or ask in chat:
"Investigate the spike in 5xx errors on the payments service since 14:00. Test the recent deploy, database connections and upstream dependencies, and tell me what you ruled out."
"Re-investigate this incident with the new errors from the second region and show what changed in the root cause."