2:47 A.M. in Someone Else's Production: CloudThinker's On-call Team

What a 2:47 a.m. page looks like when Resolve investigates and a human SRE approves the fix from her phone. How CloudThinker's on-call team is wired, the nights the agent is wrong, and the actions it's never allowed to take.

Use caseUpdated Nov 2, 2026

Cover Image for 2:47 A.M. in Someone Else's Production: CloudThinker's On-call Team

2:47 A.M. in Someone Else's Production: CloudThinker's On-call Team

Here's the shape of one page on the night shift (the details are illustrative):

02:47  [P1] payments-api: 5xx rate 0.4% -> 18%, p99 climbing
02:48  Resolve acknowledges. Correlating: API errors vs DB connection wait time
02:51  Investigation posted: connection pool ceiling lowered in yesterday's
       deploy; saturation began at the overnight traffic uptick.
02:53  Proposed fix: restore pool ceiling + rolling restart. Sent for approval.
02:57  Human on-call (CloudThinker SRE) reviews on her phone. Approves.
02:58  Fix applied inside a sandboxed session. Watching.
03:04  5xx back to 0.4%. p99 recovered. Validation: green.
03:06  Jira ticket resolved. Report attached. Customer still asleep.

One approval, given from a phone. No laptop, no war room, no "can someone share the dashboard link?" The customer's engineers read the story over coffee.

CloudThinker runs on-call for customers every night with a team that's part human, part agent. The agents, running on Resolve, do the reading. Our engineers make every call that touches production. The parts worth your time are the nights the agent is wrong and the actions it's never allowed to take, because that's where trust is either real or it isn't.

The team

Here's the pitch you won't hear from us: "AI answers your pager. No humans needed."

We operate production systems every day and we don't believe it. So the rotation is staffed by an Operation Team of humans: SREs, security engineers and DevOps engineers. They aren't a fallback bolted on for the sales deck. They own every shift, and their names are on every approval. Customers get this team through Managed Cloud Service 24/7.

What the platform changes is how they spend the shift:

  • Before: watching dashboards, triaging duplicate alerts, grepping logs at 3 a.m., writing the postmortem at 9.
  • Now: reviewing investigations that arrive with evidence attached, approving or rejecting proposed fixes, and taking the novel failures that deserve a senior engineer's full attention.

Why the classic on-call shift burns people out

If you've carried a pager, you know the other version of the 02:47 page. The alert with no context. Five dashboards. The deploy nobody mentioned in the channel. Forty minutes to build the picture a four-minute fix needed. None of that is about lazy engineers or slow tools. It comes from how on-call is built.

The alert carries a symptom, not a case. A threshold fires on one service. It doesn't know about yesterday's deploy, the sibling alerts firing in the same minute, or that the database behind it is the real story. The person paged assembles that context by hand, at the worst hour of the day.

Context lives in heads. Which deploy lowered the pool size, which pipeline ships CDN config, which customer has a launch this week. When the engineer who knows is asleep, the shift diagnoses by folklore, and last month's fix sits in a Slack thread nobody finds.

Runbooks are static. A runbook says what to do once you know what's wrong. Most of the night goes to working out what's wrong.

Nothing confirms the fix held. The graphs recover, everyone goes back to bed, and the same symptom returns at the next peak as a brand new page with no history.

Where CloudThinker fits

01Managed surfaces

Your stack

  • AWSCloud and infrastructureAccounts, clusters, hosts, networks
  • GrafanaObservabilityMetrics, logs, traces, alerts
  • SIEMSecurity findings and audit trails
  • TicketingTickets and change records

Customer-scoped credentials

02Investigation

CloudThinkerResolve

  • DetectRelated signals in one cluster
  • InvestigateHypotheses checked, reads only
  • RespondA human approves, then a re-check
  • RememberIncident memory, recalled as leads

Operation Team above every step

03Service delivery

Customer SLAs

  • UptimeMonitoring from measured data
  • EscalationA briefed human engineer on the line
  • Request ticketsTicket SLAs in the customer Jira

Three measurable promises

Humans on the Operation Team (SREs, security engineers, DevOps engineers) own every shift and every approval. Resolve does the reading; they make the calls.

The platform reaches each customer's cloud, observability stack, SIEM and ticketing through Connections, using credentials the customer scopes, with nothing installed in their environment. Every page then runs through the same four stages, Resolve's own:

01Signal

Detect

Pulse groups related alerts, findings and tickets into one cluster.

02Evidence

Investigate

Hypotheses checked against metrics, logs and changes, with reads only.

03Human gate

Respond

A named engineer approves the fix, then the alarm telemetry is re-checked.

04Memory

Remember

The case, the reasoning and whether the fix held, kept as a lead.

The order is fixed. Nothing is marked resolved because an agent says so.

Detect. Signals stream in: an alert, a SIEM finding, a customer ticket. Pulse, Resolve's detection layer, drops duplicates and known noise and groups related signals into one cluster instead of a page storm. The investigation starts before any human is interrupted.

Investigate. Hypotheses get checked against metrics, logs and recent changes, and the result carries an evidence label: Verified, Inferred or Cause unproven. Investigation only reads.

Respond. The proposed fix goes to a human with the evidence attached. Anything that changes production waits for someone on the Operation Team. Once it runs, the same telemetry that raised the alarm is re-checked before the case closes, and a failed check reopens it.

Remember. The case, the reasoning and whether the fix held go into incident memory. The next similar page starts with that history as a lead, not as a verdict.

Writes, from a restart to a scaling change to a ticket, go through Auto Mode, so the customer decides what runs on its own and what waits for an engineer. Commands run in CloudThinker's Sandbox.

What the agents are never allowed to do

Some actions never run alone, however much trust a fix pattern has earned:

  • Destroy data. No dropping databases, deleting snapshots or emptying buckets. Deletion proposals are filed as tickets for humans.
  • Change identity and access. IAM policies, security groups and network exposure always need a named human approval.
  • Run schema migrations. Proposed and evidenced, then handed to the customer's change process.
  • Cut over traffic. Cross-region failovers and DNS changes are human decisions with an agent-prepared briefing.
  • Touch anything the customer marks off-limits. Customer boundaries are enforced, not advisory.

We set these limits for every customer we run, and they're enforced on every action, through the access each customer grants and through Auto Mode, rather than kept in a policy document. An agent that structurally can't delete your database is a different proposition from one that promises not to.

One page, end to end

Take the 02:47 page. Three readings fit, and each predicts different data:

  • The deploy broke something in code. Errors start with the first request after the deploy.
  • The database degraded. Query latency is high for every client, not only this service.
  • Traffic outgrew a limit. Errors start only when load crosses a line.

Resolve pulled the pool metrics, the deploy diff and the traffic curve. The third reading held: the deploy cut the pool ceiling, and errors began exactly when overnight traffic rose past it. The fix (restore the ceiling, rolling restart) went to the on-call engineer as a briefing, not an alarm.

A CloudThinker on-call engineer reviews a pending approval on her phone in bed at night. The approval card reads: K8s pod crash loop detected, proposed remediation ready for review. It summarizes what happened (a payment-service pod crashing repeatedly due to a memory leak), the root cause (unclosed DB connections in handleRequest), the proposed fix (add connection pooling and close idle connections), evidence tiles with CPU usage, pod status, log tail, and the code diff, the customer impact, and Reject and Approve buttons.
A CloudThinker on-call engineer reviews a pending approval on her phone in bed at night. The approval card reads: K8s pod crash loop detected, proposed remediation ready for review. It summarizes what happened (a payment-service pod crashing repeatedly due to a memory leak), the root cause (unclosed DB connections in handleRequest), the proposed fix (add connection pooling and close idle connections), evidence tiles with CPU usage, pod status, log tail, and the code diff, the customer impact, and Reject and Approve buttons.

The card in the photo is from a different night on the same rotation, a payment-service pod in a crash loop rather than a pool ceiling, but every approval arrives in this shape: what happened, the root cause, the proposed fix, the evidence and the blast radius. The two buttons are the whole human interface at 2 a.m., and nothing touches production until one of them is pressed.

When the agent is wrong

Every vendor shows you the 02:47 timeline. Here are two nights we could leave out and won't.

The confident wrong answer. An API starts timing out minutes after a redeploy, and the investigation blames the deploy. Plausible, well evidenced, wrong. The on-call engineer notices what the correlation missed: a CDN configuration change that went out through a different pipeline at almost the same minute. She rejects the proposed rollback, redirects the investigation, and the real fix ships twenty minutes later. The rejected proposal is recorded next to the correct one, so the next similar case starts with that second pipeline as a lead.

The fix that didn't hold. A memory remediation passes its check, metrics recover, and everyone moves on until the next traffic peak brings the symptom back. The re-check catches it, reopens the original case with its history instead of opening a fresh one, and escalates to a human with a note that the first fix wasn't enough. The permanent fix goes through the customer's change process the next morning.

That's why the loop is shaped this way. The human gate exists because agents are sometimes confidently wrong. The re-check exists because "it worked when I left" isn't a resolution.

Results

We don't publish rotation-wide numbers here. What customers get is three promises, each reported from measured data:

  • Uptime. Continuous monitoring of the services we manage.
  • Escalation. A staffed on-call path. When something needs a conversation, a human engineer is on the other end, already briefed by the investigation.
  • Request tickets. Work lands in the customer's own Jira with ticket SLAs, inside the tools their team already uses.

Every approval and action is recorded: what was detected, what was proposed, who approved it, what ran and how recovery was checked. Raising a request is usually one message in the shared channel:

"The checkout service felt slow around 9 p.m. last night. Can you check whether anything happened and whether we need to scale before the weekend campaign?"

Why this beats the traditional approach

Traditional on-callCloudThinker's on-call team
First look at a pageA person wakes up and starts opening dashboardsThe investigation is already posted with evidence
Alert stormsOne page per symptomRelated signals grouped into one cluster
DiagnosisFrom memory and grepHypotheses checked against metrics, logs and changes
Changing productionWhoever is awake runs commandsProposed fix, named human approval, sandboxed execution
Did it holdGraphs looked fine when we leftSame telemetry re-checked; a failed check reopens the case
KnowledgeIn the heads of whoever was on callKept in incident memory and recalled as a lead

The deeper difference is where human attention goes. A traditional rotation spends its best engineers on assembling context at 3 a.m. This team spends them on judgment: reading a briefing, saying yes or no, and taking the failures nobody has seen before.

Getting started

Teams meet this way of working in one of two ways. Run the platform yourself: connect your cloud, observability and ticketing, watch every detection and proposal before anything acts, and let individual actions run on their own as they earn it. Or hand us the pager, and get our Operation Team on your platform, your SLAs and your existing chat, Jira and on-call tools.

Either way, the first ask can be a sentence:

"Watch production for me tonight. Tell me everything you see and propose fixes, but don't touch anything without my approval yet."

That "yet" is where the relationship starts. A week later, the second ask is usually narrower:

"For staging only, let rolling restarts on confirmed connection-pool saturation run on their own. Everything in production still comes to me."

Next step: connect one production account and run the first prompt tonight. Resolve explains how each page runs from alert to verified fix.

Steve Tran, CTO, CloudThinker

About the author

ST

Steve Tran

Founder & CTO, CloudThinker

Steve Tran is the founder and CTO of CloudThinker, which builds Frontier Investigation Agents for production cloud operations. He writes about incident investigation, cloud cost, and the practices teams use to safely run autonomous AI agents against production AWS, GCP, and Azure environments.

oncallagenticopssredevopsmanagedcloudslaincidentresponsecloudthinker