E-commerce and retail / SRE / on-call engineer / Before / after

3,000 alerts, 900 real, 1 rota

Alert noise for a retail ops team, before and after investigation-first triage.

Published September 1, 2026/01 / 06

The same alert fired forty times last night. Everyone knows it is noise. Nobody mutes it, because the forty-first time might be real. That is the trap: the rota pays attention tax on every false positive to avoid missing the one that matters.

Before: 3,000 alerts a night across the retail estate, maybe 900 tied to anything real, one rota absorbing all of it. After: agents investigate first, correlate duplicates into single incidents, close what self-heals, and page a human with evidence, not with a raw alert.

The proof anchor is Amela on AWS: investigation and remediation from hours to under 30 minutes. Multiply that across 900 real events and the rota gets its nights back.

#AISRE #Ecommerce #Oncall #CloudThinker

02 / 06 The problem

The nightly maths of a retail rota

3,000

alerts across the estate on a normal night. Peaks are worse.

900

of them correspond to something real. Nobody knows which 900 until a person looks.

1

rota absorbing the difference, one page at a time.

Why nobody mutes it

Every threshold was set for a reason someone has forgotten. Muting risks the real one; keeping it costs the team its nights. So it stays.

Alert fatigue is not a tooling gap. It is unpaid triage, done by the sleepiest person in the company.

03 / 06 Why the current answers fail

Two ways to fail: keep doing it by hand, or hand production to an agent on day one

Option A: the status quo

  • One-off scripts

    one per account, none the same

  • Tribal knowledge

    the standard lives in someone’s head

  • Manual follow-up

    tickets chased, dashboards watched

  • Quarterly evidence

    compliant in audit week, unknown after

Resultsafe, does not scale

Option B: AI with full access

  • Trust on a promise

    a demo is not a control

  • Unscoped credentials

    one key for cloud and on-prem alike

  • Recommendations only

    still a human acts, still no evidence

  • No audit trail

    the regulator asks; the chat log answers

Resultscales, nobody signs off

What the SRE teams we work with chose

Neither. Every alert gets investigated by an agent first. Humans see incidents with evidence attached, not raw pages.

Trust is not granted to a system. It is earned, one audited check at a time.

04 / 06 Frontier Investigation Agents

Four steps from read-only to autonomous. The team decides when to take each one.

Step 1

Verify

Read only

Live inventory, drift, waste and health checks. Nothing changes. Every finding explains what was checked, what was found, why it matters.

Step 2

Encode

Standards as Skills

Naming, tagging, guardrails, cost controls and runbooks written once and run the same way in every account. Tribal knowledge becomes a file.

Step 3

The gate

Act with approval

Agents investigate and propose. A person approves where risk, policy or production impact demands it. Every action is written to the audit trail.

Step 4

Auto mode

Autonomous under guardrails

Per environment, the team sets Notify, Act-with-Approval or Autonomous. Low-risk remediations run on their own. Production keeps the gate.

The on-call engineer’s question

“Will it silently close something real?”

It closes nothing silently. Deduplicated and self-healed events are still written to the log with their evidence. You can audit every closure it made at 03:00, at 09:00, with coffee.

Frontier Investigation Agents for your cloud. They scale the team’s judgment, they do not replace it.

05 / 06 Proof

The same night, after

3,000 raw alerts

Dozens

of investigated incidents surfaced to the rota, each with a correlated cause and a drafted next step.

Hours per incident

<30 min

from alert to investigated root cause and applied remediation on AWS.

Days to root-cause

Minutes

Resolve correlates signals across the observability stack and hands the team a structured summary.

The rota did not get thicker skin. It got a filter that shows its work.

06 / 06 One number, and the ask

The only number we ask you to check yourself

Read-only, first 30 days

Zero changes to production

Start the way our customers did. One scoped, read-only key. The engine inventories your estate and reports drift, waste and health. Your team checks the findings against what it already knows. If they hold, you decide the next step.

What you get

1 report

Full inventory, drift and waste findings, each with what was checked and why it matters.

What you risk

1 read key

Scoped and revocable. Every check it runs is in the audit log from day one.

Do not trust the AI. Verify it. Then let it earn the next step.

Sources

The evidence

Amela

Incident investigation and remediation from hours to under 30 minutes.

Read the full evidence

The module

OnCall

The part of CloudThinker this story runs on, in product terms.

Explore OnCall
Read-only, first 30 days

Start a read-only pilot

One scoped, revocable read key. One report on inventory, drift, waste and health. Zero changes to production. Your team judges whether the findings hold.