The Hidden Cost of Incident Time

At 2:14 a.m. the payments system raises a latency alert, and the on-call engineer gets paged. They open the first monitoring tool, then a second, a third and a fourth. Nothing explains the latency on its own, so they pull in a database engineer, then the payments lead.

At 3:40 the three of them find it. An afternoon deploy doubled the worker count, and the workers drained the connection pool. At 3:48 a one-line config change restores service.

Two details make this night expensive. The same failure was analyzed in a postmortem six weeks earlier, and its prevention item never shipped. And the connection pool alert had been firing since 1:00. It was ignored, like the forty times it fired before that month.

This is a composite. I've seen it in many forms, at many companies. This is part 1 of the Agentic IT Operational Excellence series, and it's about where that night's minutes went.

The one-line report

The next morning the CIO got one line: payments degraded, restored after 94 minutes.

What the line left out was most of the cost. Three engineers were up. Four tools were searched by hand. A lesson the company had already paid for was paid for again. None of that is in an incident report, because incident reports record when a ticket opened and when it closed.

That's the hidden part. Incident time rarely shows up as a line in a budget, because it sits in payroll, not on an invoice. It shows up instead as roadmap work that slips, and as engineers who are tired before the week starts.

Forty percent of the week

The numbers are big enough to matter at board level.

In February 2026, NeuBird surveyed 1,039 SRE, DevOps and IT operations professionals. Most said their teams spend 40% or more of their time on incident management instead of building, and that share was consistent from engineers up to executives. Catchpoint's SRE report found the same direction: median toil rose from 20% to 34% in a single year. Both are vendor-run surveys, so read them alongside independent sources, but they agree with what I see.

For scale, Google's original SRE guidance caps operational work at 50%, so engineers always keep at least half their time for engineering. Many organizations are now close to that cap, and most of them don't know it, because nobody adds the hours up.

Three causes behind most incident time

The payments night shows the three causes that, in my experience, take up most incident time.

01Repeat

A known failure

The same class of fault, already analyzed, comes back.

02Noise

Ignored alerts

An alert that fires often without action hides the real signal.

03Change

Visible risk

A risky change that could have been caught before deploy.

A known failure repeats. The fault was already understood. The knowledge existed, in a postmortem and in someone's head, and it didn't reach the person who needed it at 2:14.

Noisy alerts hide real signal. The connection pool alert fired an hour early. But an alert that fires forty times a month without leading to action has lost its value as an alert. People learn to ignore it, and they're right to, until the one time they're wrong.

A risky change was visible before deploy. Doubling the worker count against a fixed connection pool is the kind of change a reviewer can catch, if the reviewer remembers the last incident.

The third cause is likely to grow. The DORA 2025 report found that AI helps teams ship faster, and that it's also associated with a higher change failure rate. More changes, shipped faster, means more chances for a risky one to get through.

Three clocks in every incident

Most organizations report one number for an incident. That number hides where the time goes, so split it into three clocks:

• Time to detect (MTTD): from the moment the incident starts to the moment an alert fires.
• Time to investigate (MTTI): from the alert to a known root cause and a fix to apply.
• Time to resolve (MTTR): from having the fix to service restored and verified.

Here's the payments night on that timeline.

012:09

Incident starts

The connection pool runs dry. No alert yet.

022:14

Alert fires

Five minutes of detection. The report clock starts here.

033:40

Root cause found

Four tools and three engineers later: 86 minutes of investigation.

043:48

Service restored

One line of config. Eight minutes to fix and verify.

The incident started at 2:09, five minutes before the alert. The report said 94 minutes because it only counted from the alert. End to end it was 99.

Where the 99 minutes went

Split into the three clocks, the night looks like this. These numbers are illustrative, from the composite incident, but the shape is the one I see most often.

01MTTD

5 min

Detect: from incident start to alert. Monitoring is already good at this.

02MTTI

86 min

Investigate: from alert to root cause and a fix. The clock nobody reports.

03MTTR

8 min

Resolve: from fix to service restored and verified.

Detection was fast. With modern monitoring, detection delay is measured in minutes. It depends on metric collection intervals and alert conditions. Here the alert fired five minutes after the incident started, and the earliest signal arrived more than an hour before that. The problem with detection is mostly signal quality, not speed.

Repair was short. Once the cause is clear, the fix is usually a rollback or a config change. The real risk in this clock is whether the change is safe, and whether the prevention item ever gets done.

Investigation was most of the night. It took 86 of the 99 minutes. That's the clock where the money went, and it's the one almost nobody measures on its own.

Why investigation is the clock to measure

Investigation is a chain of sequential steps:

1. Gather context from several tools.
2. Line it up against recent changes.
3. Build hypotheses and rule them out.
4. Bring in more specialists when the evidence crosses into their area.
5. Find the knowledge the company already has.

Each step waits for the one before it. Each engineer you add raises the cost of the incident, but rarely shortens it by the same amount. Three engineers didn't make the payments night three times faster. They made it a lot more expensive.

It's also the least measured stage. Most ITSM systems record when an incident opened and when it closed. Detection time comes from monitoring. Repair time is visible in the deploy log. The long middle, from "we know something's wrong" to "we know why", has no timestamp at all.

That's why I think every effort to cut incident time should start by measuring MTTI on its own. You can't shorten a clock you can't see, and if you only watch the total, you'll spend on detection and remediation tooling that improves the two short clocks and leaves the long one alone.

The step that hurt most on the payments night was the last one: finding knowledge the company already had. Part 2 is about why that step is slow even in strong teams, and why the cause is structural, not a skills problem. Why root cause analysis takes hours covers the same ground from the engineer's chair.

The same night with agents

Now replay the night with agents doing the investigation. The numbers below are illustrative too.

Before deploy. The worker count change would have been flagged at review, because it matches the postmortem from six weeks earlier. That's the job of Review: checking a change against what the company already learned.

At 1:00. If the change shipped anyway, the early connection pool alert gets investigated instead of ignored, and it's linked to the afternoon deploy.

At 2:14. The agent matches the evidence to the known failure and proposes restoring the worker count. The on-call engineer checks the evidence and approves. Service is back in about nine minutes from the alert, with one engineer awake.

Almost all of the time saved comes from investigation. Detection stays at five minutes. Repair gets a little shorter. Investigation drops from 86 minutes to 5.

That's how we designed Resolve, around the three clocks:

• Detect. Your monitoring stays in place. Agents add self-managed detectors for what it misses, like the early warnings nobody trusts and the silent failures that never fire an alert.
• Investigate. Frontier Investigation Agents gather evidence across logs, metrics, traces and deploys, and match it against company memory, so the root cause arrives in minutes. Inside an investigation walks through one step by step.
• Resolve. The agent proposes the fix, and Auto Mode decides whether it runs on its own or waits for approval. The postmortem and the follow-up are drafted for you.

The loop is the familiar one: Detect, Analyze, Resolve, Validate. The difference is who does the analysis, and how much of what was learned last time is in front of them.

Start measuring MTTI this month

You don't need any new product to see your own version of this. You need three timestamps instead of two.

• Add a "root cause identified" timestamp to your incident records, next to opened and closed.
• Report MTTD, MTTI and MTTR separately in the monthly ops report, not as one total.
• Track the share of engineering time spent on incidents. Add up the hours for every engineer pulled in, not only the on-call.
• Tag repeats. Mark every incident whose root cause has appeared before, and count them.

After a month you'll know how big your investigation clock is and how much of it is spent rediscovering things you already knew. In my experience, that second number is the one that changes budgets.

Giving engineers their time back starts with seeing where it goes.

Related reading

• After ten years of transformation, the next chapter is operational excellence
• Why root cause analysis takes hours
• Inside an investigation
• Incident memory that can be wrong
• CloudThinker Resolve

Sources

• NeuBird AI, 2026 State of Production Reliability and AI Adoption Report, survey of 1,039 professionals, February 2026 (vendor-run survey).
• Catchpoint, The SRE Report 2026 (vendor-run survey).
• Google, Site Reliability Engineering, chapter 5, "Eliminating Toil". sre.google
• Google Cloud DORA, 2025 State of AI-assisted Software Development.

The incident in this post is a composite, and its numbers are illustrative.