What a 99.999% availability target actually asks of you
“Operational excellence” is one of those phrases that has been said so many times in so many all-hands meetings that it has stopped meaning anything. It sounds like a virtue, something you achieve by caring a lot and trying hard. It is not. Operational excellence is a set of concrete engineering practices built on top of a few numbers, and the numbers are unforgiving. This post is about those numbers and what they actually demand, written for someone who has heard “five nines” thrown around and wants to know what it really costs.
I work on infrastructure with high availability targets, and the thing I most wish people understood is this: you do not reach a reliability bar by being careful. You reach it by designing for it, measuring against it, and spending a budget you have agreed to in advance. Caring is necessary and nowhere near sufficient.
Start with the math, because it is brutal
Availability is usually quoted in “nines”: 99.9%, 99.99%, 99.999%. Each extra nine sounds like a small step. It is not. Here is what each one allows as total downtime, per year and per day:
| Availability | Downtime per year | Downtime per day |
|---|---|---|
| 99% (two nines) | ~3.65 days | ~14.4 minutes |
| 99.9% (three nines) | ~8.77 hours | ~1.44 minutes |
| 99.99% (four nines) | ~52.6 minutes | ~8.6 seconds |
| 99.999% (five nines) | ~5.26 minutes | ~0.86 seconds |
Read the five-nines row again. Five minutes and sixteen seconds of downtime for the entire year. That is less time than it takes a human to notice an alert, open a laptop, and read a dashboard. Which means the most important consequence of the number is immediate: at four or five nines, a human cannot be in the recovery loop. If your plan for an outage is “someone gets paged and fixes it,” your floor is roughly three nines, because human response time alone burns your entire four-nines budget on a single incident.
So the first thing the target asks of you is to design systems that detect and recover from common failures automatically, and to reserve humans for the genuinely novel. Everything else follows from that.
The three letters people confuse: SLI, SLO, SLA
You cannot manage what you do not define, and most reliability conversations go sideways because three related terms get used interchangeably. They are different things and the difference matters.
- An SLI (Service Level Indicator) is a measurement: the actual number you observe. “Over the last 30 days, 99.97% of requests succeeded in under 200 milliseconds.” It is a fact about reality.
- An SLO (Service Level Objective) is a target: the number you are aiming for internally. “We want 99.95% of requests to succeed in under 200 milliseconds.” It is a goal you hold yourself to.
- An SLA (Service Level Agreement) is a contract: a promise to a customer, usually with money or credits attached if you miss it. “If availability drops below 99.9% this month, you get a 10% credit.” It is a legal and financial commitment.
The relationship between them is deliberate. Your SLO should be stricter than your SLA, so that you start responding well before you breach a contractual promise. And your SLI is how you know where you stand against both. A team that says “we have an SLA” but cannot tell you their current SLI is flying blind: they have made a promise they have no way of knowing whether they are keeping.
Choosing the SLI is itself an act of engineering judgment. “Availability” is meaningless until you define what counts as a successful request, measured from where, over what window. The same service can be “99.99% available” or “99.9% available” depending entirely on those choices, and the honest version measures from as close to the user as possible, because that is what the user actually experiences.
Error budgets turn reliability into a decision tool
Here is the idea that makes the whole thing operable rather than aspirational. If your SLO is 99.9%, then you are explicitly allowing 0.1% unreliability. That 0.1% is not a failure. It is a budget.
An error budget reframes reliability from a moral question into an accounting one. You have, say, 43 minutes of downtime to spend this month. Spend it however creates the most value: shipping features faster and accepting more risk, running a risky migration, doing a load test in production. The budget is permission to take risk, bounded by a number everyone agreed to.
This solves the oldest fight in software, the one between the people who want to ship and the people who want stability. Without an error budget, that argument is a values clash with no resolution, decided by whoever has more political capital. With one, it is arithmetic. Budget remaining? Ship, take the risk. Budget exhausted? Reliability work becomes the priority until the budget recovers, automatically, no debate. The number, agreed in advance, is what makes that conversation end instead of recur.
The corollary catches people off guard: being too reliable is also a problem. If you set a 99.9% SLO and consistently deliver 99.999%, you are not a hero, you are leaving velocity on the table. You spent enormous effort buying reliability nobody asked for, effort that could have gone into the product. The goal is to hit the target, not to maximize the metric.
What the target forces into the architecture
Once “recover from common failures without a human, within seconds” is the design center, a set of decisions stops being optional. None of these are about trying harder. They are structural.
Redundancy with no shared fate. If one machine, rack, zone, or region failing can take you down, your ceiling is the reliability of that single thing. Real availability targets force redundancy across independent failure domains, with traffic able to move away from a sick one automatically. The hard part is the “independent”: two replicas that share a power supply, a network path, or a dependency are not redundant, they just look redundant on the diagram.
Health checks and automated failover. The system has to know it is sick and route around the sick part faster than a human could. That means real health signals (not just “the process is running” but “the process is actually serving correct responses”) wired to automated traffic management.
Graceful degradation. When a dependency fails, the question is whether the whole product dies or just one feature. A system designed for a high availability bar fails partially: it serves stale-but-valid data, disables one feature, or returns a sensible default, rather than collapsing entirely. The user notices a smaller thing.
Blast-radius control. Changes and failures both need containment. This is why mature platforms roll changes out gradually and isolate tenants and regions, so that a bad change or a bad day affects a slice, not the whole. A single global switch that can take everything down at once is incompatible with the budget.
Observability that leads the incident. You cannot recover in seconds from something you learn about in minutes. Telemetry has to be fast and trustworthy enough to drive automated decisions, which makes monitoring a first-class part of the system rather than something bolted on after. Honest telemetry is itself a serious data-infrastructure problem, not a free side effect.
The human practices that hold it together
Automation handles the common cases, but the novel ones still reach people, and operational excellence is as much about how a team operates as how the system is built.
On-call that is sustainable. A rotation where the pager goes off constantly is not a sign of diligence, it is a sign of an unreliable system burning out its engineers. High alert volume is a bug. Every page should be actionable and genuinely urgent; everything else should be a ticket or, better, an automated fix.
Runbooks for the predictable. When something known goes wrong at 3am, the responder should not be reverse-engineering the system from first principles. The recovery steps for known failure modes should be written down, current, and ideally executable. The best runbook is one that has been automated out of existence.
Blameless postmortems. After an incident, the question that matters is “what about the system let this happen,” not “who typed the wrong command.” People make mistakes; that is a constant you design around, not an variable you can eliminate by assigning blame. Blameless review is not about being nice, it is about getting honest information, because in a culture that punishes the messenger, the messenger goes quiet and you lose the data you need to actually improve.
Treating reliability as a feature. This is the cultural keystone. Reliability work competes with feature work for the same engineering time, and if it always loses, the number drifts down until an outage forces a reckoning. Operationally excellent teams budget for reliability the way they budget for features, deliberately and on purpose, rather than paying for it in emergencies.
Why AI infrastructure inherits the strictest version of this
It would be easy to think of AI systems as more forgiving than transactional ones. “It is just inference, the user can retry.” That intuition is wrong for any AI feature wired into a real workflow, which is most of them now. When the model behind an assistant stops responding, every workflow that embeds it stops too, so the blast radius is the union of everything that calls it. Many of these features sit inside a synchronous interaction where a thirty-second blip reads as “the product is broken,” not “the model was slow.”
So AI platforms do not get an easier reliability bar, they get a harder one, the same five-nines-class expectations as the transactional systems they are increasingly embedded in. And this connects back to the thesis running through most of what I write: the data infrastructure is the bottleneck for enterprise AI, not the model. The model gets the headlines. Whether the platform around it stays up, recovers itself, and stays inside its budget is what decides if any of that capability is actually usable in production.
The one question worth asking
If you want to know how operationally mature a system really is, skip the slideware about excellence and ask one thing: what is your SLO, what is your current SLI against it, and how much error budget is left this month?
A team that can answer crisply has done the real work. The number exists, it is measured, and it drives decisions. A team that cannot is relying on care and effort, and care and effort are exactly what run out at 3am during the incident nobody designed for. Operational excellence is not how hard you try. It is whether the system is built to hold the number when no one is trying at all.