Write the incident review for the person with the budget

Most incident reviews are good documents that change nothing. They are honest, they are detailed, they are blameless, and they are read by exactly the people who were already in the incident channel. Six weeks later the same class of failure happens again, and the new review links to the old one under "related incidents," which is the closest most organizations get to admitting the process is decorative.

The failure is not in the writing. It is in the addressee. A review written for responders teaches responders. The thing that would have prevented the recurrence, more often than not, needed a director to move money, headcount, or a roadmap, and the director was never the intended reader. They skimmed it, nodded at "blameless," and went to their next meeting.

Our position is that every review has two readers, and the second one has a budget. Write for both. The first half is a learning artifact and should stay exactly as Allspaw described it. The second half is a decision artifact, and almost nobody writes it.

Allspaw was right, and it still isn't enough

John Allspaw's 2012 piece for Etsy's Code as Craft blog, "Blameless PostMortems and a Just Culture," is the origin of most of what the industry now does after an incident, and it deserves to be taken at its word. Its claim is narrow and correct: if engineers fear punishment for describing what they did and what they were thinking, they will describe less, and the organization will learn less. Blamelessness is not kindness. It is an information-gathering strategy.

We have no quarrel with that. Every failure mode we describe below happens in organizations that have fully adopted the blameless format, and none of them would be fixed by returning to blame.

The gap is what the format was designed to produce. Allspaw's argument is about getting a true and detailed account of the event out of the people who lived it. It is silent on what happens to that account afterward. In a small enough organization that silence is fine: the people writing the review and the people funding the fixes sit close enough that the handoff is implicit. In an organization of two hundred engineers, with a director between the on-call rotation and the budget, the handoff is the whole problem.

The template most teams inherited from that lineage has a section for timeline, a section for contributing factors, and a section for action items. All three are addressed to people who understand the system. None of them are addressed to the person who decides whether the action items get staffed. So the review is finished, the action items go into a backlog with no owner above the team, and the team is told, sincerely, that reliability is a priority.

Learning artifact versus decision artifact

A learning artifact answers: what happened, why did it make sense to the people involved at the time, and what did we not know that we now know. Its reader is a responder, present or future. Its quality is measured by how much surprise it removes from the next incident. Allspaw's format is good at this, and we would not change a line of it.

A decision artifact answers a different question: given what we now know, what should the organization do differently, what does that cost, and who decides. Its reader has authority the responders do not. Its quality is measured by whether a decision gets made, in either direction, by someone who can make it stick.

Most reviews attempt the second job by adding an action items list to the first document. That does not work. Action items written by responders are scoped to what responders can do without asking anyone. "Add an alert on queue depth." "Document the failover runbook." "Add a retry with backoff." Those are real and they should be done. But they are the items that were always in the team's power, which means they were always available and the incident still happened. The item that would retire the class of failure, "stop running the billing pipeline and the reporting pipeline on the same database," costs a quarter and is not on the list. The person writing the list knows they cannot authorize it, so they do not write it down.

The result is a review whose recommendations are precisely calibrated to change nothing above the team. The director reads a list of small, reasonable, already-in-progress items and correctly concludes that no decision is required of them. That conclusion is what the document told them to reach.

Sponsored:

What the second reader needs to see

A director reading an incident review is asking three questions, and the standard format answers none of them.

How exposed are we, in units I can compare to other things I fund? Not "the API was degraded for ninety minutes." Ninety minutes of what, costing what. Take a back-of-the-envelope version: an ordering service that handles a hundred orders a minute at an average of forty dollars, degraded to a thirty percent failure rate for ninety minutes, is roughly 2,700 failed orders and roughly 108,000 dollars of attempts that did not complete, before you count the ones that came back and the ones that did not. Add the responder cost: five engineers for three hours during the incident, and the same five for a further two days of follow-up, is something on the order of ninety engineering hours. Those figures are illustrative and the exact numbers matter less than their existence. A director can weigh ninety engineering hours and a hundred thousand dollars of order attempts against a quarter of platform work. They cannot weigh "degraded for ninety minutes" against anything.

Is this a recurring class, and how often? The single most useful sentence a review can contain for a director is "this is the fourth incident in nine months whose root cause is the shared database, and here are the other three." A one-off is bad luck. A class is a standing cost, and standing costs are what budgets are for. Most review processes make it structurally difficult to write that sentence, because each review is filed alone and nobody is paid to read across them. Someone should be, and it should be a named responsibility, not a hope.

What is the choice on the table, and what does each side cost? The director does not need a recommendation; they need a decision framed so they can make it. "We can separate the pipelines onto their own databases, which is roughly two engineers for a quarter and delays the reporting redesign by the same amount. Or we can accept roughly one incident of this class per quarter at the exposure above, and put the engineers on the redesign as planned." That is a sentence a product lead can argue with. It puts the reliability cost and the roadmap cost in the same currency, and it makes "do nothing" an explicit option with a price instead of the default that happens when nobody chooses.

None of this requires the director to understand the timeline. The timeline is for the first reader. The second reader needs exposure, class, and choice, in that order, in under a page.

Action items as a negotiation, not a veto

Once the choice is framed with costs on both sides, the review stops being an engineering document and becomes the opening position in a negotiation with product. That is the correct outcome and most reliability cultures are built to avoid it.

The veto model is familiar: engineering declares something unsafe, product either accepts the delay or overrides it, and whichever side loses resents the other. Reviews written as learning artifacts feed the veto model, because their action items arrive as obligations, "we have to do this," with no price attached and no alternative offered. Product hears a demand and does what people do with demands they cannot evaluate, which is to push them to next quarter.

Error budgets, as the Google SRE book describes them, were designed to fix exactly this. The service has an availability target; the gap between the target and perfection is a budget; the budget is spent on change. The book's framing is that when the budget is exhausted, feature work pauses until reliability is restored. That is the version most teams copy, and in practice it turns the budget into a scoreboard. Engineering waits for the budget to run out so it can say no; product watches the burn rate to see how much it can get away with. The number becomes a weapon, and weapons do not get negotiated with.

The version we would argue for treats the budget as a table, not a score. The incident review reports how much of the quarter's budget the class of failure consumed, and the choice on the table is stated in those terms: "this class has eaten sixty percent of the budget for the last two quarters; the fix costs two engineers for one quarter; not fixing it means the remaining forty percent is what everything else on the roadmap has to fit inside." Now product has something to trade. They can decide the roadmap item is worth running hot. They can decide to fund the fix and slip the redesign. They can offer a middle path, a partial fix that buys back thirty percent of the budget for a third of the cost. All of those are decisions, made by the people whose work the decision affects, with the numbers in front of them. None of them are a veto.

This only works if engineering gives up something too: the right to declare an action item mandatory without pricing it. An unpriced obligation is a veto in a nicer font.

The review that recommends nothing

Some reviews should end with "no change." This is unfashionable and it is correct.

The pressure after an incident is to produce action items, because a review with none looks like a review that did not try. So teams generate them: add a runbook, add an alert, add a dashboard. Each one is defensible. Together they are a tax on the next incident, paid in alerts nobody trusts and runbooks nobody reads, and they crowd out the one expensive item that would have mattered.

A review is allowed to conclude that the failure was rare, the exposure was small, the fix is expensive, and the organization should accept the risk. Written for the second reader, that conclusion is a decision, not a shrug. "We estimate this class recurs roughly once a year, at an exposure of about ten engineering hours and no customer impact. The fix is a month of one engineer. We recommend accepting the risk and revisiting if the recurrence rate changes." A director can sign that. What they cannot sign is a blank action items section, because a blank section says the team did not think, and "accept the risk" says the team did.

The habit this builds is the important part. A team that can write "accept the risk" with a straight face and a number attached is a team whose "fund the fix" recommendations carry weight, because they are visibly not reflexive.

The second-reader section

Here is what we would add to the review template, described rather than drawn, because a fill-in form is how good sections become boilerplate.

It sits at the top of the document, before the timeline, and it is under a page. It opens with one sentence stating the class of failure in plain language and whether it has happened before, with links to the prior reviews if so. It states the exposure of this occurrence in two figures: the customer or revenue effect, estimated and labeled as an estimate, and the engineering hours consumed, including follow-up. It states, in one sentence, how much of the relevant error budget this class has consumed over the trailing two quarters. It then presents the choice: the fix that would retire the class, its cost in engineers and calendar, and what it displaces; and the alternative, which is to accept the class at its current rate, with the expected exposure per quarter. If the team recommends one side, it says so in one sentence and says why. It ends with a name: the person who is being asked to decide, and the date by which the team needs the decision to plan the next quarter.

It is written by the incident lead with the engineering manager, not by the responders, because the responders should be spending their energy on the learning half. It is reviewed by someone who reads all such sections across the organization, so the "has this happened before" sentence is true. And it is sent to the named director directly, not discovered by them in a wiki.

How to know within a quarter

If the second-reader section is working, four things are observable by the end of the following quarter, and none of them require a survey.

Every review has a named decider, and the decision is recorded next to the review, in either direction, within the stated deadline. Reviews with no recorded decision are the metric that matters, and the target is zero.

Some decisions are "accept the risk." If every choice comes back "fund the fix," the choices are not being priced honestly, or the director is not reading them. Either way the section has become a formality.

The recurrence count for at least one class has moved, because a fix that was funded shipped, and the reviews that follow can say "this class was retired in Q3" instead of linking to the previous four.

And the roadmap has visibly changed shape at least once because of a review. A slipped feature with a written reason pointing at an incident is the clearest evidence the process is doing its job. If the roadmap looks exactly as it would have without the reviews, the reviews are not decision artifacts, whatever they are called.

What to do on Monday

Pull the last five incident reviews for your most important service. For each one, answer three questions without rereading the timeline: what was the exposure in hours and dollars, is it a recurring class, and what decision was put in front of someone with a budget.

If the answer to the third question is "none" for most of them, you do not have a postmortem culture problem. You have a review process with one reader, and the fix is not a better template for that reader. It is a second one.