On-call and blameless postmortems for small teams

In a team of three or four people, on-call tends to set itself up: the alert goes to whoever installed the system, at any hour, and nobody writes down what happened. By the third month that person is tired, alerts get ignored and the same incident happens again.

The most widely used references on on-call and incidents are Google's SRE books (Site Reliability Engineering and The Site Reliability Workbook), PagerDuty's incident response documentation, published under the Apache 2.0 license, and Atlassian's incident handbook. They are written by large companies, but the Workbook's on-call chapter states that much of what they learned applies to organizations of any size. This guide takes what works with few people and closes with templates and a checklist.

1. Decide what on-call covers

Before building the calendar, define which services on-call covers, during which hours and with what response time. Google's numbers are a useful reference. In the Being On-Call chapter of the SRE book, Andrea Spadaccini explains that an SRE should spend no more than 25% of their time on-call. From that comes the minimum for a 24/7 rotation: eight people if the team is on a single site, and six per site if there are two. The Workbook's on-call chapter (Ollie Cook, Sara Smollett, Andrea Spadaccini, Cara Donnelly, Jian Ma and Garrett Plasky) starts from a minimum of five people per site when a 24/7 rotation spans several sites and eight on a single site, and recommends adding one more per site as a buffer against departures: six per site, or nine on a single site.

Three people cannot reach those numbers. The honest move is to narrow the scope:

• Response time by impact. The book gives 5 minutes as a typical value for user-facing services and 30 minutes for less time-sensitive systems. The Workbook adds that a minor problem, such as a failing backup, can be handled within a few hours. Only urgent issues justify waking someone up.
• Short shifts. The Workbook recommends shifts of 12 hours at most and considers 24 hours of on-call without relief unsustainable. Another option it proposes is shifts shorter than a week, for example 3 days on and 4 days off.
• Primary and backup. Many teams have a primary and a secondary on-call. PagerDuty, in its Going On-Call guide, suggests that the second escalation level be the person who was on-call the previous week, because they still have the context.
• Measurable load. According to the book, handling an incident takes 6 hours on average across analysis, remediation and postmortem; hence the maximum of 2 incidents per 12-hour shift. The Workbook tracks load with a 21-day trailing average and calls for corrective action when it rises.
• Compensation. Google compensates on-call with time off or cash, capped at a proportion of salary.
• Preparation. PagerDuty asks for company equipment, reliable home internet and verified access before joining the rotation. In one of the Workbook's cases, newcomers shadow someone experienced before taking shifts.

If your team is in Chile

Chile's Labor Code (text on LeyChile) has articles worth reading before you define on-call. Article 21 counts as working time the time a person is at the employer's disposal without performing work, for reasons not attributable to them. Article 30 defines overtime as hours that exceed the legal maximum or the agreed one. Article 31 allows agreeing up to two overtime hours per day in work that does not harm health. Article 32 says overtime may only be agreed for temporary needs, in writing and for a maximum of three renewable months; that even without a written agreement, hours worked beyond the agreed schedule with the employer's knowledge count as overtime; and that overtime is paid with a 50% premium. Since Law 21.561, the same article allows agreeing in writing to compensate overtime with additional vacation days: up to five business days a year, at one and a half hours of vacation for each overtime hour. Article 25 ter expressly regulates standby or call shifts, but only for railway crews. This is not legal advice: work out with your labor advisor how on-call is agreed, recorded and paid.

2. Which alerts wake someone up

In the Monitoring Distributed Systems chapter, Rob Ewaschuk separates two questions: what is broken (the symptom) and why (the cause). For him, every page must be actionable, must be about a novel problem and, if the response is mechanical, should not be a page. His test for each rule: does it detect an otherwise undetected condition that is urgent, actionable and actively or imminently user-visible?

Benjamin Treynor Sloss, in the book's introduction, separates three monitoring outputs: alerts (someone must act now), tickets (someone must act, but not now) and logs (nobody needs to look at them except for diagnosis). Anything that does not require immediate action goes to a ticket, not to a phone.

The most precise way to alert on symptoms is to alert on the SLO. In Alerting on SLOs, Steven Thurgood and coauthors define burn rate as how fast the service consumes its error budget relative to the SLO. For a 99.9% SLO they propose:

The Workbook adds two practices: test every new alert in production before upgrading it to a page, to weed out false positives, and look for the root cause of every page.

3. Runbooks

Treynor Sloss writes that recording best practices ahead of time in a playbook improves mean time to repair roughly 3 times compared with improvising. The Workbook specifies that a playbook gives high-level instructions; if it is a fixed list of commands that always runs the same way, it should be automated. A useful runbook per alert answers:

Alert: <exact name, as it appears in the page>
What it means: <symptom and user impact>
Dashboard: <link>
First checks: <2 or 3 steps>
Mitigation: <rollback, shift traffic, scale resources>
Escalate to: <person or team, and when>
Last reviewed: <date and owner>

4. Shift handoff

In the Workbook, whoever starts on-call reads the handoff from the previous shift, and whoever ends it sends one to the next person; if it falls short of expectations, the team talks about it. PagerDuty recommends a formal handoff (a meeting, an email or a shared document) that includes recent deployments, pending postmortem actions, open tasks and staff changes on key teams. It also suggests switching on Mondays, to avoid changing shifts over the weekend.

On-call handoff · <date> · outgoing <role> → incoming <role>
Incidents this shift: <count, link to each>
Open now: <what is still ongoing and next step>
Noisy alerts: <which ones, ticket filed>
Recent or scheduled deployments: <what and when>
Pending postmortem actions: <links>
Absences or escalation changes: <who is out>

5. Roles during an incident

In the Managing Incidents chapter, Andrew Stribblehill describes four roles: incident commander (holds the full picture and assigns tasks), operations (the only group that modifies the system), communications (the public face) and planning (files bugs and handles what comes next). For him, it is an incident if a second team is needed, if customers can see it, or if it remains unsolved after an hour of analysis. The commander's most important task is to keep a living incident document, and the role is handed off explicitly.

With few people, the key rule is in the Workbook's Incident Response chapter (Jennifer Mace and coauthors): the commander holds every role not yet delegated, and if the incident shrinks, communications goes back to the commander. Their advice is to declare incidents early and often. Applied to a small team, one option is this: with two people, one commands and communicates and the other operates; with three, a third person takes over communications. PagerDuty adds the scribe role, who records the timeline, and its severity levels give a useful rule: if you are unsure between two levels, use the higher one. Atlassian stresses that the incident manager is a role, not a person, so that anyone prepared can take it.

6. Blameless postmortems

In Postmortem Culture, John Lunney and Sue Lueder define a postmortem as the written record of an incident: impact, mitigation actions, causes and actions to prevent recurrence. They suggest writing one when there is visible downtime or degradation beyond a threshold, data loss, on-call intervention (rollback, rerouting traffic), slow resolution or a monitoring failure. It is blameless if it looks for causes without indicting people or teams: you cannot "fix" people, but you can fix systems and processes.

John Allspaw, in his 2012 article at Etsy, explains why it works: without punishment, the people involved tell what they did, what they saw, what they expected and what they assumed. Atlassian, in its postmortem process, asks to refer to people by role and to ask why the system allowed that action. The Workbook (Daniel Rogers and coauthors) shows the difference: instead of blaming whoever ran a process manually, it states that the server did not validate the commands it sent.

On timing, PagerDuty schedules the meeting within 3 calendar days for SEV-1 and 5 business days for SEV-2, lasting 15 to 30 minutes. This template follows Google's example postmortem and PagerDuty's template:

Postmortem · <title> · <date> · status: draft/reviewed
Owner: <role>    Participants: <roles>
Summary: <2 sentences: what happened, how long, who was affected>
Impact: <users, transactions, minutes, data>
Detection: <alert, customer or person; how long it took>
Trigger: <change or event that started it>
Causes: <system and process conditions, no names>
Resolution: <what mitigated it and what resolved it>
Timeline (local time): <time · event · source>
What went well / what went wrong / where we got lucky
Actions: <action · type (mitigate/prevent/process) · owner · date · ticket>

7. Following up on actions

A postmortem without closed actions changes nothing. The Workbook warns that without a formal process, actions are forgotten and the incident comes back; that is why Google files them as bugs in a central system and tracks their closure. Each action needs an owner and a verifiable end state. It also notes that changing human behavior is less reliable than changing automated systems.

Atlassian labels the actions that fix the root cause as "priority actions" and gives them a 4- or 8-week deadline depending on the service, with reminders. It uses the five whys technique and manager approval, although it acknowledges that in teams with few incidents that approval may not be necessary.

With few people: one ticket per action in the same system as daily work, a committed date and a monthly review of open items.

Checklist

Sources

• Google · Site Reliability Engineering: Introduction (Benjamin Treynor Sloss).
• Google · Being On-Call (Andrea Spadaccini).
• Google · Monitoring Distributed Systems (Rob Ewaschuk).
• Google · Managing Incidents (Andrew Stribblehill).
• Google · Postmortem Culture: Learning from Failure (John Lunney and Sue Lueder) and Example Postmortem.
• Google · The Site Reliability Workbook: On-Call, Alerting on SLOs, Incident Response and Postmortem Culture.
• PagerDuty · Incident Response Documentation: roles, severity levels, process and postmortem template; Apache 2.0 license.
• PagerDuty · Going On-Call: Technical Preparations and Setting Team Norms.
• Atlassian · Incident Management Handbook and Incident postmortems.
• John Allspaw · Blameless PostMortems and a Just Culture (Etsy, May 22, 2012).
• Chile's Labor Code, DFL 1 of 2002, on LeyChile: articles 21, 25 ter, 30, 31 and 32.

If you want to review your alerts or your on-call rotation with your team, let's talk.