Real-Time SRE Interview Questions: Incident Drills (2027)
44 real-time SRE interview questions in incident order: first page, Linux triage, rollback calls, error budget maths, comms and postmortems.
Real-time SRE interview questions put you inside a live incident and ask what you do next: the first five minutes after a page, how you triage a slow host, when you roll back, who talks to stakeholders, how much error budget an outage burned, and what goes in the postmortem. This page walks through 44 such questions in the order an incident unfolds.
Incident roles and practices below follow Google's published SRE book and workbook chapters, and the Linux triage list follows Netflix's public "Linux performance analysis in 60 seconds" post. Links are in the references.
Key takeaways
- Mitigate first, diagnose second. A rollback or traffic shift that stops user pain beats a perfect root cause found an hour later.
- Declare an incident early and name an incident commander. The SRE book separates command, operations, communication and planning roles.
- Learn the error budget numbers: 99.9% over 30 days allows 43.2 minutes of full downtime; 99.99% allows about 4.3 minutes.
- Have a fixed first-minute triage routine for a Linux host so you are not improvising under pressure.
- Postmortems are blameless and end with owned, dated action items.
Which SRE page to use
The keeper is 100+ SRE interview questions. For design-heavy incident stories read scenario-based SRE questions; for the full interview loop read site reliability engineer questions by round; for senior topics read advanced SRE questions. The SRE sub-pillar, the role of SREs in DevOps teams, explains the job itself. This page is a timeline drill: each section is one phase of a live incident.
| Availability SLO | Error budget (30 days) | Allowed full downtime |
|---|---|---|
| 99% | 1% | 7.2 hours |
| 99.5% | 0.5% | 3.6 hours |
| 99.9% | 0.1% | 43.2 minutes |
| 99.95% | 0.05% | 21.6 minutes |
| 99.99% | 0.01% | 4.32 minutes |
Phase 1: the page arrives (freshers and juniors)
1. You are paged at 3 a.m. What are your first three actions?
Acknowledge the page so it does not escalate, open the alert's dashboard and runbook, and confirm user impact (error rate, latency, traffic). Only then decide whether this is an incident to declare or a known, minor issue.
2. How do you confirm an alert is real?
Check a second signal: if the alert says 5xx errors are up, look at the load balancer or synthetic checks too. Then check whether the monitoring itself is broken, for example a scrape failure that shows as zero traffic.
3. When do you declare an incident?
The SRE book's criteria: a second team must be involved, the outage is visible to customers, or the issue is still unsolved after an hour of focused analysis. Declaring early costs little; declaring late means confused, uncoordinated work.
4. What is an incident commander?
The person who holds the overall picture, assigns work and makes decisions. In the SRE book's model, the commander delegates operations work, communication and planning to others and does not fix things directly unless no one else is available.
5. What are the other incident roles?
An operations lead who makes changes to the system, a communications lead who updates stakeholders and status pages, and a planning lead who handles longer tasks such as tracking follow-ups and handoffs. On a small team one person may hold several roles; say so.
6. What should the first status update contain?
What users see, when it started, the current severity, what is being done, and when the next update will come. Do not guess at causes in a public update.
7. What does "mitigate first" mean?
Stop the user impact before understanding it fully: roll back, fail over, shed load, disable a feature flag or add capacity. The root cause can be found after users are safe.
8. Why keep a timeline during the incident?
Memory is unreliable at 3 a.m. A timestamped log of what was seen and done makes handoffs clean and makes the postmortem accurate.
Phase 2: triage under pressure (2 to 5 years)
9. A host is slow. What do you run in the first minute?
A fixed list, adapted from Netflix's public 60-second checklist:
uptime # load averages, trend
dmesg -T | tail # OOM kills, disk or network errors
vmstat 1 5 # run queue, swapping, CPU split
mpstat -P ALL 1 3 # one hot CPU?
pidstat 1 3 # which process
iostat -xz 1 3 # disk latency and utilisation
free -m # memory and cache
sar -n DEV 1 3 # network throughput
sar -n TCP,ETCP 1 3 # connections, retransmits
top # confirm
Some of these come from the sysstat package, so say you would check it is installed.
10. Load average is 40 on an 8-core box but CPU is mostly idle. What does that tell you?
Linux load includes tasks waiting in uninterruptible sleep, usually on disk or NFS. Check iostat for high await and vmstat for the blocked column, and look for a hung mount.
11. The disk is full on a production host. What do you do now?
Find what grew with df -h and du -xh --max-depth=1. Free space safely: compress or move old logs, clear known caches. If df says full but du does not, a deleted file is still open; find it with lsof +L1 and restart that process.
12. Error rates rose right after a deploy. What is your call?
Roll back first. A deploy that lines up with the start of errors is the most likely cause, and a rollback is quick and reversible. Investigate the change once users are recovered.
13. Errors rose and nothing was deployed. Where do you look?
Config and feature flag changes, dependency health (database, cache, third-party APIs), traffic shape (a spike or a single heavy client), certificate expiry, and infrastructure events from the cloud provider.
14. How do you tell if one bad instance is causing the errors?
Break errors down by instance, zone and version on the dashboard. If one instance or zone stands out, remove it from the load balancer and keep it for analysis.
15. Latency p99 doubled but p50 is flat. What does that suggest?
A subset of requests is slow: one slow dependency, one heavy endpoint, garbage collection pauses, or one overloaded instance. Look at traces for the slow requests rather than averages.
16. A service runs out of database connections. Immediate options?
Find the client holding the most connections, restart it or scale down the noisy consumer, and check for slow queries holding connections. Raising the pool size can make the database worse, so do it only with headroom.
17. How do you check if the network is the problem?
Look for TCP retransmits (sar -n ETCP, ss -ti), DNS resolution time, and errors on interfaces. Test the path with curl -w timings from the client side, which split DNS, connect, TLS and first byte.
18. What is load shedding and when do you use it?
Rejecting some requests on purpose, ideally the least important, so the service can serve the rest. Use it when demand exceeds capacity and scaling cannot catch up fast enough.
19. What is a cascading failure?
One overloaded component fails, its load moves to others, and they fail in turn. Retries without backoff often make it worse. The fix during an incident is to cut load: shed, rate limit, or turn off retries.
20. When do you escalate to another team?
As soon as the evidence points into their system, or when your runbook ends. Give them the symptom, the time it began, what you checked and what you need.
21. How do you hand off an incident at the end of your shift?
Explicitly: state current status, what has been tried, open questions and who holds each role now, and get a verbal or written acknowledgement. The SRE book stresses a clear handoff of the commander role.
Phase 3: numbers you calculate on the spot
22. How much downtime does a 99.9% monthly SLO allow?
30 days is 43,200 minutes; 0.1% of that is 43.2 minutes of full outage.
23. A 20-minute outage hit 50% of requests. How much of a 99.9% budget is gone?
Effective full-outage time is 20 x 0.5 = 10 minutes. 10 / 43.2 is about 23% of the monthly budget.
24. Your service has 99.95% availability and calls two dependencies that each offer 99.9%. Is your SLO realistic?
If both are hard dependencies in series, the best case is about 0.999 x 0.999, roughly 99.8%, so a 99.95% SLO is not realistic without caching, retries across replicas, or graceful degradation.
25. What is a burn rate of 14.4 in plain terms?
Spending budget 14.4 times faster than the rate that would use it all in 30 days. At that pace, a 99.9% budget lasts about two days. The SRE Workbook uses 14.4 over one hour as its fast-burn page.
26. How do you compute MTTR from incident data, and why be careful with it?
Average the time from start (or detection) to recovery across incidents. Be clear about which start time you use, and note that a few long incidents dominate the mean, so report the median too.
27. Traffic is 2,000 requests per second and the error ratio is 0.3%. How many errors per minute?
2,000 x 0.003 = 6 errors per second, or 360 per minute.
Phase 4: recovery and follow-up (senior)
28. When is an incident over?
When user impact has ended and metrics have been stable for an agreed period, not when the first fix goes out. The commander declares it over and announces it.
29. What is a blameless postmortem?
A written review that looks at how the system and process allowed the failure, without blaming individuals. The SRE book argues that blame makes people hide information, which stops the organisation learning.
30. What sections does a good postmortem have?
Summary, impact (with numbers), timeline, root causes and triggers, what went well, what went badly, where we got lucky, and action items with owners and dates.
31. How do you make sure action items get done?
Each one is a ticket with one owner and a date, prioritised against feature work. Review open items in a regular meeting, and track how many are closed on time.
32. When is a postmortem required?
Set criteria in advance, for example user-visible downtime above a threshold, data loss, an on-call intervention such as a rollback, or a monitoring failure. Anyone may also request one.
33. How does an incident affect the error budget policy?
If the budget is exhausted, the agreed policy applies, for example freezing feature launches until reliability work brings the service back within SLO. The policy should be written and agreed with product owners before incidents happen.
34. How do you reduce time to detect?
Alert on symptoms users feel, use burn-rate alerts rather than static thresholds, add synthetic checks for key journeys, and review incidents found by customers before monitoring.
35. How do you reduce time to mitigate?
Make rollback one command, keep feature flags for risky changes, practise failover, and keep runbooks current with the exact commands. Test them in game days.
36. What is a game day or incident drill?
A planned exercise where the team handles a simulated or injected failure, practising roles, tools and communication. It finds broken runbooks and missing access before a real outage does.
37. How do you keep on-call sustainable?
Track pages per shift and out-of-hours pages, remove noisy alerts, cap time spent on operational work, and make sure people get rest after a night incident. The SRE book recommends a cap on operational load for SRE teams.
Scenario drills
38. Checkout errors jump to 30% at 7 p.m. during a sale. Walk through it.
Acknowledge, confirm impact on dashboards, declare a high-severity incident and take or assign command. Check recent changes; if a deploy or flag went out, roll it back. If not, check capacity and the payment dependency, shed non-essential traffic, scale out, and post updates every 15 to 30 minutes.
39. The rollback does not fix it.
The deploy was not the cause, or it changed data or a schema that the rollback did not undo. Widen the search: dependencies, config, traffic. Bring in the owning teams and keep updating stakeholders.
40. Monitoring is down during an outage.
Fall back to other signals: load balancer logs, cloud provider metrics, synthetic checks from outside, and customer reports. Afterwards, make sure monitoring does not depend on the same infrastructure it watches.
41. Two engineers start making conflicting changes.
The commander stops both, picks one plan, and makes sure only the operations lead changes production. Record the decision in the timeline.
42. An executive joins the incident call and demands an ETA.
The communications lead gives the facts: impact, what is being tried, and the time of the next update, without promising a fix time you cannot back. Keep the working channel for the people fixing it.
43. A certificate expired and took down an API.
Renew and deploy the certificate first. In the postmortem, add expiry monitoring with alerts weeks ahead and automate renewal where the platform allows.
44. The same incident has happened three times this quarter.
The action items are not being done or do not address the cause. Review the three postmortems together, escalate the fix as a priority with the error budget data, and check whether the policy is being followed. For more incident stories see scenario-based SRE questions and why time to restore service matters.
How to prepare for real-time rounds
Practise out loud. Take a past outage write-up from a public postmortem collection or an engineering blog and narrate what you would do minute by minute, naming roles, commands and updates. Run the triage list on a VM until it is automatic. Our DevOps training includes monitoring and incident practice, in the Kharadi, Pune classroom and live online.
Frequently asked questions
What are real-time SRE interview questions?
Questions that place you in a live incident and ask what you do next, from the first page through mitigation, communication and the postmortem.
How much downtime does 99.9% availability allow per month?
About 43.2 minutes of full downtime in a 30-day month.
Should I find the root cause before fixing an outage?
No. Mitigate first, for example by rolling back or failing over, then find the root cause once users are safe.
What does an incident commander do?
Holds the overall picture, assigns roles and work, makes decisions and declares the incident over, while others make changes and handle communication.
What goes into a blameless postmortem?
Impact, timeline, causes, what went well and badly, and action items with owners and due dates, without blaming individuals.
What Linux commands should I know for live triage?
uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar and top, plus df, du, ss and lsof for common follow-ups.
What is a burn rate?
How fast an error budget is being spent compared with the rate that would use it exactly over the SLO period.
Official references
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0