SRE Monitoring and Automation Interview Questions (2027)
44 SRE monitoring and automation interview questions: SLOs, burn-rate alerts, PromQL, Alertmanager, toil, auto-remediation and scenarios.
SRE monitoring and automation interview questions test two linked skills: measuring a service well enough to know when users are hurting (SLIs, SLOs, alerting on symptoms, dashboards) and removing repetitive operational work with code (auto-remediation, runbook automation, infrastructure as code, safe deploys). Interviewers want answers grounded in Prometheus, OpenTelemetry, Kubernetes and the Google SRE practices.
This page has 44 questions with direct answers, grouped by experience level, followed by scenario questions. It focuses only on monitoring and automation. For the full SRE question bank (culture, capacity planning, incident command) see our 100 SRE interview questions, and for the role itself read what SREs do in DevOps teams.
Key takeaways
- Monitoring questions start from users: define SLIs from what users experience, set SLOs, and alert when the error budget burns too fast.
- Know the four golden signals (latency, traffic, errors, saturation) and the RED and USE methods, and when each fits.
- Prometheus is the default reference tool: PromQL, recording rules, alerting rules with
for, and Alertmanager routing, grouping and inhibition. - Automation questions are about toil: work that is manual, repetitive, automatable and grows with the service. Expect to explain how you pick what to automate first.
- Senior answers include safety: automation needs rate limits, dry runs, audit logs and a way to turn it off.
- Scenario rounds check whether you can connect a noisy alert, a slow rollout or a recurring outage to a concrete monitoring or automation change.
How the questions are organised
| Section | Questions | What interviewers check |
|---|---|---|
| Freshers and junior SREs | 1 to 14 | Vocabulary, golden signals, basic Prometheus and alerting |
| 2 to 5 years | 15 to 30 | SLO alerting, PromQL, Alertmanager, Kubernetes automation, runbooks |
| Senior SREs | 31 to 38 | Monitoring strategy, safe automation, scaling observability, trade-offs |
| Scenarios | 39 to 44 | Applying all of the above to a real problem |
Freshers and junior SREs: questions 1 to 14
1. What is the difference between monitoring and observability?
Monitoring watches known signals and alerts when they cross a line; observability is how well you can explain new, unexpected behaviour from the telemetry the system emits. Monitoring answers "is it broken?"; observability helps answer "why?". Our post on observability patterns in modern cloud goes further.
2. What are the four golden signals?
Latency, traffic, errors and saturation, as described in the Google SRE book. If you can measure only four things about a user-facing service, measure these. Track latency of failed requests separately from successful ones, because fast errors can make average latency look better.
3. What are SLIs, SLOs and SLAs?
An SLI is a measurement (for example the proportion of requests served in under 300 ms); an SLO is the target for it (99.9% over 30 days); an SLA is a contract with consequences if the target is missed. SLOs are internal and usually stricter than SLAs.
4. What is an error budget?
The amount of unreliability an SLO allows. A 99.9% availability SLO over 30 days allows about 43 minutes of full downtime, or the equivalent in failed requests. When the budget is spent, teams slow feature releases and work on reliability.
5. What is toil?
Operational work that is manual, repetitive, automatable, tactical, has no lasting value and grows linearly with the service. Restarting a stuck job by hand every morning is toil. Google's SRE book suggests keeping toil below 50% of an SRE's time.
6. What are the RED and USE methods?
RED (Rate, Errors, Duration) is for request-driven services; USE (Utilization, Saturation, Errors) is for resources such as CPU, disks and queues. Use RED on every microservice dashboard and USE on infrastructure dashboards.
7. What are the three main telemetry types?
Metrics, logs and traces. Metrics are cheap numeric time series for alerting and trends. Logs are detailed event records. Traces follow one request across services. OpenTelemetry provides a vendor-neutral way to produce all three, and profiles are being added as a fourth signal.
8. What is Prometheus and how does it collect data?
Prometheus is an open-source monitoring system and time series database that pulls (scrapes) metrics over HTTP from targets that expose a /metrics endpoint. It finds targets through static config or service discovery (for example Kubernetes), stores the data locally and evaluates alerting rules.
9. What are the four Prometheus metric types?
Counter, gauge, histogram and summary. A counter only goes up (requests served). A gauge goes up and down (memory in use). Histograms and summaries record distributions such as request duration; histograms can be aggregated across instances, which is why they are usually preferred.
10. Why should you alert on symptoms rather than causes?
Users feel symptoms (errors, slowness), not causes (high CPU). High CPU with no user impact should not wake anyone. Page on symptoms tied to SLOs and keep cause metrics on dashboards for diagnosis.
11. What makes a good alert?
It is urgent, actionable, real and points to a runbook. If the person paged cannot do anything useful, it should be a ticket or a dashboard, not a page.
12. What is a runbook?
A short document that tells the on-call engineer how to diagnose and fix a specific alert. Good runbooks list what the alert means, the first checks, safe mitigations and who to escalate to. Every paging alert should link to one.
13. What is Grafana used for in an SRE team?
Dashboards and visualisation on top of data sources such as Prometheus, Loki and Tempo. SREs use it for service dashboards, SLO views and incident investigation. See our guide to using Prometheus and Grafana together.
14. Give an example of a simple automation an SRE might write in the first month.
A script or scheduled job that does a manual task the team repeats, with logging. Examples: cleaning old log files when disk passes a threshold, rotating a certificate, or gathering diagnostics automatically when an alert fires so the on-call engineer sees them in the ticket.
2 to 5 years of experience: questions 15 to 30
15. Write a Prometheus alerting rule for a high error rate.
Use rate() over counters, divide errors by total, and add a for duration so brief spikes do not page.
groups:
- name: checkout
rules:
- alert: CheckoutHighErrorRate
expr: |
sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m])) > 0.02
for: 10m
labels:
severity: page
annotations:
summary: "Checkout 5xx ratio above 2% for 10 minutes"
runbook_url: "https://wiki.example.com/runbooks/checkout-errors"
16. Why is rate() used on counters instead of the raw value?
The raw counter value only grows and resets on restart. rate() calculates the per-second increase over a window and handles counter resets, which is what you actually want to graph or alert on.
17. How do you calculate p99 latency in Prometheus?
Use histogram_quantile over the rate of histogram buckets, aggregated by le.
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))
The result is an estimate whose accuracy depends on the bucket boundaries you chose.
18. What are recording rules and why use them?
Recording rules precompute expensive queries and store the result as a new time series. They make dashboards fast and keep alert expressions simple, especially for SLO calculations over long windows.
19. What does Alertmanager do?
It receives alerts from Prometheus and handles deduplication, grouping, routing, silencing and inhibition, then sends notifications to email, chat or paging tools. Prometheus decides that something is wrong; Alertmanager decides who hears about it and how.
20. Explain grouping and inhibition in Alertmanager.
Grouping combines related alerts into one notification; inhibition suppresses some alerts while another is firing. For example, group by cluster and alertname so a cluster failure sends one message, and inhibit all service alerts in a data centre when a "datacenter down" alert fires.
21. What is burn rate, and how do multi-window, multi-burn-rate alerts work?
Burn rate is how fast you consume the error budget relative to the SLO period; a burn rate of 1 uses exactly the whole budget by the end of the period. The Google SRE Workbook recommends pairing a long and a short window, for example paging when the burn rate exceeds 14.4 over 1 hour (confirmed over 5 minutes), and opening a ticket for slower burns such as 1 over 3 days. This catches fast outages quickly and slow leaks eventually, with few false pages.
22. How do you choose a good SLI for an API?
Measure as close to the user as practical, as a ratio of good events to valid events. Load balancer logs or metrics are often a better source than the application itself because they include requests the app never saw. Exclude invalid requests such as client errors you do not control.
23. How do you monitor a batch job or cron job?
Alert on the absence of success, not on failure messages. Push a "last successful run" timestamp (for example with the Prometheus Pushgateway for short-lived jobs) and alert when it is older than expected. A job that silently stops running never logs an error.
24. What Kubernetes features are forms of automation SREs rely on?
Controllers reconcile desired state: Deployments replace failed pods, liveness and readiness probes restart or remove unhealthy containers, and the Horizontal Pod Autoscaler scales replicas on metrics. PodDisruptionBudgets keep enough replicas running during voluntary disruptions such as node drains.
25. How would you automate remediation for a known failure?
Start with diagnosis, then add a safe, reversible action behind a trigger. For example, when an alert fires for a stuck worker, an automation collects diagnostics, restarts the worker, posts what it did in the incident channel and escalates if the alert does not clear. Tools range from Alertmanager webhooks calling a small service, to Ansible playbooks, to Kubernetes operators.
26. How do you treat monitoring configuration as code?
Keep alert rules, dashboards and SLO definitions in Git, review them like application code, and deploy them through CI. Prometheus rules can be validated with promtool check rules and unit-tested with promtool test rules. Grafana dashboards can be provisioned from files.
promtool check rules alerts/checkout.yml
promtool test rules tests/checkout_test.yml
27. What is the role of OpenTelemetry in an SRE monitoring stack?
OpenTelemetry standardises how applications produce traces, metrics and logs, and the Collector receives, processes and exports them to any backend. It lets you change vendors without re-instrumenting and add sampling or attribute filtering centrally.
28. How do you control high cardinality in metrics?
Keep unbounded values (user IDs, request IDs, full URLs) out of metric labels. Put them in logs or trace attributes instead. Use relabelling to drop unneeded labels at scrape time and watch the number of active series per job.
29. How do you make deployments safer with automation?
Use progressive delivery: canary or blue-green releases with automated analysis that compares metrics between old and new versions and rolls back on failure. Argo Rollouts and Flagger are common tools on Kubernetes. Pair this with feature flags so risky code can be turned off without a redeploy.
30. What should an on-call handover include?
Open incidents, alerts that fired and why, ongoing mitigations, known risks such as planned changes, and any monitoring gaps found. Many teams automate the first draft from the alerting tool so the handover takes minutes.
Senior SREs: questions 31 to 38
31. How would you design monitoring for a new platform used by 50 teams?
Provide a paved road: standard instrumentation libraries, default dashboards and SLO templates generated per service, and alert routing by team ownership metadata. Make the defaults good enough that teams only add what is specific to them, and track adoption.
32. How do you decide what to automate first?
Rank toil by frequency, time spent, risk of human error and how often it pages people at night. Automate the tasks that score high on several of these. Measure time saved afterwards so you can justify the next project.
33. What safeguards should auto-remediation have?
Rate limits, scope limits, dry-run mode, audit logs, an easy kill switch and escalation to a human when the action does not work. An automation that restarts every pod in a cluster because of a bad metric can cause the outage it was meant to prevent.
34. How do you scale Prometheus for many clusters and long retention?
Keep a Prometheus per cluster for local alerting, and add a long-term, global layer such as Thanos, Cortex or Grafana Mimir (or remote write to a managed service) for cross-cluster queries and retention. Use recording rules to reduce what you send upstream.
35. How do you know if your alerting is healthy?
Track pages per on-call shift, the percentage that were actionable, time to acknowledge, and incidents found by users before alerts. Review these monthly and remove or fix alerts that are not useful.
36. When should an SLO be changed?
When it no longer matches what users need or what the business will fund. If users are unhappy while the SLO is met, it is too loose; if the team is always out of budget but users are fine, it may be too strict. Change SLOs deliberately, with product owners, not to make a bad month look good.
37. How do you monitor the monitoring system?
Use a separate, simple watchdog. An always-firing "dead man's switch" alert sent to an external service tells you when the alert pipeline stops. Monitor Prometheus and Alertmanager themselves from a second instance or a managed probe.
38. How do you balance build-versus-buy for observability?
Compare the full cost: licences or usage fees against the engineering time to run, scale and upgrade an open-source stack. Standardising on OpenTelemetry keeps both options open. Many teams run open-source for metrics and buy a service for areas where they lack depth.
Scenario questions: 39 to 44
39. The on-call engineer received 60 pages last week and most needed no action. What do you do?
Review every page, tag it actionable or not, then delete, downgrade or rewrite. Replace threshold alerts on causes with SLO burn-rate alerts, add for durations and grouping, and route warnings to tickets. Report the before-and-after page count.
40. An outage was reported by customers 20 minutes before any alert fired. How do you close the gap?
Find which signal changed first and why nothing watched it. Often the SLI was measured in the wrong place or the alert window was too long. Add a symptom-based alert at the edge, add synthetic probes for the main user journey, and test the alert by replaying the incident data.
41. Engineers restart the same service by hand three times a week. What do you propose?
Short term, automate the restart safely; long term, fix the cause. Add diagnostics capture and a guarded restart (rate-limited, logged, announced in chat), then use the captured data to find the leak or deadlock and get the fix prioritised through the error budget discussion.
42. A canary deployment passed automated analysis, but the full rollout caused errors. Why might that happen?
The canary did not see the same traffic or the analysis looked at the wrong metrics. Check whether the canary served too little traffic for statistical confidence, whether a specific customer segment or region was missing, and whether the metrics compared included error rate per endpoint. Extend the analysis and canary duration.
43. Your Prometheus server runs out of memory every few days. How do you investigate?
Check active series and which jobs or metrics produce the most. The TSDB status page and queries such as topk(10, count by (__name__)({__name__=~".+"})) show the largest metrics. A new deploy that added a high-cardinality label is the usual cause; drop it with relabelling and add a limit per scrape job.
44. Leadership asks you to cut infrastructure toil by half in a quarter. How do you plan it?
Measure toil first, then automate the biggest items with clear owners and deadlines. Ask engineers to log manual operational work for two weeks, group it, estimate hours, and pick the top items. Track hours saved per item and report monthly. Infrastructure as code with SRE automation tools usually covers several items at once.
Preparing for the interview
Run a small lab: a Kubernetes cluster with a demo app, Prometheus, Alertmanager and Grafana. Write an SLO, a burn-rate alert and a small auto-remediation webhook, then break the app and watch what happens. Our scenario-based SRE interview questions and advanced SRE questions for experienced engineers cover the incident and culture side. For guided labs, our Prometheus and Grafana training covers PromQL, alerting and dashboards, and the DevOps training program covers the Kubernetes and CI/CD automation around it.
Frequently asked questions
What monitoring topics are asked in SRE interviews?
Expect SLIs, SLOs and error budgets, the four golden signals, RED and USE methods, Prometheus metric types and PromQL, alerting rules, Alertmanager routing and grouping, and how to alert on symptoms instead of causes.
What is a burn-rate alert?
It alerts on how fast the error budget is being consumed. The Google SRE Workbook suggests pairing long and short windows, for example paging at a burn rate of 14.4 over one hour, so fast outages page quickly and slow burns create tickets.
What automation questions do SRE interviews include?
How you identify and measure toil, how you choose what to automate, how you build safe auto-remediation, how monitoring config is managed as code, and how progressive delivery automates safe rollouts and rollbacks.
Which tools should I know for SRE monitoring interviews?
Prometheus, Alertmanager and Grafana are the common reference stack, plus OpenTelemetry for instrumentation. Knowing one commercial platform such as Datadog or New Relic, and Kubernetes probes and autoscaling, also helps.
What is toil in SRE?
Toil is operational work that is manual, repetitive, automatable, tactical, has no lasting value and grows with the service. Google's SRE book recommends keeping it below half of an SRE's time.
How do I prepare for SRE automation interview questions without production experience?
Build a lab with a demo app on Kubernetes, Prometheus and Grafana. Write SLOs and alerts, automate a restart or cleanup with a script or Ansible, break things on purpose and explain what you observed.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0