Prometheus Alerting Interview Questions: Alert Rules (2027)
42 Prometheus alerting interview questions on alert rules, for and keep_firing_for, templates, SLO burn-rate alerts and promtool rule tests.
Prometheus alerting interview questions test whether you can write alert rules that page for real user impact and stay quiet otherwise. Expect questions on the rule fields (expr, for, keep_firing_for, labels, annotations), templating, the pending and firing states, SLO burn-rate alerts, and unit testing rules with promtool. This page answers 42 of them.
Everything here is about the Prometheus side: the rules that decide when an alert fires. Routing, grouping, silences and receivers live in Alertmanager and are covered in our Alertmanager interview prep guide. Field names and behaviour were checked against the Prometheus documentation in November 2026.
Key takeaways
- An alerting rule is a PromQL expression; every series it returns becomes one alert instance identified by its labels.
forholds an alert in pending until the condition has been true for that long.keep_firing_forkeeps it firing after the condition clears, which stops flapping.- Templates can use
$labels,$valueand$externalLabels. - Page on symptoms (errors, latency, error budget burn), not on causes such as CPU.
- The Google SRE Workbook's multiwindow burn-rate table (14.4, 6 and 1) is the reference answer for SLO alerts.
How this page fits with the other alerting content
The monitoring pillar is top observability tools for DevOps. For broad Prometheus coverage use 80+ Prometheus interview questions; for configuration files, see most asked Prometheus config questions. If you need a list of which alerts to create in the first place, read 10 monitoring alerts every DevOps team should configure.
| Field | Required | What it does |
|---|---|---|
alert | Yes | Alert name, becomes the alertname label |
expr | Yes | PromQL; each returned series is one alert instance |
for | No | Time the condition must hold before firing |
keep_firing_for | No | Time to keep firing after the condition was last true |
labels | No | Extra labels such as severity; overwrite conflicting labels; templatable |
annotations | No | Summary, description, runbook URL; templatable |
Freshers: alert rule basics
1. What is an alerting rule in Prometheus?
A rule in a rules file that evaluates a PromQL expression on every evaluation cycle. Each series in the result becomes an active alert. Prometheus sends active alerts to Alertmanager, which handles notification.
2. Write a simple alert for a target that is down.
groups:
- name: availability
rules:
- alert: TargetDown
expr: up == 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} target {{ $labels.instance }} is down"
3. What does the for clause do?
It makes the alert wait. While the expression is true but for has not elapsed, the alert is pending. Once the duration passes, it fires. Without for, the alert fires on the first evaluation that returns a result.
4. What are the alert states?
Inactive (no matching series), pending (condition true, for not yet met) and firing. You can see all three on the Alerts page of the Prometheus UI.
5. How often are alert rules evaluated?
Every evaluation_interval, which defaults to one minute, unless the rule group sets its own interval. A for: 30s on a one-minute interval effectively behaves like one minute.
6. What is the difference between labels and annotations?
Labels identify the alert and drive routing in Alertmanager, so keep them stable and low cardinality (severity, team). Annotations carry human text such as a summary, a description and a runbook link, and do not affect identity.
7. Which variables can you use in alert templates?
$labels for the alert's labels ($labels.instance), $value for the evaluated value, and $externalLabels for the server's external labels. Templates use Go template syntax in double braces.
8. What is the ALERTS metric?
A synthetic series Prometheus writes for each pending or firing alert, with labels alertname and alertstate and a value of 1. You can graph it, or build meta-alerts and dashboards from it.
9. Where do alerting rules live and how are they loaded?
In YAML rule files listed under rule_files in prometheus.yml, grouped under groups. Alerting and recording rules can share a file. Changes are picked up on a config reload.
10. How do you check a rules file for syntax errors?
Run promtool check rules rules/alerts.yml. It catches YAML and PromQL errors before the file reaches a server.
11. Why does Prometheus not send emails itself?
Prometheus only evaluates rules. Alertmanager deduplicates, groups, silences, inhibits and routes alerts to receivers such as email, Slack or PagerDuty. Splitting the two lets several Prometheus servers share one notification pipeline.
12. What does a good alert summary contain?
What is broken, where, and how bad: the service, the instance or cluster, and the current value. Add a runbook_url annotation so the person paged knows the first steps.
2 to 5 years: writing alerts that hold up
13. Why should you alert on symptoms rather than causes?
Users feel errors and latency, not CPU. A busy CPU with healthy latency needs no page; a slow service with idle CPU does. The Google SRE book makes the same case in its monitoring chapter. Keep cause alerts as tickets or dashboard signals.
14. Write an error-ratio alert.
- alert: HighErrorRatio
expr: |
sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "{{ $labels.service }} 5xx ratio is {{ $value | humanizePercentage }}"
15. Why use rate() and not the raw counter?
Counters only go up and reset on restart. rate() handles resets and gives a per-second rate over a window, which is what a threshold can compare against.
16. How do you write a latency alert from a histogram?
Use histogram_quantile() over the rate of the bucket series, aggregated by le and the labels you route on.
expr: |
histogram_quantile(0.99,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5
17. How do you alert when a metric disappears?
Use absent(), for example absent(up{job="payments"}). A plain up == 0 returns nothing if the target vanished from discovery, so it never fires.
18. How do you alert before a disk fills?
Use predict_linear() on free space, for example predicting four hours ahead from six hours of data, and alert if the prediction drops below zero. Combine it with a static threshold so a sudden spike is also caught.
19. What problem does keep_firing_for solve?
Flapping. If the condition dips below the threshold for one evaluation and comes back, the alert would resolve and re-fire, sending two notifications. With keep_firing_for: 10m it stays firing until the condition has been false for ten minutes.
20. How do you choose a for duration?
Long enough to skip blips your users would not notice, short enough that the page still arrives in time to act. For burn-rate alerts the window does this job, so for is often short or omitted.
21. Why aggregate with by() in alert expressions?
Without aggregation you get one alert per pod or instance, which can mean dozens of pages for one outage. Aggregate to the level someone acts on, usually service and cluster.
22. What labels should every alert carry?
A severity that Alertmanager routes on and an owner label such as team. Avoid labels with changing values, like a timestamp or a value, because they change the alert's identity and break deduplication.
23. Why should $value go in annotations, not labels?
Labels define identity. If the value is a label, every evaluation creates a new alert and the old one resolves. Annotations can change freely.
24. How do recording rules help alerting?
They precompute expensive expressions, such as error ratios over several windows, so alert rules stay cheap and consistent with dashboards. Burn-rate setups rely on them.
25. How do you unit test an alert?
Write a test file with input_series and an alert_rule_test that states the expected alerts at an eval_time. Run promtool test rules. An empty exp_alerts asserts that nothing fires.
rule_files:
- alerts.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'up{job="api", instance="a:9100"}'
values: '1 1 0 0 0 0 0 0'
alert_rule_test:
- eval_time: 7m
alertname: TargetDown
exp_alerts:
- exp_labels:
severity: warning
job: api
instance: a:9100
exp_annotations:
summary: "api target a:9100 is down"
26. What does the values notation '1 1 0x5' mean in tests?
Expanding notation: 0x5 repeats 0 for five more steps after the first. a+bxn grows by b each step. _ means a missing sample and stale writes a stale marker.
Senior: SLO alerts and alert hygiene
27. What is a burn rate?
How fast you consume the error budget relative to the SLO period. A burn rate of 1 uses exactly the whole budget over the period; 14.4 uses it 14.4 times faster. For a 99.9% SLO, the budget is a 0.1% error rate.
28. Give the multiwindow, multi-burn-rate parameters for a 99.9% SLO.
| Severity | Long window | Short window | Burn rate | Budget consumed |
|---|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 | 2% |
| Page | 6 hours | 30 minutes | 6 | 5% |
| Ticket | 3 days | 6 hours | 1 | 10% |
These come from Table 5-8 of the SRE Workbook's "Alerting on SLOs" chapter. Budget consumed is burn rate times window divided by the 30-day period: 14.4 x 1h / 720h = 2%.
29. Why both a long and a short window?
The long window confirms the burn is significant. The short window confirms it is still happening, so the alert resets quickly once the incident is fixed instead of firing for another hour.
30. Write the fast-burn page rule.
- alert: ErrorBudgetFastBurn
expr: |
job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
and
job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001)
labels:
severity: page
The two ratio series are recording rules. Tools such as Sloth and Pyrra generate this set from an SLO definition.
31. How do you find noisy alerts?
Query the ALERTS series over 30 days and count how often each alert fired, then compare with incident records. Alerts that fire often and lead to no action should be retuned, demoted to tickets or deleted.
32. How do you alert on Prometheus itself failing?
Run a second Prometheus, or an external check, that alerts on the first one's up, on rule evaluation failures and on failed notifications to Alertmanager. Also send a watchdog alert that always fires, so its absence downstream tells you the pipeline is broken.
33. What is a watchdog or dead man's switch alert?
An alert with expr: vector(1) that fires continuously. Alertmanager routes it to an external service that pages if the heartbeat stops. It proves the whole chain from rule evaluation to receiver is working.
34. How do you handle alerts across HA Prometheus pairs?
Both replicas evaluate the same rules and send to the same Alertmanager cluster, which deduplicates identical alerts. Remove replica-specific external labels with alert_relabel_configs so the alerts match.
35. What does the limit field do for alerting rules?
It caps how many alerts one rule may produce. If exceeded, all alerts for that rule, whether active, pending or inactive, are cleared and the event counts as an evaluation error. It protects against a rule accidentally generating thousands of alerts.
36. How do you keep alert rules reviewable at scale?
Store them in Git, one file per service, require a runbook_url and an owner label, run promtool check rules and unit tests in CI, and lint label conventions. In Kubernetes, ship them as PrometheusRule objects with the service's Helm chart.
Scenario questions
37. An alert fires every night at 2 a.m. and resolves by itself. What do you do?
Check what runs at 2 a.m. (backups, batch jobs, log rotation). If users are not affected, the alert measures the wrong thing: switch it to a symptom, lengthen for, or exclude the known window. Do not just silence it forever.
38. Users report an outage, but no alert fired.
Check whether the series existed (target dropped from discovery means up == 0 returns nothing), whether the rule evaluated without errors, whether it was pending because of a long for, and whether Alertmanager received and routed it. Add an absent() rule and a test for the gap.
39. One deploy causes 300 pages, one per pod.
Aggregate the expression to the service level with sum by (service), and make sure Alertmanager groups by alertname and service. Per-pod alerts belong on dashboards.
40. An alert keeps resolving and refiring every few minutes.
The value hovers near the threshold. Add keep_firing_for, use a longer rate window to smooth the signal, or set separate fire and clear behaviour with a slightly different threshold in a second rule.
41. The team wants to page on CPU above 80%.
Ask what user impact high CPU causes. If latency or errors rise, page on those. Keep the CPU rule as a ticket or capacity warning, perhaps with predict_linear on saturation.
42. A new SLO alert fires during every deploy.
Deploys are spending budget, which is real information. Check rollout settings such as readiness probes and surge, then decide whether the deploy process needs fixing. Muting during deploys hides the cost. For more incident drills see scenario-based Prometheus questions.
Preparing for an alerting round
Write five rules for a sample service (availability, errors, latency, saturation, burn rate), write a unit test for each, and run them with promtool. Then route them in a local Alertmanager. The Prometheus and Grafana course covers alert rules and Alertmanager in labs, in our Kharadi, Pune classroom and live online.
Frequently asked questions
What is the for clause in a Prometheus alert?
It sets how long the condition must stay true before the alert moves from pending to firing.
What is keep_firing_for?
An optional field that keeps an alert firing for a set time after its condition was last met, which reduces flapping.
What is the difference between Prometheus alerting rules and Alertmanager?
Rules in Prometheus decide when an alert is active. Alertmanager groups, deduplicates, silences and routes those alerts to receivers.
How do I test Prometheus alert rules?
Write a test file with input series and expected alerts, then run promtool test rules.
What burn rates are used for SLO alerts?
The SRE Workbook recommends 14.4 over 1h/5m and 6 over 6h/30m for pages, and 1 over 3d/6h for tickets, for a 99.9% SLO.
How do I alert when a metric stops reporting?
Use absent() on the series, because a comparison on a missing series returns nothing.
Which template variables are available in annotations?
$labels, $value and $externalLabels.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0