Prometheus Engineer Interview Questions: PromQL Tasks (2027)
48 Prometheus engineer interview tasks with answers: PromQL queries, relabeling, recording and alerting rules, promtool tests, Alertmanager routing.
Prometheus engineer interviews usually include a practical round: write a PromQL query on a whiteboard, fix a scrape config, add a relabeling rule, turn a slow query into a recording rule, or write an alert with the right for duration and test it with promtool. This page has 48 task-style questions, each with the query or YAML you would write and why.
Think of it as the hands-on companion to our 80+ Prometheus interview questions and answers, which explains the concepts, and our scenario-based Prometheus questions, which cover troubleshooting under pressure. Syntax and commands were checked against prometheus.io documentation in November 2026.
Key takeaways
- Engineer-level rounds test writing, not defining. Practise typing PromQL and YAML from memory on a local Prometheus with node_exporter.
- Four query patterns cover most tasks:
rate()on counters,sum byaggregation, ratios for error rates, andhistogram_quantile()over bucket rates. - Always
rate()first, thensum(). Summing raw counters before rate breaks counter-reset handling. - Relabeling is where configs go wrong. Know the difference between
relabel_configs(before the scrape, on targets) andmetric_relabel_configs(after the scrape, on samples). - Validate everything with
promtool check config,promtool check rulesandpromtool test rulesbefore you reload. - Default ports to remember: 9090 for Prometheus, 9093 for Alertmanager, 9100 for node_exporter.
Set up a practice environment
Run Prometheus and node_exporter locally (binaries or containers), point Prometheus at node_exporter on port 9100, and open the expression browser on port 9090. Every query below works against node_exporter metrics or a simple instrumented app exposing http_requests_total and http_request_duration_seconds.
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "rules/*.yml"
scrape_configs:
- job_name: node
static_configs:
- targets: ["localhost:9100"]
- job_name: app
static_configs:
- targets: ["localhost:8080"]
Prometheus engineer questions for freshers
1. Write a query that shows which targets are down.
up == 0
Prometheus records up for every scrape: 1 if the scrape succeeded, 0 if it failed. It is the first query to run when metrics go missing.
2. Show the per-second request rate over the last five minutes.
rate(http_requests_total[5m])
rate() calculates the per-second average increase of a counter over the range and handles counter resets after restarts.
3. Show total requests per second for each service, ignoring instance.
sum by (service) (rate(http_requests_total[5m]))
4. What is the difference between rate() and irate()?
rate() averages over the whole range and suits alerts and dashboards. irate() uses only the last two samples, so it reacts faster but is noisy; use it for zoomed-in graphs of fast-moving counters.
5. How many requests did the service handle in the last hour?
sum(increase(http_requests_total[1h]))
increase() is rate() multiplied by the range in seconds. Results can be non-integers because of extrapolation at the range edges.
6. Show memory used as a percentage on each Linux node.
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
7. Show CPU usage percentage per instance from node_exporter.
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
Idle time per second averaged across CPUs gives the idle fraction; one minus that is usage.
8. Show filesystems with less than 10% free space.
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.10
9. Filter a metric using a regular expression on a label.
http_requests_total{status=~"5.."}
=~ matches a regex, !~ excludes one. Prometheus regexes are fully anchored, so "5.." matches only three-character values starting with 5.
10. What is the difference between an instant vector and a range vector?
An instant vector has one sample per series at a point in time, for example http_requests_total. A range vector has a window of samples per series, for example http_requests_total[5m], and is what functions like rate() take as input. You cannot graph a range vector directly.
11. Show the top five pods by CPU usage.
topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m])))
This uses cAdvisor metrics exposed through the kubelet. container!="" drops the pod-level aggregate series.
12. Compare current traffic with the same time a week ago.
sum(rate(http_requests_total[5m]))
/ sum(rate(http_requests_total[5m] offset 1w))
13. How do you check a config file before reloading Prometheus?
promtool check config prometheus.yml
It validates the main file and any rule files it references. Then reload with a SIGHUP or, if Prometheus was started with --web.enable-lifecycle, with curl -X POST http://localhost:9090/-/reload.
14. Add a static label to every series from one job.
- job_name: app
static_configs:
- targets: ["app1:8080", "app2:8080"]
labels:
env: prod
team: payments
Prometheus engineer questions for 2 to 5 years of experience
15. Write an error-rate query as a percentage of all requests, per service.
100 *
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
Both sides aggregate by the same label, so they match one-to-one.
16. Calculate the 95th percentile latency from a histogram, per service.
histogram_quantile(0.95,
sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))
Keep the le label in the aggregation; without it the quantile cannot be computed. The result is an estimate whose accuracy depends on bucket boundaries.
17. Why can you aggregate histograms across instances but not summaries?
Histogram buckets are counters that can be summed and then turned into a quantile. A summary's quantiles are calculated inside each process, and averaging percentiles from several instances does not give a correct overall percentile.
18. Why is rate(sum(...)) wrong?
Summing raw counters first hides each counter's resets, so a restart on one instance looks like a huge drop and rate() misreads it. Apply rate() to each series, then sum(). PromQL will not even accept rate() on the output of sum() without a subquery, which is a hint.
19. Join a metric with labels from an info metric.
rate(http_requests_total[5m])
* on (instance) group_left (version)
app_build_info
on (instance) picks the matching label; group_left (version) copies version from the one-per-instance info series onto each request series. Info metrics have value 1, so multiplying leaves values unchanged.
20. Find series that stopped reporting.
absent(up{job="payments"})
absent() returns 1 when no series match, which is useful for alerting when a whole job disappears from service discovery. For a single series that stopped updating, absent_over_time(metric[10m]) works over a window.
21. Predict when a disk will fill up.
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0
It fits a linear regression over the last six hours and returns series whose free space is projected to drop below zero within four hours.
22. Write a recording rule for a request rate used on many dashboards.
groups:
- name: http_recording
interval: 30s
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
The name follows the level:metric:operations convention from the Prometheus docs. Dashboards then query the cheap precomputed series.
23. Write an alerting rule for a high error rate.
groups:
- name: http_alerts
rules:
- alert: HighErrorRate
expr: |
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "{{ $labels.service }} error rate above 5%"
description: "Current value: {{ $value | humanizePercentage }}"
for: 10m keeps the alert pending until the condition has held for ten minutes, which filters short spikes.
24. How do you unit test an alerting rule?
# tests.yml
rule_files:
- rules/http_alerts.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'http_requests_total{service="api",status="500"}'
values: '0+10x20'
- series: 'http_requests_total{service="api",status="200"}'
values: '0+90x20'
alert_rule_test:
- eval_time: 15m
alertname: HighErrorRate
exp_alerts:
- exp_labels:
severity: page
service: api
promtool test rules tests.yml
Ten errors out of a hundred requests per minute is a 10% error rate, so the alert should be firing at 15 minutes.
25. Keep only targets with a specific Kubernetes annotation.
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
The prometheus.io/scrape annotation is a common convention, not something Prometheus itself defines. Labels starting with __meta_ are dropped after relabeling unless copied to a normal label.
26. Drop a high-cardinality metric you never use.
metric_relabel_configs:
- source_labels: [__name__]
regex: "go_gc_duration_seconds.*"
action: drop
This runs after the scrape, on samples, so it saves storage but not scrape bandwidth. Fixing the exporter is better where possible.
27. Remove a label that causes cardinality explosions.
metric_relabel_configs:
- regex: "request_id"
action: labeldrop
Only drop a label if the remaining labels still uniquely identify each series; otherwise you create duplicate series that collide.
28. How do you find which metrics have the most series?
topk(10, count by (__name__) ({__name__=~".+"}))
This query is expensive on a large server. The TSDB status page in the web UI shows the same top-cardinality information more cheaply.
29. Configure Alertmanager to send critical alerts to one receiver and everything else to another.
route:
receiver: team-email
group_by: [alertname, service]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity = "page"
receiver: oncall-pager
receivers:
- name: team-email
email_configs:
- to: "[email protected]"
- name: oncall-pager
webhook_configs:
- url: "http://pager-gateway.internal/alert"
Check it with amtool check-config alertmanager.yml.
30. Silence pages for a service during planned maintenance.
amtool silence add service="payments" --duration=2h \
--comment="DB upgrade CHG-1234" --alertmanager.url=http://localhost:9093
31. Suppress warning alerts when a critical alert for the same service is firing.
inhibit_rules:
- source_matchers: [severity = "page"]
target_matchers: [severity = "warning"]
equal: [service]
32. Monitor a website's availability from outside.
Use the Blackbox exporter with the multi-target pattern: the target URL becomes a parameter, and relabeling points the scrape at the exporter.
- job_name: blackbox-http
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets: ["https://www.example.com"]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-exporter:9115
Alert on probe_success == 0.
33. How do you expose metrics from a batch job?
Push them to a Pushgateway at the end of the job, including a last-success timestamp, and scrape the Pushgateway with honor_labels: true. Alert when time() - job_last_success_unixtime exceeds the expected interval. Do not use the Pushgateway for long-running services.
Senior Prometheus engineer questions
34. Write a multi-window burn-rate alert for a 99.9% availability SLO.
- alert: ErrorBudgetBurnFast
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: page
A burn rate of 14.4 spends about 2% of a 30-day error budget in one hour, the pattern described in Google's SRE Workbook. The short window makes the alert resolve quickly after recovery. In production you would build this on recording rules.
35. How do you send data to long-term storage?
remote_write:
- url: "https://metrics-store.internal/api/v1/push"
queue_config:
max_samples_per_send: 2000
write_relabel_configs:
- source_labels: [__name__]
regex: "debug_.*"
action: drop
Systems such as Thanos, Cortex, Grafana Mimir and VictoriaMetrics accept remote write. Watch the prometheus_remote_storage_* metrics for falling behind or dropped samples.
36. How would you run Prometheus for high availability?
Run two identical Prometheus servers scraping the same targets, each with a distinct external label such as replica, and point both at an Alertmanager cluster, which deduplicates notifications. For a global view and deduplicated queries, add a layer such as Thanos or a remote-write backend.
37. What is Prometheus agent mode?
A mode started with --agent that scrapes and forwards data through remote write but disables local querying, alerting and long-term local storage. It suits edge clusters that send everything to a central store.
38. How do you set retention?
With --storage.tsdb.retention.time (for example 30d) and optionally --storage.tsdb.retention.size. Whichever limit is reached first triggers deletion of the oldest blocks.
39. How do you find slow or expensive queries?
Enable the query log with query_log_file in the global config, check the prometheus_engine_query_duration_seconds metric, and look for dashboards that query raw high-cardinality series over long ranges. Replace them with recording rules.
40. What changed in Prometheus 3.0 that engineers should know?
Prometheus 3.0, released in November 2024, introduced a new web UI, support for UTF-8 metric and label names, OTLP ingestion, and Remote Write 2.0, and removed some deprecated feature flags. Check the migration guide before upgrading old configurations.
41. How do you secure the Prometheus web endpoints?
Prometheus supports TLS and basic authentication through a web configuration file passed with --web.config.file. Also restrict network access, avoid exposing --web.enable-lifecycle and --web.enable-admin-api publicly, and put a reverse proxy with single sign-on in front of the UI where needed.
Practical scenario questions
42. Your error-rate query returns no data even though errors are happening. What do you check?
Whether label values match exactly (status "500" versus "5xx"), whether both sides of the division share the same labels after aggregation, and whether the counter exists with that name. Run each half of the query alone first.
43. A histogram_quantile result is flat at the highest bucket boundary. Why?
Most requests are slower than the largest finite bucket, so the estimate is capped. Add higher buckets in the instrumentation, or consider native histograms if your client libraries and Prometheus version support them.
44. After adding a user_id label, Prometheus memory doubled. What do you do?
Remove the label at the source, because each user creates new series. As a stopgap, use labeldrop in metric_relabel_configs or drop the metric, and set sample_limit on the job to protect the server.
45. Relabeling should rename the namespace label, but the new label never appears. Why?
The source label was probably a __meta_ label referenced in metric_relabel_configs, where meta labels no longer exist. Move the rule to relabel_configs.
46. An alert works in promtool tests but never fires in production. What do you check?
That the rule file is listed in rule_files and loaded (Status, Rules page), that the label values in production match the test series, that for is not longer than the incidents last, and that Alertmanager routing does not send it to a receiver nobody watches.
47. Two Prometheus replicas send duplicate notifications. How do you fix it?
Point both replicas at every member of the Alertmanager cluster rather than one each, and make sure the Alertmanagers are clustered with the --cluster.peer flags so they share notification state.
48. A dashboard panel takes 30 seconds to load. How do you speed it up?
Find the query, check the number of series it touches, add a recording rule for the aggregated result, shorten the default time range, and set a sensible minimum step so Grafana does not request thousands of points.
Quick reference
| Task | Tool or function |
|---|---|
| Rate of a counter | rate(), increase(), irate() |
| Percentiles | histogram_quantile() over sum by (le) |
| Missing data | up, absent(), absent_over_time() |
| Forecast | predict_linear() |
| Joins | on, ignoring, group_left, group_right |
| Validate config and rules | promtool check config, promtool check rules |
| Unit test rules | promtool test rules |
| Validate Alertmanager | amtool check-config |
Our guide to using Prometheus and Grafana together shows how these queries become dashboards, and top observability tools for DevOps places Prometheus among alternatives. For instructor-led labs, see our Prometheus and Grafana training in Kharadi, Pune and live online.
Frequently asked questions
What is asked in a Prometheus engineer practical round?
Writing PromQL for rates, error ratios and percentiles, fixing scrape and relabel configs, writing recording and alerting rules, testing them with promtool and configuring Alertmanager routing.
Why apply rate() before sum() in PromQL?
rate() handles counter resets per series. Summing raw counters first hides those resets and gives wrong results after any restart.
What is the difference between relabel_configs and metric_relabel_configs?
relabel_configs runs on targets before the scrape and can use __meta_ labels from service discovery. metric_relabel_configs runs on scraped samples and is used to drop or rewrite metrics and labels.
How do I test Prometheus alert rules?
Write a test file with input series and expected alerts, then run promtool test rules. Check syntax first with promtool check rules.
What ports do Prometheus components use by default?
Prometheus uses 9090, Alertmanager 9093 and node_exporter 9100. The Blackbox exporter defaults to 9115 and the Pushgateway to 9091.
When should I use a recording rule?
When a query is expensive or used by many dashboards and alerts. The rule precomputes the result on a schedule so later queries read a small, cheap series.
Is there a Prometheus certification?
Yes, the Linux Foundation and CNCF offer the Prometheus Certified Associate (PCA). Check the Linux Foundation Training site for the current curriculum and price.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0