Prometheus Engineer Interview Questions: PromQL Tasks (2027)

48 Prometheus engineer interview tasks with answers: PromQL queries, relabeling, recording and alerting rules, promtool tests, Alertmanager routing.

Sep 13, 2025 - 14:10
Nov 8, 2026 - 20:50
 0  131
Prometheus Engineer Interview Questions: PromQL Tasks (2027)

Prometheus engineer interviews usually include a practical round: write a PromQL query on a whiteboard, fix a scrape config, add a relabeling rule, turn a slow query into a recording rule, or write an alert with the right for duration and test it with promtool. This page has 48 task-style questions, each with the query or YAML you would write and why.

Think of it as the hands-on companion to our 80+ Prometheus interview questions and answers, which explains the concepts, and our scenario-based Prometheus questions, which cover troubleshooting under pressure. Syntax and commands were checked against prometheus.io documentation in November 2026.

Key takeaways

  • Engineer-level rounds test writing, not defining. Practise typing PromQL and YAML from memory on a local Prometheus with node_exporter.
  • Four query patterns cover most tasks: rate() on counters, sum by aggregation, ratios for error rates, and histogram_quantile() over bucket rates.
  • Always rate() first, then sum(). Summing raw counters before rate breaks counter-reset handling.
  • Relabeling is where configs go wrong. Know the difference between relabel_configs (before the scrape, on targets) and metric_relabel_configs (after the scrape, on samples).
  • Validate everything with promtool check config, promtool check rules and promtool test rules before you reload.
  • Default ports to remember: 9090 for Prometheus, 9093 for Alertmanager, 9100 for node_exporter.

Set up a practice environment

Run Prometheus and node_exporter locally (binaries or containers), point Prometheus at node_exporter on port 9100, and open the expression browser on port 9090. Every query below works against node_exporter metrics or a simple instrumented app exposing http_requests_total and http_request_duration_seconds.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]
  - job_name: app
    static_configs:
      - targets: ["localhost:8080"]

Prometheus engineer questions for freshers

1. Write a query that shows which targets are down.

up == 0

Prometheus records up for every scrape: 1 if the scrape succeeded, 0 if it failed. It is the first query to run when metrics go missing.

2. Show the per-second request rate over the last five minutes.

rate(http_requests_total[5m])

rate() calculates the per-second average increase of a counter over the range and handles counter resets after restarts.

3. Show total requests per second for each service, ignoring instance.

sum by (service) (rate(http_requests_total[5m]))

4. What is the difference between rate() and irate()?

rate() averages over the whole range and suits alerts and dashboards. irate() uses only the last two samples, so it reacts faster but is noisy; use it for zoomed-in graphs of fast-moving counters.

5. How many requests did the service handle in the last hour?

sum(increase(http_requests_total[1h]))

increase() is rate() multiplied by the range in seconds. Results can be non-integers because of extrapolation at the range edges.

6. Show memory used as a percentage on each Linux node.

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

7. Show CPU usage percentage per instance from node_exporter.

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

Idle time per second averaged across CPUs gives the idle fraction; one minus that is usage.

8. Show filesystems with less than 10% free space.

node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.10

9. Filter a metric using a regular expression on a label.

http_requests_total{status=~"5.."}

=~ matches a regex, !~ excludes one. Prometheus regexes are fully anchored, so "5.." matches only three-character values starting with 5.

10. What is the difference between an instant vector and a range vector?

An instant vector has one sample per series at a point in time, for example http_requests_total. A range vector has a window of samples per series, for example http_requests_total[5m], and is what functions like rate() take as input. You cannot graph a range vector directly.

11. Show the top five pods by CPU usage.

topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m])))

This uses cAdvisor metrics exposed through the kubelet. container!="" drops the pod-level aggregate series.

12. Compare current traffic with the same time a week ago.

sum(rate(http_requests_total[5m]))
  / sum(rate(http_requests_total[5m] offset 1w))

13. How do you check a config file before reloading Prometheus?

promtool check config prometheus.yml

It validates the main file and any rule files it references. Then reload with a SIGHUP or, if Prometheus was started with --web.enable-lifecycle, with curl -X POST http://localhost:9090/-/reload.

14. Add a static label to every series from one job.

- job_name: app
  static_configs:
    - targets: ["app1:8080", "app2:8080"]
      labels:
        env: prod
        team: payments

Prometheus engineer questions for 2 to 5 years of experience

15. Write an error-rate query as a percentage of all requests, per service.

100 *
  sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
  sum by (service) (rate(http_requests_total[5m]))

Both sides aggregate by the same label, so they match one-to-one.

16. Calculate the 95th percentile latency from a histogram, per service.

histogram_quantile(0.95,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

Keep the le label in the aggregation; without it the quantile cannot be computed. The result is an estimate whose accuracy depends on bucket boundaries.

17. Why can you aggregate histograms across instances but not summaries?

Histogram buckets are counters that can be summed and then turned into a quantile. A summary's quantiles are calculated inside each process, and averaging percentiles from several instances does not give a correct overall percentile.

18. Why is rate(sum(...)) wrong?

Summing raw counters first hides each counter's resets, so a restart on one instance looks like a huge drop and rate() misreads it. Apply rate() to each series, then sum(). PromQL will not even accept rate() on the output of sum() without a subquery, which is a hint.

19. Join a metric with labels from an info metric.

rate(http_requests_total[5m])
  * on (instance) group_left (version)
    app_build_info

on (instance) picks the matching label; group_left (version) copies version from the one-per-instance info series onto each request series. Info metrics have value 1, so multiplying leaves values unchanged.

20. Find series that stopped reporting.

absent(up{job="payments"})

absent() returns 1 when no series match, which is useful for alerting when a whole job disappears from service discovery. For a single series that stopped updating, absent_over_time(metric[10m]) works over a window.

21. Predict when a disk will fill up.

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

It fits a linear regression over the last six hours and returns series whose free space is projected to drop below zero within four hours.

22. Write a recording rule for a request rate used on many dashboards.

groups:
  - name: http_recording
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))

The name follows the level:metric:operations convention from the Prometheus docs. Dashboards then query the cheap precomputed series.

23. Write an alerting rule for a high error rate.

groups:
  - name: http_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.service }} error rate above 5%"
          description: "Current value: {{ $value | humanizePercentage }}"

for: 10m keeps the alert pending until the condition has held for ten minutes, which filters short spikes.

24. How do you unit test an alerting rule?

# tests.yml
rule_files:
  - rules/http_alerts.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="api",status="500"}'
        values: '0+10x20'
      - series: 'http_requests_total{service="api",status="200"}'
        values: '0+90x20'
    alert_rule_test:
      - eval_time: 15m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              service: api
promtool test rules tests.yml

Ten errors out of a hundred requests per minute is a 10% error rate, so the alert should be firing at 15 minutes.

25. Keep only targets with a specific Kubernetes annotation.

- job_name: kubernetes-pods
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: "true"
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod

The prometheus.io/scrape annotation is a common convention, not something Prometheus itself defines. Labels starting with __meta_ are dropped after relabeling unless copied to a normal label.

26. Drop a high-cardinality metric you never use.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_gc_duration_seconds.*"
    action: drop

This runs after the scrape, on samples, so it saves storage but not scrape bandwidth. Fixing the exporter is better where possible.

27. Remove a label that causes cardinality explosions.

metric_relabel_configs:
  - regex: "request_id"
    action: labeldrop

Only drop a label if the remaining labels still uniquely identify each series; otherwise you create duplicate series that collide.

28. How do you find which metrics have the most series?

topk(10, count by (__name__) ({__name__=~".+"}))

This query is expensive on a large server. The TSDB status page in the web UI shows the same top-cardinality information more cheaply.

29. Configure Alertmanager to send critical alerts to one receiver and everything else to another.

route:
  receiver: team-email
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "page"
      receiver: oncall-pager

receivers:
  - name: team-email
    email_configs:
      - to: "[email protected]"
  - name: oncall-pager
    webhook_configs:
      - url: "http://pager-gateway.internal/alert"

Check it with amtool check-config alertmanager.yml.

30. Silence pages for a service during planned maintenance.

amtool silence add service="payments" --duration=2h \
  --comment="DB upgrade CHG-1234" --alertmanager.url=http://localhost:9093

31. Suppress warning alerts when a critical alert for the same service is firing.

inhibit_rules:
  - source_matchers: [severity = "page"]
    target_matchers: [severity = "warning"]
    equal: [service]

32. Monitor a website's availability from outside.

Use the Blackbox exporter with the multi-target pattern: the target URL becomes a parameter, and relabeling points the scrape at the exporter.

- job_name: blackbox-http
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets: ["https://www.example.com"]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

Alert on probe_success == 0.

33. How do you expose metrics from a batch job?

Push them to a Pushgateway at the end of the job, including a last-success timestamp, and scrape the Pushgateway with honor_labels: true. Alert when time() - job_last_success_unixtime exceeds the expected interval. Do not use the Pushgateway for long-running services.

Senior Prometheus engineer questions

34. Write a multi-window burn-rate alert for a 99.9% availability SLO.

- alert: ErrorBudgetBurnFast
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page

A burn rate of 14.4 spends about 2% of a 30-day error budget in one hour, the pattern described in Google's SRE Workbook. The short window makes the alert resolve quickly after recovery. In production you would build this on recording rules.

35. How do you send data to long-term storage?

remote_write:
  - url: "https://metrics-store.internal/api/v1/push"
    queue_config:
      max_samples_per_send: 2000
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "debug_.*"
        action: drop

Systems such as Thanos, Cortex, Grafana Mimir and VictoriaMetrics accept remote write. Watch the prometheus_remote_storage_* metrics for falling behind or dropped samples.

36. How would you run Prometheus for high availability?

Run two identical Prometheus servers scraping the same targets, each with a distinct external label such as replica, and point both at an Alertmanager cluster, which deduplicates notifications. For a global view and deduplicated queries, add a layer such as Thanos or a remote-write backend.

37. What is Prometheus agent mode?

A mode started with --agent that scrapes and forwards data through remote write but disables local querying, alerting and long-term local storage. It suits edge clusters that send everything to a central store.

38. How do you set retention?

With --storage.tsdb.retention.time (for example 30d) and optionally --storage.tsdb.retention.size. Whichever limit is reached first triggers deletion of the oldest blocks.

39. How do you find slow or expensive queries?

Enable the query log with query_log_file in the global config, check the prometheus_engine_query_duration_seconds metric, and look for dashboards that query raw high-cardinality series over long ranges. Replace them with recording rules.

40. What changed in Prometheus 3.0 that engineers should know?

Prometheus 3.0, released in November 2024, introduced a new web UI, support for UTF-8 metric and label names, OTLP ingestion, and Remote Write 2.0, and removed some deprecated feature flags. Check the migration guide before upgrading old configurations.

41. How do you secure the Prometheus web endpoints?

Prometheus supports TLS and basic authentication through a web configuration file passed with --web.config.file. Also restrict network access, avoid exposing --web.enable-lifecycle and --web.enable-admin-api publicly, and put a reverse proxy with single sign-on in front of the UI where needed.

Practical scenario questions

42. Your error-rate query returns no data even though errors are happening. What do you check?

Whether label values match exactly (status "500" versus "5xx"), whether both sides of the division share the same labels after aggregation, and whether the counter exists with that name. Run each half of the query alone first.

43. A histogram_quantile result is flat at the highest bucket boundary. Why?

Most requests are slower than the largest finite bucket, so the estimate is capped. Add higher buckets in the instrumentation, or consider native histograms if your client libraries and Prometheus version support them.

44. After adding a user_id label, Prometheus memory doubled. What do you do?

Remove the label at the source, because each user creates new series. As a stopgap, use labeldrop in metric_relabel_configs or drop the metric, and set sample_limit on the job to protect the server.

45. Relabeling should rename the namespace label, but the new label never appears. Why?

The source label was probably a __meta_ label referenced in metric_relabel_configs, where meta labels no longer exist. Move the rule to relabel_configs.

46. An alert works in promtool tests but never fires in production. What do you check?

That the rule file is listed in rule_files and loaded (Status, Rules page), that the label values in production match the test series, that for is not longer than the incidents last, and that Alertmanager routing does not send it to a receiver nobody watches.

47. Two Prometheus replicas send duplicate notifications. How do you fix it?

Point both replicas at every member of the Alertmanager cluster rather than one each, and make sure the Alertmanagers are clustered with the --cluster.peer flags so they share notification state.

48. A dashboard panel takes 30 seconds to load. How do you speed it up?

Find the query, check the number of series it touches, add a recording rule for the aggregated result, shorten the default time range, and set a sensible minimum step so Grafana does not request thousands of points.

Quick reference

TaskTool or function
Rate of a counterrate(), increase(), irate()
Percentileshistogram_quantile() over sum by (le)
Missing dataup, absent(), absent_over_time()
Forecastpredict_linear()
Joinson, ignoring, group_left, group_right
Validate config and rulespromtool check config, promtool check rules
Unit test rulespromtool test rules
Validate Alertmanageramtool check-config

Our guide to using Prometheus and Grafana together shows how these queries become dashboards, and top observability tools for DevOps places Prometheus among alternatives. For instructor-led labs, see our Prometheus and Grafana training in Kharadi, Pune and live online.

Frequently asked questions

What is asked in a Prometheus engineer practical round?

Writing PromQL for rates, error ratios and percentiles, fixing scrape and relabel configs, writing recording and alerting rules, testing them with promtool and configuring Alertmanager routing.

Why apply rate() before sum() in PromQL?

rate() handles counter resets per series. Summing raw counters first hides those resets and gives wrong results after any restart.

What is the difference between relabel_configs and metric_relabel_configs?

relabel_configs runs on targets before the scrape and can use __meta_ labels from service discovery. metric_relabel_configs runs on scraped samples and is used to drop or rewrite metrics and labels.

How do I test Prometheus alert rules?

Write a test file with input series and expected alerts, then run promtool test rules. Check syntax first with promtool check rules.

What ports do Prometheus components use by default?

Prometheus uses 9090, Alertmanager 9093 and node_exporter 9100. The Blackbox exporter defaults to 9115 and the Pushgateway to 9091.

When should I use a recording rule?

When a query is expensive or used by many dashboards and alerts. The rule precomputes the result on a schedule so later queries read a small, cheap series.

Is there a Prometheus certification?

Yes, the Linux Foundation and CNCF offer the Prometheus Certified Associate (PCA). Check the Linux Foundation Training site for the current curriculum and price.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.