Real-Time Prometheus Interview Questions (2027)

40 real-time Prometheus interview questions: scraping, PromQL error rates and p95, Alertmanager routing, retention, Kubernetes and incident scenarios.

Sep 13, 2025 - 14:18
Nov 14, 2026 - 18:05
 0  56
Real-Time Prometheus Interview Questions (2027)

Real-time Prometheus interview questions test whether you can run Prometheus in production: scraping and service discovery, PromQL for error rates and latency, alerting rules and Alertmanager routing, retention and capacity, and Kubernetes integration. This page has 40 practical questions with short answers, grouped by topic and finished with incident scenarios.

Facts were checked against the Prometheus and Alertmanager documentation in November 2026. The newest release at that time was Prometheus 3.16.0 (4 November 2026), and 3.13 was the current long-term support release, supported until 31 July 2027. For deeper internals such as the TSDB head and compaction, read our advanced Prometheus interview questions. For incident drills, see the scenario-based Prometheus questions.

Key takeaways

  • Prometheus pulls metrics over HTTP. The global defaults are a 1 minute scrape interval, a 10 second scrape timeout and a 1 minute rule evaluation interval.
  • Local storage keeps 15 days by default. Retention can be limited by time, by size, or both, and local storage is neither clustered nor replicated.
  • Use rate() on counters, histogram_quantile() on histogram buckets, and recording rules for expressions that dashboards run repeatedly.
  • An alert moves from pending to firing only after its for duration passes. keep_firing_for stops flapping alerts from resolving too early.
  • Alertmanager defaults are group_wait 30s, group_interval 5m and repeat_interval 4h.
  • Prometheus 3 changed behaviour you may be asked about: strict scrape Content-Type handling, UTF-8 metric names, normalised le labels and removal of the Alertmanager v1 API.

How the questions are grouped

SectionQuestionsWhat an interviewer is checking
Setup, scraping and storage1 to 14Can you install it, configure targets and size the disk
PromQL in production15 to 26Can you write queries for rates, ratios and latency
Alerting27 to 32Can you build alerts that page for the right reasons
Kubernetes and scale33 to 36Can you discover pods and scale beyond one server
Incident scenarios37 to 40Can you reason from symptom to cause to fix

Setup, scraping and storage

1. How does Prometheus collect metrics?

It pulls them. Prometheus sends an HTTP request to each target's metrics endpoint (path /metrics by default) at every scrape interval and stores the samples it receives. Targets come from static lists or from service discovery. Because Prometheus initiates the request, a failed scrape is itself a signal: it records an up series with value 0 for that target.

2. What does a minimal configuration look like?

A global section plus one scrape job. The defaults below match the documentation, so you only need to write them when you want different values.

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m

rule_files:
  - alerts.yml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]

3. How do the scrape interval and timeout relate?

The timeout cannot be greater than the interval. If a target takes longer to respond than the timeout, the scrape fails and up is 0. Set both per job when a slow exporter needs more time than the global value allows.

4. Which Prometheus release should you run?

For stability, run an LTS release. The project starts a new minor release cycle every six weeks, and an LTS release is supported for one year with selected high-severity fixes. At the time of writing, 3.13 was the LTS release, supported until 31 July 2027, and the 3.5 LTS line had already reached end of life in July 2026.

5. A target shows DOWN on the Targets page. How do you debug it?

Read the error text next to the target first, because it usually names the cause. Then check in order:

  • Can the Prometheus host reach the target address and port (firewall, security group, network policy)?
  • Does the endpoint return metrics within the scrape timeout?
  • Is the scheme right (http versus https) and is the path correct?
  • On Prometheus 3, does the response carry a recognised Content-Type header? Scrapes fail when it is missing or unknown unless you set fallback_scrape_protocol.

6. What is the default retention and how do you change it?

The default is 15 days. It applies only when neither retention flag is set. Use --storage.tsdb.retention.time (for example 30d) and --storage.tsdb.retention.size (disabled by default). If both are set, whichever limit is reached first applies.

7. How do you estimate disk space?

Use the formula from the documentation: retention in seconds multiplied by ingested samples per second multiplied by bytes per sample. Prometheus averages roughly 1 to 2 bytes per sample after compression. The documentation also advises keeping the size limit at 80 to 85 percent of the disk, so compaction has room to work.

8. How do you reload configuration without a restart?

Send SIGHUP to the process, or call POST /-/reload if you started Prometheus with --web.enable-lifecycle. The HTTP endpoint is disabled by default. Validate first with promtool check config prometheus.yml, because an invalid file is rejected and the old configuration keeps running.

9. Which endpoints do load balancers and probes use?

/-/healthy always returns 200 while the process is up. /-/ready returns 200 when Prometheus is ready to serve queries. Use readiness for Kubernetes probes and load balancer checks.

10. Is Prometheus storage highly available?

No. Local storage is not clustered or replicated, and network filesystems such as NFS (including AWS EFS) are not supported for it. The usual pattern is to run two identical Prometheus servers scraping the same targets and use remote write to a long-term store that can deduplicate.

11. When is the Pushgateway the right tool?

Only for service-level batch jobs that are not tied to one machine, such as a nightly cleanup of users. The documentation lists the drawbacks: it becomes a single point of failure, you lose the automatic up metric, and it never forgets pushed series until you delete them through its API. For jobs tied to a host, the Node Exporter textfile collector is the suggested alternative.

12. What is federation and when would you use it?

One Prometheus scrapes selected series from another server's /federate endpoint. Use it for hierarchical setups, where each data centre keeps detailed data and a global server keeps aggregates, or to pull a few metrics from another team's server. You must pass at least one match[] selector, and the scraping job should set honor_labels: true so source labels are not overwritten.

13. What is the difference between relabel_configs and metric_relabel_configs?

relabel_configs runs on a target's labels before the scrape, so it decides which targets are scraped and what labels they get. metric_relabel_configs runs on the scraped samples after the scrape, so it can drop expensive series or rename labels before storage.

14. What does honor_labels do?

It defaults to false. In that case, if a scraped label conflicts with a label Prometheus attaches (such as job or instance), the scraped one is renamed with an exported_ prefix. With true, the scraped value wins. Use true for federation and the Pushgateway.

PromQL in production

15. What are the four metric types?

A counter only increases (or resets to zero on restart). A gauge goes up and down. A histogram counts observations into buckets and exposes _bucket, _sum and _count series. A summary exposes pre-computed quantiles from the client and cannot be aggregated across instances.

16. What is the difference between rate, irate and increase?

rate() is the per-second average over the whole range and is the right choice for alerts and slow-moving counters. irate() uses only the last two samples, so it suits graphs of fast-moving counters but is jumpy. increase() is rate() multiplied by the range length, and can return non-integers because of extrapolation. All three handle counter resets.

17. How do you calculate the HTTP error ratio?

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Regex matchers are fully anchored, so 5.. matches exactly three characters starting with 5. Add by (job) to both sums to get a ratio per service.

18. How do you get a p95 latency from a histogram?

histogram_quantile(0.95,
  sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))

Keep the le label in the aggregation or the function cannot work. The result is an estimate that is interpolated inside a bucket, so its accuracy depends on how well the bucket boundaries match your real latencies.

19. Why should you not use rate() on a gauge?

rate(), irate() and increase() are meant for counters because they assume resets. For gauges use delta() or deriv(). To predict a disk filling up, use predict_linear(), which is for gauges:

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

20. How do you alert when a metric disappears?

Use absent(), which returns a single series with value 1 when the selector matches nothing, or absent_over_time() for a range. Pair it with an up == 0 alert, because a target that is down and a metric that vanished are different failures.

21. What is staleness and the lookback window?

When a query evaluates at a time with no sample, Prometheus uses the newest sample within the lookback period, which defaults to 5 minutes (--query.lookback-delta). Series that stop being scraped are marked stale so they do not linger in results.

22. What is the difference between by and without?

sum by (job) (...) keeps only the listed labels. sum without (instance) (...) drops the listed labels and keeps everything else. Use without when new labels may appear later and you want them preserved.

23. What do offset and subqueries do?

offset 1d shifts a selector back in time, so rate(x[5m]) / rate(x[5m] offset 1d) compares now with yesterday. A subquery, written expr[range:resolution], evaluates an instant expression over a range and returns a range vector, for example the maximum of a computed rate over 6 hours.

24. What are recording rules for?

They precompute an expensive or frequently used expression and store the result as a new series, so dashboards and alerts query a cheap series instead of recalculating. Rules live in groups in a rule file, each with record and expr. Validate with promtool check rules.

groups:
  - name: http
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

25. What is high cardinality and how do you handle it?

Each unique combination of label values is a separate time series, so a label such as user ID or email address can create millions of series and exhaust memory. Keep unbounded values out of labels, set sample_limit on scrape jobs as a guard, and drop offenders with metric_relabel_configs. A quick, expensive way to find the worst metric names is topk(10, count by (__name__) ({__name__=~".+"})). Run it sparingly.

26. Which naming rules do interviewers expect?

Use base units (seconds, bytes), a unit suffix such as _seconds, and a _total suffix on counters. Add one application prefix, such as http_. Do not put label names in the metric name, and express percentages as ratios from 0 to 1.

Alerting rules and Alertmanager

27. How does an alerting rule work?

Prometheus evaluates the expression every evaluation interval. When it returns results, each is a pending alert. After the for duration it becomes firing and is sent to Alertmanager.

groups:
  - name: availability
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        keep_firing_for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} has been down for 5 minutes"

28. What is keep_firing_for?

It keeps an alert firing for a set time after its condition last held, which stops a flapping alert from resolving and re-firing repeatedly. Without it, the alert resolves on the first evaluation where the condition is false.

29. What does Alertmanager do that Prometheus does not?

Prometheus only decides that an alert is active. Alertmanager deduplicates, groups, routes, silences and inhibits alerts and sends notifications to receivers. Route defaults are group_wait 30s, group_interval 5m and repeat_interval 4h.

30. How do grouping and inhibition reduce noise?

Grouping with group_by (for example [alertname, cluster]) sends one notification for many related alerts. Inhibition suppresses target alerts while a source alert fires, provided the labels listed in equal match. A classic rule mutes pod alerts in a cluster while a "cluster unreachable" alert is firing.

31. What is the difference between a silence and a time interval?

A silence is a temporary mute you create at runtime for matching alerts, for example during maintenance. A time interval (time_intervals in the configuration, referenced from routes) is a recurring schedule such as weekday nights. Times are interpreted in UTC unless you set a location.

32. How do you test alerting rules before shipping them?

Write a unit test file with input_series and alert_rule_test assertions and run promtool test rules test.yml. Run it in CI next to promtool check rules, so a broken expression never reaches production.

Kubernetes integration and scale

33. How does Prometheus discover pods in Kubernetes?

Through kubernetes_sd_configs with a role of node, pod, service, endpoints, endpointslice or ingress. Each target carries meta labels such as __meta_kubernetes_namespace and __meta_kubernetes_pod_name, which you turn into final labels with relabeling.

scrape_configs:
  - job_name: pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The prometheus.io/scrape annotation is a community convention that you implement through this relabeling. Prometheus does not act on it by itself.

34. What does the Prometheus Operator add?

It manages Prometheus through Kubernetes custom resources. ServiceMonitor and PodMonitor declare what to scrape, PrometheusRule holds alerting and recording rules, and Probe covers blackbox-style checks. Teams add monitoring for their own services without editing a central configuration file.

35. When would you use agent mode?

Start Prometheus with the --agent flag when it should only scrape and forward samples through remote write, with no local querying or alerting. It suits many small clusters that ship metrics to one central backend.

36. How do you scale beyond a single server?

Options, from simplest: split scraping by team or environment into separate servers, use hierarchical federation for aggregates, or send data by remote write to a long-term store. Remote write sends snappy-compressed protobuf over HTTP, and a receiving Prometheus must be started with --web.enable-remote-write-receiver. With remote read, only raw series are fetched and all PromQL evaluation still happens in Prometheus.

Real-time incident scenarios

37. Scrapes started failing right after you upgraded to Prometheus 3. Why?

The most likely cause is the stricter Content-Type check. Prometheus 3 fails a scrape when the response header is missing, unparsable or unrecognised. Fix the exporter, or set fallback_scrape_protocol on the job as a stopgap, for example to PrometheusText0.0.4.

38. Dashboards and alerts that match le="1" return nothing after the upgrade. What changed?

Prometheus 3 normalises le and quantile values to floats on ingestion, so le="1" becomes le="1.0". Update queries that match whole-number values, or retain the old format for affected metrics with metric_relabel_configs. Also check that your Alertmanager is 0.16.0 or later and that api_version is v2, because the v1 API was removed.

39. Prometheus is using far more memory since a new service shipped. What do you do?

Suspect new high-cardinality series. Identify the metric with the biggest series count, find the label that varies without bound (request path with IDs, user IDs, pod hash), and fix it in the application. Until the fix ships, drop the label or the metric with metric_relabel_configs and set sample_limit on that job so one service cannot take the server down again.

40. An alert fires and resolves every few minutes all night. How do you stop the noise?

Look at the expression first. Use a longer rate() window, add or lengthen for, and add keep_firing_for so a brief dip does not resolve it. In Alertmanager, check that group_by is not splitting one problem into many notifications. If the alert is not actionable, delete it rather than tune it.

Frequently asked questions

What should I learn first to answer Prometheus interview questions?

Start with the pull model, the four metric types, rate(), sum by and histogram_quantile(). Then learn alerting rules and Alertmanager routing, followed by service discovery and storage sizing.

What is the default scrape interval in Prometheus?

The global default is 1 minute, with a 10 second scrape timeout. Many teams lower the interval to 15 or 30 seconds, but each halving doubles the samples stored.

How long does Prometheus keep data by default?

15 days, unless you set --storage.tsdb.retention.time or --storage.tsdb.retention.size. If both are set, the first limit reached applies.

What is the difference between Prometheus and Grafana?

Prometheus collects, stores and queries metrics and evaluates alert rules. Grafana is a visualisation layer that queries Prometheus and other data sources to draw dashboards. They are usually deployed together.

Do I need to know Prometheus 3 for interviews in 2027?

Yes. New installs run version 3, so expect questions about UTF-8 names, scrape protocol handling, normalised le labels and the removal of the Alertmanager v1 API.

How do I prepare for hands-on Prometheus questions?

Run Prometheus, Node Exporter and Alertmanager locally, write an error ratio query, create an alert and a recording rule, and test the rule with promtool test rules. Break a target on purpose and debug it from the Targets page.

Is the Pushgateway a good way to monitor short-lived jobs?

Only for service-level batch jobs. It does not expire pushed series and removes the up signal, so it is a poor fit for per-instance metrics.

Where can I practise with a mentor?

Our Prometheus and Grafana training runs in the Kharadi classroom in Pune and live online across India. To connect metrics to dashboards, read how Prometheus and Grafana work together, then CloudWatch versus Prometheus and the monitoring alerts guide.

Official references

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.