Prometheus Interview Questions: Freshers to Seniors (2027)

42 Prometheus interview questions by level: fresher basics, mid-level alerting and relabeling, senior HA, remote write and Prometheus 3 upgrades.

Sep 13, 2025 - 14:48
Nov 9, 2026 - 12:54
 0  69
Prometheus Interview Questions: Freshers to Seniors (2027)

Prometheus interview questions change with experience. Freshers are asked about the pull model, metric types, ports and basic PromQL. Candidates with two to five years get alert routing, recording rules, relabeling and cardinality. Senior candidates are asked to design for high availability, long-term storage and upgrades such as the move to Prometheus 3. This page gives 42 questions across those levels.

Each answer starts with the direct reply an interviewer expects, then adds the detail that separates a stronger candidate. It belongs to our Prometheus interview series. The main page is 80+ Prometheus interview questions and answers, which covers the concepts in breadth. This page follows one candidate's path from first job to senior role, so you can see how the same topic is asked differently at each level. Defaults and flags were checked against the Prometheus documentation in November 2026, when the latest release was v3.16.0.

Key takeaways

  • Fresher rounds test vocabulary: pull model, the four metric types, prometheus.yml, exporters and simple PromQL with rate().
  • Mid-level rounds test operations: Alertmanager grouping and inhibition, recording rules, relabeling, label cardinality and safe config reloads.
  • Senior rounds test design: HA pairs, remote write, long-term storage with tools such as Thanos or Mimir, federation, SLO alerting and Prometheus 3 migration.
  • Know the defaults: scrape_interval 1m, scrape_timeout 10s, evaluation_interval 1m, retention 15d, ports 9090 (server), 9093 (Alertmanager), 9091 (Pushgateway) and 9100 (node exporter).
  • Validate before you reload: promtool check config, promtool check rules and promtool test rules.

What each level is expected to know

LevelTypical topicsWhat a good answer shows
FreshersArchitecture, metric types, exporters, scrape config, basic PromQLCorrect terms and a working mental model
2 to 5 yearsAlerting rules, Alertmanager routing, recording rules, relabeling, cardinality, Kubernetes discoveryHands-on config and debugging habits
SeniorHA, remote write, long-term storage, federation, SLOs, capacity, upgradesTrade-offs and reasons behind a design

Prometheus questions for freshers

1. What is Prometheus?

Prometheus is an open source monitoring and alerting toolkit that collects numeric time series by scraping HTTP endpoints, stores them in a local time series database, and lets you query them with PromQL. It is a graduated CNCF project. Alerts are evaluated by Prometheus and sent to Alertmanager for routing and notification.

2. Does Prometheus push or pull metrics?

It pulls. Prometheus scrapes targets on a schedule from an endpoint, usually /metrics. The exception is the Pushgateway, which lets short-lived batch jobs push metrics that Prometheus then scrapes. The Pushgateway project itself says its main valid use is service-level batch jobs, not general push monitoring.

3. Name the four metric types.

Counter (only goes up, resets on restart), gauge (goes up and down), histogram (counts observations in buckets plus a sum and count) and summary (client-side quantiles plus sum and count). Prometheus also supports native histograms, which became stable in the 3.x line; from v3.9 the old feature flag does nothing and scraping them is enabled with scrape_native_histograms.

4. What default ports should you know?

9090 for the Prometheus server, 9093 for Alertmanager, 9091 for the Pushgateway and 9100 for the node exporter. These come from the project's default port allocation list.

5. What goes into prometheus.yml?

A global block with defaults, rule_files, an alerting block pointing to Alertmanager, and scrape_configs that define jobs and targets.

global:
  scrape_interval: 30s
  evaluation_interval: 30s
rule_files:
  - "rules/*.yml"
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["node1:9100", "node2:9100"]

6. What are the default scrape interval, scrape timeout and evaluation interval?

scrape_interval defaults to 1m, scrape_timeout to 10s and evaluation_interval to 1m. Many teams set 15s or 30s globally, but the built-in defaults are the ones to quote.

7. What is an exporter?

A process that reads metrics from a system that does not speak Prometheus format and exposes them on an HTTP endpoint. Examples are the node exporter for Linux host metrics and the blackbox exporter for probing endpoints.

8. What is a label, and why does it matter?

A key-value pair that identifies a time series, such as job="api" or status="500". Every unique label combination is a separate series, so labels make queries flexible but also drive memory use.

9. What does rate() do?

It calculates the per-second average rate of increase of a counter over a range and handles counter resets. Use it on counters, never on gauges.

sum by (job) (rate(http_requests_total[5m]))

10. What is the difference between rate() and irate()?

rate() averages over the whole range and suits alerts and slow-moving graphs. irate() uses only the last two samples, so it reacts fast but is noisy and a poor choice for alerting.

11. How long does Prometheus keep data by default?

15 days, if neither --storage.tsdb.retention.time nor --storage.tsdb.retention.size is set. When both are set, whichever limit is reached first applies.

12. What is the difference between Prometheus and Grafana?

Prometheus collects, stores and queries metrics and evaluates alerts. Grafana visualizes data from Prometheus and many other sources. Our guide to using Prometheus and Grafana together shows the setup.

13. How do you check that Prometheus is up?

Call /-/healthy for liveness and /-/ready for readiness to serve queries, and look at the Targets page to see which scrapes are failing.

14. How should metrics be named?

Use a prefix for the domain, base units such as seconds and bytes, and a _total suffix for counters, for example http_request_duration_seconds and process_cpu_seconds_total. Do not mix units in one metric.

Prometheus questions for 2 to 5 years of experience

15. How do you write an alerting rule that does not flap?

Use a for duration so the condition must hold before the alert fires, and base it on rate() over a sensible window. keep_firing_for keeps an alert firing for a period after the condition clears, which reduces flapping on resolve. Both default to 0s.

groups:
  - name: api
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API 5xx ratio above 5% for 10 minutes"

16. What does Alertmanager add on top of Prometheus alerts?

Grouping, deduplication, routing to receivers, silences and inhibition. Prometheus decides that an alert is firing. Alertmanager decides who hears about it and how often.

17. Explain group_wait, group_interval and repeat_interval.

group_wait (default 30s) is how long Alertmanager waits to collect alerts for a new group before the first notification. group_interval (default 5m) is the wait before notifying about new alerts added to an existing group. repeat_interval (default 4h) is how long before a still-firing notification is sent again.

18. What is inhibition?

A rule that mutes some alerts while another alert is firing, for example muting all service alerts in a data centre when a "data centre down" alert is active. It cuts noise during big outages.

19. Why use recording rules?

To precompute expensive or frequently used expressions and store the result as a new series. Dashboards and alerts then query the cheap result. Name them with the level:metric:operations convention, for example job:http_requests:rate5m.

20. How do you validate config and rules before deploying?

Run promtool check config prometheus.yml and promtool check rules rules.yml in CI, and write unit tests for alert rules run with promtool test rules.

promtool check config prometheus.yml
promtool check rules rules/*.yml
promtool test rules tests/alerts_test.yml

21. How do you reload configuration without restarting?

Send SIGHUP to the process, or POST to /-/reload. The HTTP endpoint is disabled by default and needs the --web.enable-lifecycle flag. If the new file is invalid, Prometheus keeps the old configuration and logs an error.

22. What is high cardinality, and how do you find it?

Too many unique label combinations, often from labels such as user ID, request ID or full URL. It raises memory use and slows queries. Find it with the TSDB status page in the UI, which lists series counts by metric and label, or a query such as topk(10, count by (__name__) ({__name__=~".+"})) on a test instance.

23. What is the difference between relabel_configs and metric_relabel_configs?

relabel_configs runs before the scrape and works on target labels from service discovery, so it decides what gets scraped and how targets are labelled. metric_relabel_configs runs after the scrape on each sample, so it can drop or rewrite metrics before storage.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_gc_.*"
    action: drop

24. How does Prometheus discover Kubernetes targets?

With kubernetes_sd_configs, using roles such as node, pod, service, endpoints, endpointslice and ingress. Relabeling then keeps the targets you want. Many teams use the Prometheus Operator, which turns ServiceMonitor and PodMonitor objects into scrape config. Our Prometheus Kubernetes monitoring questions go deeper on this.

25. What does honor_labels do?

It controls label conflicts between scraped data and target labels. With honor_labels: true, labels from the scraped data win. It is typical for the Pushgateway and federation, where the source labels must be kept.

26. How do you alert when a target disappears?

Alert on up == 0 for scraped targets. For a metric that should always exist, use absent() or absent_over_time(), because a missing series returns nothing rather than zero.

27. How do you calculate a 95th percentile latency from a histogram?

Use histogram_quantile() on the rate of the bucket series, aggregated by le:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Point out that the result is an estimate whose accuracy depends on bucket boundaries.

28. How do you predict a disk filling up?

Use predict_linear() on a gauge such as free bytes, for example alert if predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0. It gives warning hours before the disk is full.

Prometheus questions for senior roles

29. How do you make Prometheus highly available?

Run two identical Prometheus servers scraping the same targets and send alerts from both to an Alertmanager cluster, which deduplicates them. Queries can go to either server. For a single global view and long retention, add a layer such as Thanos, Cortex or Grafana Mimir.

30. What does the local storage look like, and what are its limits?

Samples are grouped into two-hour blocks, protected by a write-ahead log in 128 MB segments, and compacted into larger blocks in the background. Local storage is not clustered or replicated, so it is not meant to be durable long-term storage. Plan backups or remote storage for that.

31. When would you use remote write?

To send samples to a long-term or central store, or to a managed service. Prometheus can also accept remote write with --web.enable-remote-write-receiver. Agent mode, started with the --agent flag, scrapes and forwards data without local querying, which suits edge clusters.

32. What is federation, and when is it a poor fit?

Federation lets one Prometheus scrape selected series from another through /federate. It works for pulling aggregated series up a hierarchy. It is a poor way to copy all raw data, where remote write to a long-term store scales better.

33. How do you size Prometheus?

The docs give a rough formula: needed disk space equals retention time in seconds times ingested samples per second times bytes per sample. Memory grows with active series. Measure active series and ingestion rate on a test instance before committing hardware, and leave headroom when you use size-based retention.

34. What changed in Prometheus 3 that affects upgrades?

Several feature flags became default behaviour or dedicated flags (for example --agent), UTF-8 metric and label names are allowed, targets no longer get a default port added from the scheme, and holt_winters was renamed double_exponential_smoothing behind the promql-experimental-functions flag. Read the migration guide and test rules and dashboards on a staging instance first.

35. How would you alert on SLOs?

Define an SLI such as the ratio of good requests, record it with recording rules over several windows, and alert on error budget burn rate with a fast and a slow window, as described in the Google SRE workbook. This pages on real user impact rather than on CPU or single errors.

36. How do you secure a Prometheus server?

Prometheus supports TLS and basic authentication through a web configuration file. Restrict network access to the UI and API, keep the lifecycle and admin APIs off unless needed, protect exporters that expose sensitive data, and keep credentials for scrape targets in files or secrets rather than in plain config.

37. How do you keep alert quality high across many teams?

Keep rules in Git with review, require promtool checks and rule unit tests in CI, give every alert a severity, owner and runbook link, and review noisy alerts regularly. Alerts nobody acts on should be removed or turned into dashboards.

Scenario questions across levels

38. A target shows as down on the Targets page. How do you debug it?

Read the error shown next to the target. Then curl the metrics URL from the Prometheus host to check network, TLS and authentication, and confirm that the scrape takes less than scrape_timeout. If the target is missing from the page altogether, check discovery and relabeling rules.

39. Memory use on Prometheus doubled after a release. What do you check?

Check the TSDB status page for metrics with the most series, then compare with the previous release. A new label carrying user IDs or URLs is the usual cause. Drop or rewrite it with metric_relabel_configs and fix the instrumentation in the application.

40. An alert fired, but nobody got a notification. Where do you look?

Check that the alert is firing in Prometheus, that Prometheus can reach Alertmanager, that a route matches the alert's labels, that no silence or inhibition is muting it, and that the receiver integration works. The Alertmanager UI shows active alerts, silences and groups.

41. A dashboard query times out over a 30-day range. How do you fix it?

Create a recording rule for the expensive expression and point the dashboard at it. Reduce the number of series returned with aggregation, and use a long-term store with downsampling if long ranges are common.

42. After an upgrade to Prometheus 3, some regex-based alerts behave differently. Why?

The migration guide lists PromQL changes, including that regular expressions now match newlines and that range selectors exclude samples at the left boundary. Test those rules with promtool test rules against sample data and adjust them.

Where to go next

Use these related pages for the topics this guide only touches:

For guided labs on scraping, PromQL, alerting and Grafana dashboards, see our Prometheus and Grafana training.

FAQ

What Prometheus questions are asked to freshers?

Pull versus push, the four metric types, default ports, the structure of prometheus.yml, exporters, labels and basic PromQL with rate().

What do experienced Prometheus interviews focus on?

Alerting rules and Alertmanager routing, recording rules, relabeling, cardinality control and Kubernetes service discovery. Senior rounds add HA, remote write, long-term storage and SLO alerting.

What is the default Prometheus retention period?

15 days when no retention time or size flag is set.

Which port does Prometheus use?

9090 by default. Alertmanager uses 9093, the Pushgateway 9091 and the node exporter 9100.

How do I reload Prometheus configuration?

Send SIGHUP to the process, or POST to /-/reload when --web.enable-lifecycle is enabled.

Is there an official Prometheus certification?

Yes. The Linux Foundation and CNCF offer the Prometheus Certified Associate (PCA), an online proctored multiple-choice exam.

What is the difference between rate and irate?

rate() averages over the whole range and suits alerts. irate() uses the last two samples and suits fast-moving graphs.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.