Most Asked Prometheus Interview Questions: Config (2027)

44 most asked Prometheus interview questions on prometheus.yml, relabeling, recording rules, retention flags, remote write and promtool, by level.

Sep 13, 2025 - 14:33
Nov 5, 2026 - 03:05
 0  71
Most Asked Prometheus Interview Questions: Config (2027)

The most asked Prometheus interview questions in hands-on rounds are about configuration: how prometheus.yml is laid out, what the global defaults are, how relabeling filters targets and series, how recording rules and retention flags work, and how you validate and reload changes with promtool. This page answers 44 of them, grouped by experience level.

Defaults, flags and field names below were checked against the Prometheus documentation for the 3.x release line in November 2026. If you want concept questions, PromQL drills or internals instead, the sibling pages linked in the first section cover those angles.

Key takeaways

  • Know the three global defaults by heart: scrape_interval 1m, scrape_timeout 10s, evaluation_interval 1m.
  • relabel_configs run on targets before the scrape; metric_relabel_configs run on samples after it. Mixing these up is the most common config mistake interviewers probe.
  • Retention defaults to 15 days only when neither --storage.tsdb.retention.time nor --storage.tsdb.retention.size is set.
  • Prometheus reloads its config on SIGHUP or on a POST to /-/reload when --web.enable-lifecycle is set. A malformed file is rejected and the old config keeps running.
  • Validate everything in CI with promtool check config, promtool check rules and promtool test rules.

Which Prometheus page to read

This page is the configuration and operations set. The broad keeper is 80+ Prometheus interview questions and answers. For PromQL tasks use Prometheus engineer questions, for TSDB internals use advanced Prometheus questions, for incidents use scenario-based Prometheus questions, and for one-line answers use the Prometheus FAQ page. The monitoring pillar, top observability tools for DevOps, shows where Prometheus sits next to other tools.

Setting or flagWhere it livesDefault
scrape_intervalglobal or per job1m
scrape_timeoutglobal or per job10s (cannot exceed the interval)
evaluation_intervalglobal1m
honor_labelsscrape jobfalse
sample_limitscrape job0 (no limit)
--storage.tsdb.pathcommand linedata/
--storage.tsdb.retention.timecommand line15d if no retention flag is set
--storage.tsdb.retention.sizecommand line0 (disabled)

Freshers: reading a Prometheus config

1. What are the top-level sections of prometheus.yml?

The common ones are global, rule_files, alerting, scrape_configs, remote_write and remote_read. Most small setups only use the first four.

global:
  scrape_interval: 30s
  evaluation_interval: 30s
rule_files:
  - "rules/*.yml"
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["10.0.0.11:9100", "10.0.0.12:9100"]

2. What are the default scrape and evaluation intervals?

Both default to one minute. The scrape timeout defaults to 10 seconds and the docs state it cannot be greater than the scrape interval. Many teams lower the interval to 15s or 30s, which raises sample volume proportionally.

3. What is a job and what is an instance?

A job is one entry under scrape_configs, identified by job_name. Each target the job scrapes is an instance. Prometheus attaches the job and instance labels to every scraped series automatically.

4. Can you override the global scrape interval for one job?

Yes. Set scrape_interval inside the job. Per-job values win over global ones. This is how you scrape a slow, expensive exporter every two minutes while the rest stays at 30s.

5. What does rule_files accept?

A list of file globs. Prometheus loads recording and alerting rules from every file that matches, so rules/*.yml is a typical entry.

6. How does Prometheus find Alertmanager?

Through the alerting.alertmanagers block, which supports static targets and the same service discovery mechanisms as scrape jobs. Prometheus sends alerts there; routing and notifications happen in Alertmanager.

7. What is the up metric?

A synthetic series Prometheus writes for each target on every scrape: 1 if the scrape succeeded, 0 if it failed. up == 0 is the first query to run when a dashboard goes blank.

8. How do you check a config file before restarting?

Run promtool check config prometheus.yml. It parses the file and every rule file it references and exits non-zero on errors, so it fits in a CI step.

9. How do you apply a config change without restarting?

Send SIGHUP to the process, or POST to /-/reload if Prometheus was started with --web.enable-lifecycle. If the new file is not well formed, Prometheus keeps the old configuration.

promtool check config /etc/prometheus/prometheus.yml && \
  curl -X POST http://localhost:9090/-/reload

10. Where does Prometheus store data by default?

In data/ relative to the working directory, controlled by --storage.tsdb.path. Packaged installs usually point it at a dedicated volume.

11. How long does Prometheus keep data by default?

Fifteen days, but only when neither retention flag is set. Set --storage.tsdb.retention.time (for example 30d) or --storage.tsdb.retention.size (for example 200GB). If both are set, whichever limit is hit first applies.

12. What is static_configs and when is it enough?

A fixed list of targets written in the file. It works for a handful of stable hosts. Anything that scales up and down (Kubernetes pods, autoscaling VMs) needs service discovery such as kubernetes_sd_configs, ec2_sd_configs or file_sd_configs.

13. What is file_sd_configs used for?

Prometheus watches JSON or YAML files that list targets and labels. Another tool (Ansible, a CMDB export, a script) writes those files, and Prometheus picks up changes without a reload.

14. What does honor_labels do?

It decides what happens when scraped data carries a label that Prometheus also sets, such as job. With the default false, the scraped label is renamed to exported_job. With true, the scraped value is kept. Federation and Pushgateway jobs normally set it to true.

2 to 5 years: relabeling, rules and storage

15. What is the difference between relabel_configs and metric_relabel_configs?

relabel_configs run against discovered targets before the scrape, so they decide which targets exist and what their labels are. metric_relabel_configs run against every scraped sample before storage, so they drop or rewrite series. Dropping a target in the first saves the scrape; dropping a series in the second still costs the scrape but saves storage.

16. List the relabel actions.

replace (the default), lowercase, uppercase, keep, drop, keepequal, dropequal, hashmod, labelmap, labeldrop and labelkeep.

17. What are the defaults for regex, separator and replacement?

regex defaults to (.*), separator to ; and replacement to $1. Source label values are joined with the separator before the regex runs, which is why multi-label matches look like "prod;api".

18. Write a relabel rule that keeps only pods with a scrape annotation.

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: "true"

Annotation-based scraping is a convention used by many Helm charts, not a Prometheus built-in, so say that if asked.

19. What do the double-underscore labels mean?

Labels starting with __ exist only during relabeling and are removed afterwards. __address__ is the target's host and port, __scheme__ and __metrics_path__ come from the job, __param_<name> sets a URL parameter, and __meta_* labels come from service discovery. The __tmp prefix is reserved for your own temporary values.

20. How do you drop a high-cardinality metric you do not use?

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "apiserver_request_duration_seconds_bucket"
    action: drop

Better still, fix it at the source or stop the exporter emitting it, because the scrape still transfers the data.

21. What does labelmap do?

It copies every label whose name matches the regex to a new name built from the replacement. The classic use is turning __meta_kubernetes_pod_label_(.+) into plain pod labels so they survive relabeling.

22. What does hashmod do and why is it useful?

It writes a hash of the source labels modulo modulus into a target label. Pair it with a keep on one value and each Prometheus server scrapes only its share of targets. That is horizontal sharding by config alone.

23. What is sample_limit and what happens when a target exceeds it?

The maximum number of samples allowed per scrape after metric relabeling. If a target exceeds it, the whole scrape is treated as failed, so up becomes 0. It protects the server from one exporter that suddenly emits a million series.

24. What is a recording rule and why would you write one?

A rule that evaluates a PromQL expression on a schedule and stores the result as a new series under the record name. Use it for expensive queries that dashboards or alerts run repeatedly, and for building SLO series.

groups:
  - name: http
    interval: 1m
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

25. What naming convention do recording rules follow?

The documented best practice is level:metric:operations, where level is the aggregation labels, metric is the source metric name and operations lists what was applied. The docs' own example is code:prometheus_http_requests_total:sum.

26. What do limit and query_offset do on a rule group?

limit caps how many series a recording rule (or alerts an alerting rule) may produce; if exceeded, all its output is discarded and an evaluation error is recorded. query_offset shifts evaluation into the past so late samples, for example from remote write, have arrived.

27. How is data laid out on disk?

New samples go to a write-ahead log in 128 MB segments, and the head block is cut into two-hour blocks. Compaction merges blocks into larger ones covering up to 10% of the retention time or 31 days, whichever is smaller.

28. How do you estimate disk needs?

The docs give: retention in seconds times ingested samples per second times bytes per sample, with Prometheus averaging about 1 to 2 bytes per sample. Keep the retention size limit at no more than 80 to 85% of the disk, because compaction briefly holds old and new blocks together.

29. Is WAL compression on by default?

Yes. --storage.tsdb.wal-compression arrived in 2.11.0 and has been enabled by default since 2.20.0. It roughly halves WAL size for little CPU.

30. How do you see which metrics use the most series?

Run promtool tsdb analyze against the data directory, or check the TSDB status page in the web UI. A query such as topk(10, count by (__name__) ({__name__=~".+"})) works too but is expensive on a large server.

Senior: remote write, federation and scale

31. Which queue_config settings matter most for remote write?

capacity (default 10000), max_shards (50), min_shards (1), max_samples_per_send (2000), batch_send_deadline (5s), and the backoff pair min_backoff (30ms) and max_backoff (5s). When the remote end is slow, shards rise toward the maximum; if they sit at the maximum, the receiver or network is the bottleneck.

32. How do you send only some series to remote storage?

Use write_relabel_configs on the remote_write entry. It runs after external labels are added, so you can keep only recording-rule output or drop noisy jobs before they leave the server.

33. What are external_labels for?

Labels added to every series and alert when it leaves this server: remote write, federation and Alertmanager. Set cluster and replica here so downstream systems can tell servers apart and deduplicate HA pairs.

34. How do two HA Prometheus servers avoid duplicate pages?

Both send the same alerts to Alertmanager, which deduplicates identical alerts. If the replicas carry a different replica external label, strip it with alert_relabel_configs so the alerts are identical. The docs give exactly this example.

35. How does federation work and when is it the wrong tool?

A higher-level server scrapes /federate on lower servers with match[] selectors and honor_labels: true. It suits pulling a few aggregated series up a level. It is the wrong tool for copying all raw data to a central place; use remote write to a long-term store for that.

36. How would you shard scraping across several servers?

Give each server the same job list plus a hashmod relabel with modulus equal to the server count and a keep on that server's index. Query across shards with a global layer such as Thanos or a remote-write backend.

37. How do you secure the web endpoints?

Pass a web configuration file with --web.config.file to enable TLS and basic auth, and validate it with promtool check web-config. Leave --web.enable-lifecycle and the admin API off unless you need them, because they allow reloads, shutdowns and deletes.

38. How do you test rules in CI?

Write a unit test file and run promtool test rules. Tests define input_series, then assert alerts with alert_rule_test or expression results with promql_expr_test. Add promtool check rules for syntax.

39. What does a reload actually change, and what needs a restart?

A reload re-reads prometheus.yml and rule files. Command-line flags, such as retention and storage path, only change on restart. Interviewers ask this to see whether you know retention is a flag, not a config key.

40. How do you roll out config changes safely to many servers?

Keep config in Git, render per-cluster values with a template tool, run promtool check config and rule tests in the pipeline, deploy to one canary server, compare its up and series count with a peer, then reload the rest. In Kubernetes, the Prometheus Operator generates config from ServiceMonitor and PodMonitor objects instead; see Prometheus Kubernetes monitoring questions.

Scenario questions

41. After a config change, a whole job shows up == 0. What do you check?

Open the Targets page and read the scrape error. Typical causes are a changed metrics_path, a scheme mismatch (HTTP versus HTTPS), a __address__ rewrite that dropped the port, or sample_limit now being exceeded. Diff the old and new config and test the endpoint with curl from the Prometheus host.

42. Disk is filling faster than planned. What do you do?

Set or lower --storage.tsdb.retention.size as a safety net, then find the cause with promtool tsdb analyze. It is usually a new label with unbounded values. Drop it with metric_relabel_configs now and fix the instrumentation later.

43. Your reload returns success but the new job does not appear.

Check that you edited the file Prometheus actually reads (the --config.file flag), that the job is not removed by a keep relabel, and the logs for a failed reload. The prometheus_config_last_reload_successful metric tells you whether the last reload worked.

44. Remote write is falling behind and memory grows.

Look at the shard count and pending samples metrics for the queue. If shards are at max_shards, the receiver is slow: scale it, or raise shards if the receiver has headroom. Reduce what you send with write_relabel_configs. For the deeper internals, read the advanced Prometheus page.

How to prepare for config questions

Run Prometheus and node_exporter locally, write a config from memory, break it on purpose and watch what promtool and the Targets page tell you. Add one relabel rule of each common type and one recording rule, then write a unit test for an alert. If you want guided labs, the Prometheus and Grafana training runs in our Kharadi, Pune classroom and live online, and our guide to using Prometheus and Grafana together covers the dashboard side.

Frequently asked questions

What is the default scrape interval in Prometheus?

One minute. The scrape timeout defaults to 10 seconds and cannot be longer than the interval.

What is the difference between relabel_configs and metric_relabel_configs?

relabel_configs act on targets before scraping. metric_relabel_configs act on scraped samples before they are stored.

How do I reload Prometheus configuration?

Send SIGHUP, or POST to /-/reload when the --web.enable-lifecycle flag is enabled. Invalid files are not applied.

What is the default retention in Prometheus?

15 days, used only when neither --storage.tsdb.retention.time nor --storage.tsdb.retention.size is set.

Can retention be changed with a reload?

No. Retention is a command-line flag, so it needs a restart.

Which promtool commands should I know?

check config, check rules, test rules, check metrics, check web-config, query instant and tsdb analyze.

Why set honor_labels to true for federation?

So the job and instance labels from the source server are kept instead of being renamed to exported_job and exported_instance.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.