Prometheus Kubernetes Monitoring Interview Questions (2027)
42 Prometheus interview questions on monitoring Kubernetes: service discovery, kube-state-metrics, cAdvisor, ServiceMonitor, PrometheusRule, alerts.
Prometheus monitoring interview questions for DevOps roles focus on running Prometheus against Kubernetes: how it discovers pods and services, what kube-state-metrics, cAdvisor and node_exporter each report, how the Prometheus Operator's ServiceMonitor and PrometheusRule resources work, and which alerts a cluster needs. Expect to explain both the setup and the queries. This page has 42 questions with answers, grouped by level, plus scenarios.
Facts were checked against prometheus.io, the Prometheus Operator documentation and kubernetes.io in November 2026. For general Prometheus concepts, start with our 80+ Prometheus interview questions. For quick definitions, use the Prometheus FAQs. This page stays on one job: monitoring Kubernetes clusters and the workloads on them.
Key takeaways
- Prometheus finds Kubernetes targets with
kubernetes_sd_configs, using the rolesnode,service,pod,endpoints,endpointsliceandingress. - Three sources cover most cluster metrics: node_exporter for hosts, cAdvisor in the kubelet for container usage, and kube-state-metrics for the state of Kubernetes objects.
- metrics-server feeds
kubectl topand the Horizontal Pod Autoscaler. It is not a Prometheus data source. - The Prometheus Operator replaces hand-written scrape config with custom resources such as ServiceMonitor, PodMonitor and PrometheusRule.
- The kube-prometheus-stack Helm chart from the prometheus-community repository installs the Operator, Prometheus, Alertmanager, Grafana and default rules together.
- Good cluster alerts are about symptoms users feel and about saturation: crash loops, unavailable replicas, full disks and failing scrapes.
The Kubernetes monitoring stack at a glance
| Component | What it reports | Example metric |
|---|---|---|
| node_exporter (DaemonSet) | Host CPU, memory, disk, network, filesystem | node_filesystem_avail_bytes |
| cAdvisor (inside kubelet) | Per-container CPU, memory, I/O | container_memory_working_set_bytes |
| kube-state-metrics | Desired and current state of Deployments, Pods, Nodes, Jobs and more | kube_deployment_status_replicas_available |
kubelet /metrics | Kubelet operations, volume stats | kubelet_volume_stats_available_bytes |
API server /metrics | Request rates, latencies and errors of the control plane | apiserver_request_total |
Application /metrics | Business and service metrics you instrument | http_requests_total |
Questions for freshers
1. Why is Prometheus a common choice for Kubernetes monitoring?
Its label-based data model matches Kubernetes labels, it has native Kubernetes service discovery, and Kubernetes components already expose metrics in Prometheus format. Both are CNCF graduated projects and are often deployed together.
2. How does Prometheus discover targets in Kubernetes?
Through kubernetes_sd_configs, which watches the Kubernetes API. You choose a role, such as pod or endpoints, and Prometheus turns each discovered object into a target with metadata labels like __meta_kubernetes_namespace and __meta_kubernetes_pod_name.
3. What is node_exporter and how is it deployed in a cluster?
An exporter for hardware and operating system metrics on Linux hosts, listening on port 9100 by default. In Kubernetes it runs as a DaemonSet so every node has one copy.
4. What is kube-state-metrics?
A service that listens to the Kubernetes API and generates metrics about the state of objects: replica counts, pod phases, container restarts, resource requests and limits, node conditions. It does not measure CPU or memory usage.
5. Where do container CPU and memory metrics come from?
From cAdvisor, which is built into the kubelet and exposed at /metrics/cadvisor. Metrics start with container_, for example container_cpu_usage_seconds_total.
6. What is the difference between metrics-server and Prometheus?
metrics-server collects current CPU and memory usage from kubelets and serves it through the Metrics API for kubectl top and the Horizontal Pod Autoscaler. It keeps no history. Prometheus stores time series for queries, dashboards and alerts.
7. What is the Prometheus Operator?
A Kubernetes operator that manages Prometheus, Alertmanager and related components through custom resources. You declare what to monitor in Kubernetes YAML and the Operator generates the Prometheus configuration and reloads it.
8. What is kube-prometheus-stack?
A Helm chart in the prometheus-community repository that installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and a set of default dashboards and alerting rules.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
9. How do you open the Prometheus UI in a cluster without exposing it?
Use port forwarding, for example kubectl port-forward -n monitoring svc/<prometheus-service> 9090, then browse to localhost:9090. The service name depends on the Helm release name.
10. How does an application expose metrics for Prometheus?
It uses a Prometheus client library (official ones exist for Go, Java, Python and Ruby, among others) to register metrics and serve them on an HTTP endpoint, usually /metrics. A Service or pod port then makes it scrapeable.
11. Which query shows pod restarts?
kube-state-metrics exposes kube_pod_container_status_restarts_total. To see restarts in the last hour per pod: increase(kube_pod_container_status_restarts_total[1h]).
12. Which query shows container memory use?
container_memory_working_set_bytes, filtered to real containers, for example sum by (namespace, pod) (container_memory_working_set_bytes{container!=""}). The kubelet uses working set memory when deciding on evictions, so it is the more useful figure than total usage.
13. How do you see CPU usage per pod?
sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
The result is in CPU cores, so 0.25 means a quarter of a core.
14. What does the up metric tell you in Kubernetes?
Whether each discovered target was scraped successfully. up == 0 for a job means Prometheus found the target but could not scrape it, often because of a wrong port, a network policy, or a pod that is not ready.
Questions for 2 to 5 years of experience
15. What is a ServiceMonitor?
A Prometheus Operator custom resource that tells Prometheus to scrape the endpoints behind Services matching a label selector. It names the port and path to scrape.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: orders-api
namespace: shop
labels:
release: monitoring
spec:
selector:
matchLabels:
app: orders-api
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
16. Why does Prometheus ignore a ServiceMonitor you created?
The Prometheus resource selects ServiceMonitors with serviceMonitorSelector and serviceMonitorNamespaceSelector. If your labels or namespace do not match, it is skipped. With kube-prometheus-stack the default selector usually expects a release label equal to the Helm release name, unless you change the chart values.
17. ServiceMonitor or PodMonitor: when do you use each?
Use a ServiceMonitor when the workload has a Service. Use a PodMonitor when pods expose metrics but have no Service, such as some batch or sidecar workloads.
18. What is a PrometheusRule?
A custom resource that holds recording and alerting rules. The Operator loads rules from PrometheusRule objects that match the Prometheus resource's ruleSelector.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: orders-alerts
namespace: shop
labels:
release: monitoring
spec:
groups:
- name: orders
rules:
- alert: OrdersHighErrorRate
expr: |
sum(rate(http_requests_total{job="orders-api",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="orders-api"}[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "More than 5% of orders-api requests are failing"
19. What other custom resources does the Operator provide?
Besides Prometheus, Alertmanager, ServiceMonitor, PodMonitor and PrometheusRule, the Operator documents Probe (for blackbox-style probing), AlertmanagerConfig (per-namespace routing), ThanosRuler, ScrapeConfig and PrometheusAgent.
20. Without the Operator, how do you scrape only annotated pods?
Use the pod role and a keep relabel rule on an annotation. The prometheus.io/scrape annotation is a community convention, not a built-in Kubernetes or Prometheus feature, so it only works if your config implements it.
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
21. What RBAC does Prometheus need in Kubernetes?
A ServiceAccount with a ClusterRole that can get, list and watch nodes, services, endpoints (or endpointslices), pods and ingresses, plus get on the /metrics non-resource URL. Charts create this for you, but interviewers like to check that you know why discovery fails without it.
22. How do you alert on a Deployment with unavailable replicas?
Compare desired and available replicas from kube-state-metrics:
kube_deployment_spec_replicas != kube_deployment_status_replicas_available
Add for: 15m so a normal rolling update does not page anyone.
23. How do you alert on a PersistentVolume filling up?
Use kubelet volume stats, for example fire when less than 10 percent remains: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.10. A predict_linear() rule can warn before the disk is full.
24. How do you find pods using more memory than they requested?
Join usage with requests from kube-state-metrics on namespace and pod. A ratio above 1 means the pod uses more than its request and is a candidate for eviction under node memory pressure.
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod) (kube_pod_container_resource_requests{resource="memory"})
25. How do you detect CPU throttling?
cAdvisor exposes container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total. The ratio of their rates shows how often a container hit its CPU limit.
26. How do you route alerts for one team to their own Slack channel?
Add a route in Alertmanager that matches a label such as team="payments" and points to a Slack receiver. With the Operator, a team can own an AlertmanagerConfig resource in its namespace, which the Operator merges into the main configuration.
27. How do you stop high-cardinality pod labels from flooding Prometheus?
Do not copy every Kubernetes label onto every series. Keep a short list with labelmap or explicit relabel rules, and drop labels such as pod_template_hash or controller revision hashes that change on each rollout. kube-state-metrics also only exposes Kubernetes labels you allow with its --metric-labels-allowlist flag.
28. How do you monitor HTTP endpoints from outside the pod?
Run blackbox_exporter and define a Probe resource, or a scrape job with the /probe path, that checks URLs for status code, TLS expiry and latency.
Questions for senior engineers
29. How would you design monitoring for many clusters?
Run a Prometheus (or Prometheus in agent mode) in each cluster for local scraping, add an external_labels entry such as cluster="prod-eu-1", and send data to a central store through remote write or a Thanos sidecar. Query globally through Thanos Query or Grafana Mimir, and keep alerting close to the data where possible.
30. Why add external_labels?
So series from different Prometheus servers stay distinguishable after they are combined in a central system, and so Alertmanager can tell which cluster an alert came from.
31. How do you size Prometheus for a large cluster?
Memory grows mostly with the number of active series in the head block, so measure prometheus_tsdb_head_series and scrape volume rather than node count. Reduce series with relabeling, longer scrape intervals for slow-changing targets, and sharding with the Operator's shards field when one server is not enough.
32. What are SLO burn rate alerts?
Alerts that fire when the error budget is being consumed faster than allowed. A multi-window approach, taken from the Google SRE Workbook, combines a short and a long window so alerts are fast for big outages and quiet for small blips. Recording rules for error ratios over several windows keep the queries cheap.
33. How do you monitor the control plane on a managed service like EKS or GKE?
You usually cannot scrape etcd, the scheduler or the controller manager directly. You can still scrape the API server's /metrics through the Kubernetes service and use the provider's own metrics for the managed parts. Check the provider's documentation for what it exposes.
34. How do you keep monitoring available during a cluster outage?
Monitor the cluster from outside as well: a separate Prometheus or a hosted service that runs blackbox probes against ingress endpoints, plus a "dead man's switch" alert that fires all the time and pages when it stops arriving. kube-prometheus-stack ships a Watchdog alert for this purpose.
35. How do you test alerting rules before they reach production?
Write unit tests for rules with promtool test rules, run them in CI alongside promtool check rules, and deploy PrometheusRule objects through the same GitOps flow as application manifests.
36. What are the security concerns with in-cluster Prometheus?
Prometheus can read metadata about every pod and its UI exposes all metrics, which may contain sensitive label values. Restrict access with network policies and authentication in front of the UI, scope RBAC to what discovery needs, and avoid putting secrets or personal data in labels.
Scenario questions
37. A new service's metrics never appear. How do you debug it?
Check in order: does the pod serve /metrics (kubectl port-forward and curl it), does the Service have a named port matching the ServiceMonitor, do the ServiceMonitor's labels match the Prometheus selector, does the namespace match the namespace selector, and does the target appear on the Targets page with an error. Most cases end at a label or port name mismatch.
38. All node_exporter targets in one node pool are down. What do you check?
Whether the DaemonSet pods are scheduled there (taints and tolerations are a common cause), whether port 9100 is reachable from Prometheus (security groups or network policies), and whether the pool's nodes are Ready.
39. Prometheus is OOMKilled after a release. What happened?
A likely cause is a cardinality jump, such as a new metric with a request path or user ID label. Check prometheus_tsdb_head_series over time and the TSDB Status page for the new top metric, drop it with metric_relabel_configs, then fix the instrumentation.
40. Pods are evicted but no alert fired. What would you add?
An alert on node memory and disk pressure conditions from kube-state-metrics (kube_node_status_condition), an alert on pods in the Failed phase with reason Evicted, and dashboards that compare usage with requests so the cause is visible.
41. Your dashboards show CPU usage for a pod as double the expected value. Why?
The query probably counts both the container series and the pod-level cgroup series that cAdvisor also exposes. Filter with container!="" and exclude the POD pause container where it appears.
42. Alerts fire twice, once from each Prometheus replica. How do you fix it?
Point both replicas at the same Alertmanager cluster so it can deduplicate, and make sure the replicas' alert labels are identical. If you use a replica external label, configure Alertmanager or the alert relabeling to drop it before sending.
Where to practise
Set up a kind or minikube cluster, install kube-prometheus-stack, and write one ServiceMonitor and one PrometheusRule for a sample app. Our Prometheus and Grafana guide covers dashboards, and scenario-based Prometheus questions adds more on-call drills. For Kubernetes-side failures, see 15 common Kubernetes pod failures and fixes.
Our Prometheus and Grafana training covers these labs in the Kharadi, Pune classroom and live online. If Kubernetes itself is the gap, the CKA training covers cluster operations.
Frequently asked questions
How do I monitor Kubernetes with Prometheus?
Install kube-prometheus-stack or deploy Prometheus with Kubernetes service discovery, add node_exporter and kube-state-metrics, then add ServiceMonitors for your apps.
What is the difference between kube-state-metrics and metrics-server?
kube-state-metrics reports object state for Prometheus. metrics-server reports current CPU and memory for kubectl top and autoscaling.
What is a ServiceMonitor in Kubernetes?
A Prometheus Operator custom resource that tells Prometheus which Services to scrape and on which port and path.
Why is my ServiceMonitor not picked up?
Its labels or namespace do not match the Prometheus resource's ServiceMonitor selectors, or the port name does not match the Service.
Which metric shows container memory in Kubernetes?
container_memory_working_set_bytes from cAdvisor.
Do I need the Prometheus Operator?
No, but it replaces hand-written scrape config with Kubernetes resources, which suits GitOps and multi-team clusters.
What port does node_exporter use?
9100 by default.
Official references
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0