Grafana Architect Interview Questions: LGTM Stack (2026)

40 architect-level Grafana interview questions: HA with MySQL/Postgres, alerting HA, Loki, Mimir, Tempo, Alloy, cardinality and dashboards as code.

Sep 13, 2025 - 14:52
Nov 12, 2026 - 19:23
 0  50
Grafana Architect Interview Questions: LGTM Stack (2026)

Architect-level Grafana interview questions test whether you can design an observability platform around Grafana, not just build a dashboard. Expect questions on high availability, the Loki, Mimir and Tempo backends, Grafana Alloy for collection, alerting at scale, dashboards as code, access control and cost. This page answers 40 of them, grouped by level.

Grafana is one part of a wider stack, and senior interviews treat it that way. If you are still learning the basics, start with our observability tools overview and come back. If you want troubleshooting stories rather than design questions, the Grafana scenario-based interview questions page is the better fit.

The answers were checked against the Grafana Labs documentation and release pages in November 2026. The latest Grafana release at that time was v13.3.1, published on 10 November 2026.

Key takeaways

  • A highly available Grafana needs a shared MySQL or PostgreSQL database. The embedded SQLite database is not suitable for HA.
  • Grafana Alerting has an HA mode that evaluates rules on every server and sends each notification once. It does not spread the evaluation load across servers.
  • Grafana Agent reached end of life on 1 November 2025. Grafana Alloy, an OpenTelemetry Collector distribution, replaces it for metrics, logs, traces and profiles.
  • Loki stores logs cheaply because it indexes labels, not log content, so label cardinality is the main design decision.
  • Senior candidates are expected to treat dashboards, data sources and alert rules as code and to explain how changes are reviewed.
  • Interviewers care more about trade-offs (cost, cardinality, retention, blast radius) than about menu paths in the UI.

How this page differs from the other Grafana guides

If you needRead
Platform design, scaling and governance questionsThis page
Troubleshooting stories and incident scenariosScenario-based Grafana questions
Day-to-day questions for engineers with 3+ yearsGrafana questions for experienced engineers
Prometheus and Grafana wiring from scratchPrometheus and Grafana together

Foundation questions an architect should answer quickly

1. What does Grafana itself store, and what does it leave to data sources?

Grafana stores its own state: users, teams, folders, dashboards, data source definitions, alert rules and their state, and preferences. The telemetry stays in the data sources. That split is why Grafana can be stateless at the application tier and why the database becomes the thing you back up and replicate.

2. Which database should a production Grafana use?

Use MySQL or PostgreSQL. Grafana ships with SQLite, which is fine for a laptop or a single small instance, but the Grafana HA documentation says it is not suitable once you run more than one server. A managed database also gives you backups and failover without extra work.

3. What do the letters in "LGTM stack" stand for?

Loki for logs, Grafana for visualisation, Tempo for traces and Mimir for metrics. Grafana Labs also publishes Pyroscope for continuous profiling. Each backend is a separate project with its own scaling story, which is why architects are asked about them individually.

4. Why is Loki cheaper to run than a full-text log index?

Loki indexes only the labels attached to each stream and stores the log lines compressed in object storage. Queries first select streams by label, then filter the content. You give up instant full-text search across everything in exchange for much smaller indexes and cheaper storage.

5. What problem does Mimir solve that a single Prometheus does not?

A single Prometheus server keeps data on local disk and scales vertically. Mimir accepts remote-written samples from many Prometheus servers or collectors, stores them in object storage, scales horizontally and supports multiple tenants. It gives you long retention and a global query view.

6. What replaced Grafana Agent?

Grafana Alloy. Grafana Labs moved Agent into long-term support in April 2024 and ended it on 1 November 2025, covering Static mode, Flow mode and the Agent Operator. Alloy is an OpenTelemetry Collector distribution that handles metrics, logs, traces and profiles, and Grafana publishes migration guides from Agent, Promtail and the OpenTelemetry Collector.

7. What is a Grafana organization, and when would you avoid using several?

An organization is an isolated tenant inside one Grafana instance, with its own dashboards, data sources and users. It sounds useful for separation, but sharing anything across organizations is awkward. Most teams get on better with one organization, folders per team and folder-level permissions.

8. How do folders and permissions work together?

Dashboards and alert rules live in folders. You grant viewer, editor or admin rights on a folder to users, teams or service accounts, and the contents inherit them. Design the folder tree around ownership, because that is how you will grant access later.

9. What is a service account in Grafana?

A non-human identity with its own role and tokens. Automation such as Terraform, CI jobs and provisioning scripts should use service account tokens rather than a person's API key, so access survives staff changes and can be revoked on its own.

10. What are exemplars, and why do they matter in a design?

Exemplars attach a trace ID to a metric sample. In Grafana you can click a spike on a latency graph and jump to a trace that caused it. Supporting them means your instrumentation, your metrics backend and your trace backend all have to carry the IDs through.

Design and scaling questions

11. How do you make Grafana highly available?

Run two or more Grafana servers behind a load balancer, point them at the same MySQL or PostgreSQL database and keep them stateless. Grafana uses database-backed auth tokens by default, so the load balancer does not need session affinity. Then enable alerting HA so notifications are not duplicated.

12. What does alerting HA mode actually do?

According to the Grafana documentation, every server evaluates all alert rules and the servers coordinate so that each notification goes out once. Load distribution between servers is not supported. Interviewers ask this to see whether you know HA here means redundancy, not horizontal scaling of evaluation.

13. How would you scale alert evaluation if one Grafana cannot keep up?

Move heavy rules closer to the data. Prometheus-compatible backends such as Mimir and Loki have their own rulers, so recording and alerting rules can run there and Grafana handles routing and notification. Also check for expensive queries, short intervals and rules that could be recording rules.

14. How do you control metric cardinality?

Find the label pairs that create the most series, usually user IDs, request IDs, full URLs or pod names on high-churn workloads. Drop or rewrite them at the collector with relabelling, aggregate with recording rules and set per-tenant series limits in the backend. Cardinality drives memory and cost more than sample rate does.

15. What makes a good Loki label?

A label with a small, bounded set of values that people actually filter on: cluster, namespace, app, environment, level. Anything unbounded, such as trace IDs or user IDs, belongs in the log line or in structured metadata, not in the stream labels.

16. How do you design retention across logs, metrics and traces?

Start from how far back people actually query. Keep raw metrics long enough for capacity planning, keep logs for the period your incident and audit needs require, and keep traces shorter because they are the most voluminous. Object storage lifecycle rules and per-tenant retention settings enforce the policy.

17. Single Mimir tenant or one tenant per team?

Per-team tenants give you separate limits, separate retention and cost attribution, which helps when one team floods the system. A single tenant is simpler to query across. Many platforms use tenants per environment or per business unit and use Grafana data sources to present them.

18. How do you connect metrics, logs and traces in Grafana?

Use consistent labels such as service and namespace across all three, enable exemplars from metrics to traces, add derived fields in the Loki data source to turn trace IDs in log lines into links, and configure trace-to-logs and trace-to-metrics in the Tempo data source.

19. Where should telemetry collection run in Kubernetes?

A common pattern is Alloy as a DaemonSet for node-level logs and metrics, plus a separate Alloy deployment for cluster-wide scraping and for receiving OTLP from applications. Splitting roles keeps one noisy workload from starving node collection.

20. What would you monitor about the monitoring stack itself?

Ingestion rate, rejected samples, ingester memory, query latency, compactor progress, object storage errors and alert notification failures. Send at least the critical meta-alerts through a path that does not depend on the stack being healthy.

Dashboards, alerting and governance as code

21. How do you provision data sources without clicking in the UI?

Put YAML files in Grafana's provisioning directory or manage them with the Grafana Terraform provider. Secrets come from environment variables or a secret store, not from the file in Git.

apiVersion: 1
datasources:
  - name: Mimir
    type: prometheus
    access: proxy
    url: http://mimir-gateway.monitoring.svc/prometheus
    jsonData:
      httpMethod: POST
  - name: Loki
    type: loki
    access: proxy
    url: http://loki-gateway.monitoring.svc

22. How do you keep dashboards under version control?

Store the dashboard JSON in Git and deploy it through file provisioning, the Grafana Terraform provider or Git Sync, which Grafana's documentation now describes as the preferred way to provision dashboards from a Git repository. Changes then go through review like any other code, and you can roll back a broken dashboard.

23. What goes wrong when people edit provisioned dashboards in the UI?

Their change is overwritten on the next sync, or the dashboard drifts from Git if overwriting is disabled. Decide per folder whether the UI or Git owns it and make that visible to users.

24. How would you standardise dashboards across 50 services?

Build one template with variables for service, namespace and environment, generate it from code or a library panel set, and require every service to emit the same core metrics (request rate, errors, duration and saturation). A shared template beats 50 hand-built dashboards.

25. What are contact points and notification policies?

A contact point is where a notification goes: email, Slack, PagerDuty, a webhook and so on. A notification policy is a tree that matches alert labels and routes them to contact points with grouping, timing and mute settings. Good routing depends on good alert labels.

26. How do you reduce alert noise at the design level?

Alert on symptoms users feel, such as error rate or latency against an SLO, rather than on every cause. Group related alerts, use pending periods to skip brief blips, add mute timings for maintenance and give each alert a runbook link and an owner label.

27. How do you test alert rules before they reach production?

Keep rules in Git, run the queries against staging data, check them with backend tooling where you use Prometheus-style rules, and route new rules to a low-priority channel first. Review the firing history after a week before promoting them.

28. How do you manage access for many teams?

Connect Grafana to your identity provider, map IdP groups to Grafana teams, grant folder permissions to teams and use service accounts for automation. Avoid granting the Admin role broadly, because it allows changes to data sources and settings for everyone.

29. How do you stop one team from reading another team's logs?

Grafana folder permissions control dashboards, not raw queries. Enforce isolation in the backend with separate tenants and separate data sources, then restrict who can query each data source. Without that, anyone with Explore access can query everything the data source can see.

30. What would you include in a dashboard review checklist?

Clear title and description, units on every panel, sensible default time range, variables instead of hard-coded labels, no query that scans unbounded cardinality, links to runbooks and related dashboards, and an owner.

Scenario questions for senior and architect roles

31. Dashboards load slowly every Monday morning. How do you approach it?

Check query latency in the backend and the Grafana query inspector for the slowest panels. Monday load usually means many users opening wide time ranges at once. Fix it with recording rules for heavy aggregations, shorter default ranges, query caching in the backend and fewer panels per dashboard.

32. Metric storage costs doubled in a quarter. What do you check?

Active series per tenant and per metric name, then which labels grew. New high-cardinality labels from a deployment are the usual cause. Drop or aggregate them at the collector, set series limits per tenant, and add a review step for new instrumentation.

33. An alert fired twice to PagerDuty during a Grafana rolling update. Why?

The likely cause is that alerting HA was not configured, or the servers could not reach each other to coordinate, so each one sent its own notification. Confirm HA settings and network connectivity between Grafana servers, then check that notification deduplication works during restarts.

34. Leadership wants one Grafana for production and development. What is your design?

One Grafana with separate folders and data sources per environment, or two instances if the security boundary must be hard. Production data sources should be readable only by people who need them. Label everything with environment so dashboards can switch with a variable.

35. You are migrating from Grafana Agent to Alloy across 30 clusters. What is the plan?

Inventory current Agent configs, use Grafana's conversion tooling and migration guides to generate Alloy configuration, run Alloy beside Agent in one non-production cluster and compare series counts and log volume. Roll out cluster by cluster, then remove Agent. Since Agent no longer gets security fixes, set a firm deadline.

36. Logs from a noisy service are slowing Loki for everyone. What do you do?

Confirm the source with ingestion metrics by tenant and stream, then apply per-tenant or per-stream rate limits, drop debug lines at the collector, and check whether someone added a high-cardinality label that is creating thousands of streams.

37. An auditor asks who changed a production alert rule last month. Can you answer?

If rules are managed in Git, the commit history answers it. If they were changed in the UI, you depend on Grafana's version history and audit features available in your edition. That gap is a strong argument for managing production rules as code.

38. A team wants to put customer IDs in metric labels. How do you respond?

Explain the cardinality cost and offer alternatives: put the ID in logs or trace attributes, use exemplars to connect a slow request to its trace, and keep metrics at service or tier level. If they need per-customer numbers, compute them in a reporting system, not the metrics backend.

39. How would you roll out a Grafana major version upgrade?

Read the breaking changes for each version between yours and the target, back up the database, upgrade a staging copy with production dashboards, test plugins, authentication and alerting, then upgrade production servers one at a time. Keep the database backup until you are sure you will not roll back.

40. How would you explain the value of this platform to a finance stakeholder?

Tie it to outcomes they already track: shorter incidents, fewer customer-visible outages and storage spend that is controlled by retention and cardinality limits. Show the cost per tenant so teams see what their telemetry costs.

How to prepare for an architect-level Grafana round

Build a small LGTM stack yourself with Helm or Docker Compose: Grafana, Mimir, Loki, Tempo and Alloy. Break it on purpose. Add a high-cardinality label and watch series counts grow, kill a Grafana server and see what happens to alerts, and change a provisioned dashboard in the UI to see the conflict. Interviewers can tell when an answer comes from doing the work.

Then practise drawing the architecture on a whiteboard in under five minutes: collectors, backends, object storage, Grafana, identity provider and notification targets. Be ready to say where each failure would show up and who would get paged.

For structured practice, the Prometheus and Grafana training covers instrumentation, PromQL, dashboards and alerting with labs. It runs in the Kharadi, Pune classroom and live online for learners in other cities. If you are aiming for the vendor certification route, see our Grafana certification questions page, and for broader DevOps rounds use the 200+ DevOps interview questions guide.

Frequently asked questions

What is the latest Grafana version?

Grafana v13.3.1 was the latest release on GitHub when this page was checked in November 2026. It was published on 10 November 2026 and included several security fixes, so check the release page before you upgrade.

Is SQLite fine for production Grafana?

Only for a single small instance. For high availability, Grafana's documentation requires a shared MySQL or PostgreSQL database because SQLite cannot be shared safely between servers.

Does Grafana alerting HA spread load across servers?

No. In HA mode every server evaluates all rules and the servers coordinate so each notification is sent once. Grafana's documentation states that load distribution between servers is not supported.

Is Grafana Agent still supported?

No. Grafana Agent reached end of life on 1 November 2025 and no longer receives security or bug fixes. Grafana Labs recommends Grafana Alloy as the replacement.

What is the difference between Loki and Mimir?

Loki stores logs and indexes only their labels. Mimir stores Prometheus-format metrics for the long term and scales horizontally. Both use object storage and both are queried from Grafana.

Do I need Kubernetes experience for a senior Grafana role?

Usually yes. Most LGTM deployments run on Kubernetes, and questions about collectors, labels and scaling assume you know pods, namespaces, DaemonSets and Helm.

How many Grafana questions should I prepare for an architect interview?

Fewer, deeper answers work better. Be able to talk for several minutes about HA, cardinality, retention, access control and alerting design, with an example from a lab or project you built.

Official references

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.