Scenario-Based ELK Troubleshooting Questions (2027)
24 scenario-based ELK interview questions as incident runbooks: cluster health, watermarks, mapping explosion, Logstash queues and ILM, with API calls.
Scenario-based ELK interview questions give you a symptom and ask for a diagnosis: a yellow cluster, a node at 95% disk, logs missing in Kibana, Logstash dropping events or an ILM policy that never rolls over. This page has 24 incident scenarios, each with the first check, the likely cause and the fix, using Elasticsearch APIs checked against Elastic documentation in November 2026.
The format matches how senior interviewers ask: say what you check first, name the API, state the cause, then say how you stop it recurring. Elasticsearch 9.5 was the latest release when this page was checked. For definitions and fundamentals use the 80+ ELK interview questions, for entry-level questions the freshers guide, and for a study order the 6-week plan. Hands-on labs are part of our ELK Stack training in Pune.
Key takeaways
- Start every cluster incident with
GET _cluster/health, thenGET _cat/shardsandGET _cluster/allocation/explain. Red means an unassigned primary shard, yellow means only replicas are unassigned. - Default disk watermarks are 85% (low), 90% (high) and 95% (flood stage). At flood stage Elasticsearch makes indices on that node read-only, and the block is removed automatically once usage falls below the high watermark.
- HTTP 429 can mean a circuit breaker tripped, a thread pool queue is full, or a flood-stage block. Read the error type before you act.
- Dynamic mapping stops at 1000 fields per index by default (
index.mapping.total_fields.limit). A mapping explosion points to a data-format problem upstream. - Logstash persistent queues are disabled by default, and the dead letter queue only works with the Elasticsearch output. Both are common interview traps.
- Aim for shard sizes of 10GB to 50GB and under 200 million documents per shard to avoid oversharding.
How to answer a scenario question
- Say what you check first and which API gives it.
- Separate symptom from cause. A 429 is a symptom.
- Fix the immediate problem safely, then the root cause.
- Say how you would detect it earlier next time.
| Group | Scenarios | Questions |
|---|---|---|
| Cluster health and capacity | Yellow, red, disk, memory, shards | 1 to 9 |
| Ingestion with Logstash and agents | Missing logs, parsing, queues, duplicates | 10 to 17 |
| Search and Kibana | Slow dashboards, wrong results | 18 to 20 |
| Lifecycle and recovery | ILM, retention, snapshots | 21 to 24 |
Cluster health and capacity
1. The cluster turned yellow after a node restarted. What do you do?
Yellow means no primary is unassigned but some replicas are. Often it recovers on its own as the node rejoins. Check progress and the reason.
GET _cluster/health?filter_path=status,*_shards
GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
GET _cluster/allocation/explainIf the explain output shows a disk watermark or an allocation filter, fix that rather than forcing allocation. A single-node cluster with replicas configured will stay yellow permanently, so set replicas to 0 there.
2. The cluster is red. What is your order of work?
Red means at least one primary shard is unassigned, so searches and writes to that index can fail. List the unassigned primaries, run the allocation explain API for one of them, and fix the stated reason: a missing node, a full disk, or a corrupted shard copy. Restore from a snapshot only if no good copy of the shard exists. Some ILM actions wait for green health, so lifecycle work stalls until you recover.
3. Kibana shows 'Kibana server is not ready yet'.
Kibana needs its system indices to be healthy. Check cluster health first. Elastic documents that a red or yellow status on Kibana's backing system indices, or a flood-stage disk block that affects them, can cause this message. Fix the shard or disk problem, then retry Kibana.
4. Indexing returns 429 and 'read-only-allow-delete block'. Why?
A data node crossed the flood-stage watermark (95% by default), so Elasticsearch set indices with shards on it to read-only. The first check is disk by node.
GET _cat/allocation?v=true
GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk*Free space by deleting old indices, adding disk or nodes, or moving data to a cheaper tier. The block is removed automatically once usage drops below the high watermark. Then find out why: a spike in ingest, too many replicas or oversized shards.
5. Requests fail with circuit_breaking_exception. What does it mean?
A circuit breaker stopped an operation that would have used too much JVM heap. The parent breaker trips at 95% heap by default, and Elastic recommends acting if memory usage is consistently above 85%.
GET _nodes/stats?filter_path=nodes.*.breakersLook for the breaker with the most trips. Large aggregations, huge bulk requests, too many fields or oversharding are common causes. Reduce request size, fix the query, or add memory. Raising the limit hides the problem.
6. Ingestion fails with 'Limit of total fields [1000] has been exceeded'. How do you respond?
The index hit index.mapping.total_fields.limit, which defaults to 1000, because dynamic mapping kept creating fields. This is a mapping explosion. Find the source of the uncontrolled field names, such as a JSON object whose keys are user IDs or timestamps, and fix the data shape or map that object as a single field. Raising the limit treats the symptom and costs heap, slows searches and prolongs startup.
7. One node has high CPU and search latency is up.
Check where the work is and whether queues are rejecting requests.
GET _nodes/hot_threads
GET _cat/thread_pool/search,write?v=true&h=node_name,name,active,queue,rejected
GET _tasks?detailed=true&actions=*searchElastic notes that a node can briefly saturate CPU, but a long period suggests a node sized differently from its tier peers, a workload above capacity, or one expensive task. Cancel the expensive query, then fix the cause.
8. The cluster is slow and has 30,000 small shards. What is the cause?
Oversharding. Every shard has overhead, and too many shards make the cluster unstable and slow. Elastic's guideline is to aim for shard sizes between 10GB and 50GB and fewer than 200 million documents per shard. Fix it by using rollover with ILM, increasing the rollover size, shrinking or force-merging old indices, and reviewing the daily-index-per-service pattern.
9. One data node is full while others are half empty.
Shard distribution is uneven, often because of a hot index or a poor shard count. Elasticsearch balances by shard counts, write load and disk usage, so counts may differ. Check GET _cat/allocation and GET _cat/shards sorted by size. Give the heavy index more primary shards on the next rollover so its load spreads, and avoid pinning shards by hand.
Ingestion with Logstash and agents
10. Logs stopped appearing in Kibana but the app is logging. How do you trace it?
Work along the pipeline. Is the shipper running and sending, does Logstash receive, does Elasticsearch accept, and does the Kibana data view and time range match? On each hop check counts: shipper logs, Logstash pipeline stats, then GET <index>/_count. If documents exist, the problem is the Kibana time range, a data view that does not match the index pattern, or a wrong @timestamp.
11. Documents arrive with the wrong time, so they land outside Kibana's time range.
The event time was not parsed. In Logstash, use the date filter on the field that holds the log time and set the timezone explicitly. Otherwise the ingest time is used, which hides delays and misplaces events during backlogs.
12. Events show a _grokparsefailure tag.
The grok pattern does not match the log line, often after an application changed its log format. Test patterns against real samples, handle optional parts, and send unparsed events to a separate index so you can see how many fail. Parsing at the source with structured JSON logging avoids the problem.
13. Logstash drops events with mapping errors. How do you avoid data loss?
Enable the dead letter queue. When Elasticsearch returns 400 or 404 for a document, which cannot be retried, Logstash writes the event to the DLQ instead of hanging or dropping it. The DLQ is supported only with the Elasticsearch output, so it does not help other outputs. Read the DLQ with the dead_letter_queue input, fix the mapping or event, and re-index. The DLQ also needs size monitoring and clearing.
14. Logstash was killed and events were lost. How do you prevent that?
Enable persistent queues, which store in-flight events on disk and deliver them after restart. They are disabled by default, so set queue.type: persisted in the settings file or in pipelines.yml per pipeline. Limits: inputs that do not acknowledge receipt, such as tcp and udp, cannot be fully protected. The queue is not replicated, so a disk failure still loses data, and NFS is not supported.
15. Logstash is slow and the queue keeps growing. Where is the bottleneck?
Check pipeline stats to see which stage consumes time, then look at the Elasticsearch output. If Elasticsearch returns 429 or is slow to index, scaling Logstash will not help. Tune bulk size, check indexing thread pool rejections and disk, or add data nodes. If a filter is slow, such as a heavy grok or a DNS lookup, fix that first.
16. An agent on one server stopped shipping logs.
Check that the agent process runs, that the log path and permissions are correct, and that the output is reachable and the credentials are valid. File inputs track offsets, so a rotated or truncated file can confuse them if the configuration does not match the rotation scheme. Read the agent's own log first, because it usually states the cause.
17. The same log line appears twice in Elasticsearch.
Retries or replays produced duplicates. Generate a stable ID from the event content, for example with the Logstash fingerprint filter, and use it as the document ID in the Elasticsearch output, so a repeat overwrites instead of adding a document.
Search and Kibana
18. A Kibana dashboard takes a minute to load.
Open the browser developer tools or the Inspect panel to see which query is slow. Common causes are a wide time range, aggregations on high-cardinality fields, many panels each running a heavy query, and shards that are too many or too large. Narrow the default range, pre-aggregate with transforms or rollups where it fits, and fix shard sizing.
19. Searching for an exact value returns too many or too few results.
The field is mapped as text, which is analyzed and split into tokens, but you need exact matching. Use a keyword field for exact filters, aggregations and sorting. Check the mapping and use the _analyze API to see what tokens are indexed.
20. An export fails after page 1000.
Plain from and size paging is limited by index.max_result_window, which is 10,000 by default. Use search_after with a point in time to walk through large results without raising the limit.
Lifecycle and recovery
21. An ILM policy is attached but the index never rolls over.
Ask ILM why.
GET <index>/_ilm/explain
GET _ilm/statusTypical causes: the rollover alias or data stream is not configured, the conditions are never met, ILM is stopped, the cluster is not green so the action waits, or the policy step failed. Fix the stated step and retry. All nodes should run the same version for ILM to behave predictably.
22. Disk fills with old logs even though retention is 30 days.
Check that the delete phase exists in the policy, that new indices actually use the policy through the index template, and that old indices created before the template are covered. Also check whether very large shards or too many replicas inflate usage. Then add alerting at 75% disk so you act before the 85% watermark.
23. An accidentally deleted index must be restored.
If a snapshot exists, list snapshots in the repository and restore the index.
GET _snapshot/_all
POST _snapshot/my_repo/snap_1/_restore
{
"indices": "logs-app-2026.11.01"
}Restore under a new name first if you want to compare. Without a snapshot there is nothing to restore, which is why you schedule snapshots and test restores.
24. How would you stop these incidents from recurring?
Alert on cluster health, disk usage per node, heap pressure, thread pool rejections, ILM errors and Logstash queue size. Review shard sizing at design time, enforce index templates and field limits, run snapshots, and rehearse restores. For the monitoring side see ELK monitoring interview questions and why centralized logging matters, and the Kubernetes pitfalls in 10 Kubernetes logging mistakes.
Frequently asked questions
What do scenario-based ELK interview questions test?
Your diagnosis order and your command of the APIs. Say what you check first, what the cause might be, and how you would prevent it.
What is the difference between a yellow and a red Elasticsearch cluster?
Yellow means all primaries are assigned but some replicas are not. Red means at least one primary shard is unassigned.
What are the default Elasticsearch disk watermarks?
85% low, 90% high and 95% flood stage.
Are Logstash persistent queues on by default?
No. They are disabled by default and enabled with queue.type set to persisted.
What causes a mapping explosion?
Dynamic mapping creating too many fields, often from keys that vary per event. The default limit is 1000 fields per index.
Which shard size does Elastic recommend?
Between 10GB and 50GB per shard and fewer than 200 million documents per shard.
Official references
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0