50+ Real-Time ELK Stack Interview Questions (2027)

52 real-time ELK interview questions: refresh and translog, Beats and Agent, Logstash queues, ingest pipelines, data streams, ILM and Kibana alerting.

Sep 17, 2025 - 10:09
Nov 9, 2026 - 10:03
 0  204
50+ Real-Time ELK Stack Interview Questions (2027)

Real-time ELK interview questions focus on the path from an event to a searchable, alertable document: how Beats or Elastic Agent ship logs, how Logstash and ingest pipelines process them, why Elasticsearch search is near real time (refresh), how data streams and ILM handle constant writes, and how Kibana rules raise alerts within seconds or minutes.

This page has 52 numbered questions on that real-time path. For the general ELK question bank covering mappings, shards, queries and Kibana basics, use our main guide, 80+ ELK Stack interview questions and answers. Incident-style practice is in scenario-based ELK questions, and cluster tuning is in advanced ELK questions. Facts below were checked against the Elastic documentation in November 2026.

Key takeaways

  • Elasticsearch is near real time: a document becomes searchable after the next refresh, which happens every 1 second by default on indices that are being searched.
  • A write is durable once it is in the translog, even before a refresh makes it visible to search.
  • Data streams plus ILM rollover are the standard design for append-only logs and metrics arriving continuously.
  • Backpressure flows backwards: if Elasticsearch rejects bulk requests, Logstash slows, and Beats or Agent slow their reads. Persistent queues or Kafka absorb bursts.
  • Kibana alerting rules and connectors turn indexed data into notifications; latency is the sum of shipping, processing, refresh and the rule's check interval.

The real-time path and where latency comes from

StageComponentMain latency or loss riskSetting to know
CollectFilebeat, Elastic AgentFile scan frequency, backpressure, registry statefilestream input options, output bulk size
BufferKafka or Logstash persistent queueDisk full, consumer lagqueue.type: persisted, consumer groups
ProcessLogstash or ingest pipelinesSlow grok, too few workerspipeline.workers, pipeline.batch.size
IndexElasticsearchBulk rejections (429), slow disksWrite thread pool, shard count
Make searchableRefreshRefresh intervalindex.refresh_interval
Alert or viewKibana rules, DiscoverRule check interval, lookback windowRule schedule, connectors

Real-time ELK interview questions for freshers

1. What does "near real time" mean in Elasticsearch?

Indexed documents are not searchable instantly. They become visible after a refresh, which opens a new searchable segment. With the default 1-second refresh interval, there is a short delay of about a second between indexing and search.

2. What is the default refresh interval?

1 second, set by index.refresh_interval. If the setting is left at the default and an index receives no search requests for 30 seconds (index.search.idle.after), its shards go search idle and skip refreshes until the next search arrives.

3. What is the difference between refresh and flush?

Refresh makes recent writes searchable by creating new segments in memory and the file system cache. Flush performs a Lucene commit, writing segments durably to disk and trimming the translog. Refresh is about visibility; flush is about durability and recovery time.

4. What is the translog?

A per-shard transaction log. Every index or delete operation is written to it before Elasticsearch acknowledges the request. If a node crashes before a flush, Elasticsearch replays the translog on restart. By default (index.translog.durability: request) it is fsynced before each request is acknowledged.

5. What are the roles of Elasticsearch, Logstash and Kibana in a live logging setup?

Logstash (or an ingest pipeline) parses and enriches events, Elasticsearch stores and indexes them, and Kibana lets people search, build dashboards and define alert rules. Beats or Elastic Agent usually collect the data at the source.

6. What is Filebeat?

A lightweight shipper that tails log files and sends events to Elasticsearch, Logstash or Kafka. It records how far it has read each file in a registry, so it can resume after a restart without resending everything.

7. What is Elastic Agent and how does it relate to Beats?

Elastic Agent is a single agent that can collect logs, metrics and security data using integrations. It can be managed centrally through Fleet in Kibana. Beats are the older single-purpose shippers; both remain supported, but Elastic directs new deployments towards Agent and integrations.

8. Which default ports should you know?

PortUsed by
9200Elasticsearch REST API
9300Elasticsearch transport (node to node)
5601Kibana
5044Logstash Beats input (common convention)
9600Logstash monitoring API

9. What is the Bulk API and why is it used for streaming data?

POST _bulk sends many index operations in one request. It cuts per-request overhead, so every shipper and Logstash output uses it. The right batch size depends on document size and cluster, so test rather than guess.

10. What is an ingest pipeline?

A sequence of processors that Elasticsearch runs on ingest nodes before indexing a document, for example grok, dissect, date, geoip, set and rename. It is a lighter alternative to Logstash for straightforward parsing.

11. What is a data stream?

A named, append-only abstraction over a series of hidden backing indices, designed for time series data such as logs. You write to the stream name; Elasticsearch writes to the current backing index and rolls over to a new one based on ILM or data stream lifecycle. Every document needs an @timestamp field.

12. What is the naming scheme for Elastic logs data streams?

type-dataset-namespace, for example logs-nginx.access-production. Integrations use this scheme so that templates, permissions and lifecycle policies can match on patterns.

13. How do you watch new logs arriving in Kibana?

In Discover, set a relative time range such as "Last 15 minutes" and turn on auto refresh with an interval. Dashboards have the same refresh control. Data appears once it has been indexed and refreshed.

14. What is a Kibana alerting rule?

A rule runs on a schedule, checks a condition (for example an Elasticsearch query returning more than N hits, or a metric threshold) and, when the condition is met, runs actions through connectors such as email, Slack, PagerDuty or a webhook.

15. What is grok?

A pattern language that turns unstructured text into fields using named regular expressions such as %{IP:client.ip}. It is available in Logstash and as an ingest processor. It is flexible but CPU heavy, so prefer dissect when the format has fixed delimiters.

16. Why is @timestamp important in real-time pipelines?

Kibana time filters, data streams, ILM and alerting rules all use it. If you leave it as the ingest time instead of parsing the event's own time, delayed events appear in the wrong place and alerts can miss them. Parse it with a date filter or processor.

Questions for 2 to 5 years of experience

17. How do you make a document searchable immediately after indexing?

Add ?refresh=wait_for to the index request, which waits until the next refresh before returning, or ?refresh=true, which forces one. Forcing refreshes on every write hurts indexing throughput, so use them only in tests or low-volume flows.

18. When would you increase refresh_interval?

For heavy ingest where a few seconds of extra delay is acceptable, for example 5s or 30s for bulk log indexing. Fewer refreshes mean fewer small segments and less merge work. Set it to -1 during a one-off bulk load, then restore it.

PUT logs-app-default/_settings
{ "index": { "refresh_interval": "10s" } }

19. How do you set up a data stream from scratch?

Create an index template whose pattern matches the stream name and that contains "data_stream": {}, optionally with a lifecycle policy and mappings. Then index a document with @timestamp to the stream name, or create it with PUT _data_stream/<name>.

PUT _index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": { "index.lifecycle.name": "logs-app-policy" }
  }
}

20. Why can you not update documents in a data stream with a normal index request?

Data streams are append-only and accept only create operations. To change existing documents you use update by query or delete by query on the stream, or target the specific backing index.

21. What does ILM rollover do for streaming data?

It creates a new backing index when the current one reaches a condition such as primary shard size, document count or age. This keeps shards at a manageable size while writes continue without interruption. Later phases (warm, cold, frozen, delete) move or remove older indices.

22. How do Logstash workers and batch size affect throughput and latency?

pipeline.workers (default: number of CPU cores) sets how many threads run filters and outputs. pipeline.batch.size (default 125) is how many events each worker collects before processing, and pipeline.batch.delay (default 50 ms) is how long it waits for a batch to fill. Bigger batches raise throughput but add latency and memory use.

23. What is a Logstash persistent queue?

With queue.type: persisted, Logstash stores events on disk between inputs and the pipeline, so a restart or a slow output does not lose in-flight events, and short bursts are absorbed. The default is an in-memory queue, which loses events if the process crashes.

24. What is the Logstash dead letter queue?

A place where Logstash writes events that the Elasticsearch output could not index because of mapping or document errors (HTTP 400 or 404 responses). Enable it with dead_letter_queue.enable: true and reprocess with the dead_letter_queue input after fixing the cause.

25. How does backpressure work in the Elastic Stack?

When Elasticsearch is overloaded, it rejects bulk requests with 429 errors. Logstash and Beats retry with backoff and slow down their reads. Beats stop reading files further, so data waits on disk at the source instead of being dropped. A buffer such as Kafka lets producers continue while consumers catch up.

26. When do you put Kafka between shippers and Logstash?

When producers cannot tolerate backpressure, when you need to replay data, when several consumers read the same stream, or when ingest volume is bursty. Kafka adds operational work, so it is not needed for small or steady setups.

27. How do you test an ingest pipeline before using it?

Use the simulate API with sample documents and check the output and any errors.

POST _ingest/pipeline/nginx-access/_simulate
{
  "docs": [ { "_source": { "message": "203.0.113.5 - - [07/Nov/2026:10:00:01 +0000] \"GET / HTTP/1.1\" 200 512" } } ]
}

28. How do you handle failures inside an ingest pipeline?

Add on_failure handlers at processor or pipeline level, for example to set an error field and route the document to a separate index with set on _index. Use ignore_failure only for processors whose failure really does not matter.

29. Logstash or ingest pipeline: how do you choose?

NeedIngest pipelineLogstash
Simple parsing and enrichmentGood fitWorks, but adds a tier
Many inputs and outputs (Kafka, S3, JDBC)NoGood fit
Disk bufferingNoPersistent queue
Complex conditional routingLimitedGood fit
Operational overheadNone extraSeparate JVM service

30. How do you monitor Logstash pipeline health in real time?

Query the monitoring API on port 9600, for example GET _node/stats/pipelines, to see events in and out, queue size and per-plugin durations. Stack Monitoring in Kibana charts the same data, which helps you find a slow filter.

31. How do you detect indexing pressure in Elasticsearch?

Check GET _cat/thread_pool/write?v for queue and rejected counts, GET _nodes/stats/indexing_pressure for memory used by in-flight writes, and node CPU, disk I/O and merge activity. Rising rejections mean shippers are being slowed.

32. What are the main types of Kibana rule you would use for log alerting?

The Elasticsearch query rule (run a query or ES|QL and alert on hit count), the log threshold rule in Observability, and anomaly detection rules if machine learning is licensed. Watcher still exists but Kibana alerting is the recommended route for new rules.

33. How do you avoid duplicate alerts for one ongoing problem?

Use action frequency settings such as "on status change" or a throttle interval, group alerts by a field (for example host.name) so each entity has its own alert, and use recovery actions so responders know when it cleared.

34. What is ES|QL and why does it matter for live investigation?

ES|QL is Elasticsearch's piped query language, generally available since 8.14. It lets you filter, aggregate and transform in one query (for example FROM logs-* | WHERE http.response.status_code >= 500 | STATS count = COUNT(*) BY service.name) and can be used in Discover and in alert rules.

Senior real-time ELK interview questions

35. How do you design an ELK pipeline for sub-minute alerting at high volume?

Ship with Agent or Beats to Kafka, consume with horizontally scaled Logstash using persistent queues, write to data streams with sensible primary shard counts and a refresh interval of a few seconds, and run alert rules every minute with a lookback slightly larger than the expected end-to-end delay. Measure that delay by comparing @timestamp with event.ingested.

36. What is event.ingested and how do you use it?

An ECS field holding the time the event reached Elasticsearch, usually set by an ingest pipeline with {{_ingest.timestamp}}. Subtracting @timestamp from it gives ingest lag per source, which you can chart and alert on.

37. How do you size shards for continuous log ingest?

Aim for shard sizes that Elastic's guidance recommends (tens of gigabytes, with rollover by primary shard size), spread write load across hot nodes with enough primaries, and avoid many tiny daily indices. Too many small shards waste heap and slow cluster state updates.

38. How would you set up hot-warm-cold tiers for streaming logs?

Assign node roles data_hot, data_warm, data_cold (and optionally data_frozen), keep new backing indices on fast hot nodes, and let ILM move them after rollover. Searchable snapshots can back cold and frozen tiers to cut storage cost.

39. How do you keep mappings stable when log formats change?

Use index templates with explicit mappings for known fields, ECS naming, and dynamic templates for the rest. Consider "dynamic": "runtime" or runtime fields for rarely used fields. Mapping conflicts send events to the dead letter queue or reject them, so version your templates in Git.

40. How do you protect a real-time pipeline from a log flood by one noisy service?

Route that service to its own data stream or Kafka topic, rate-limit or sample at the source, drop debug-level events in processing, and set alerts on ingest volume per dataset. This keeps one team's bug from delaying everyone else's logs.

41. How do you secure data in transit in this pipeline?

Security is enabled by default since Elasticsearch 8.0, with TLS on the HTTP and transport layers. Use TLS from Beats or Agent to Logstash and Kafka as well, API keys or service tokens with least privilege for each shipper, and separate roles for writing and reading.

42. What changes when you move from self-managed ELK to Elastic Cloud or Serverless?

You stop operating nodes, shards and upgrades yourself, and Serverless also manages scaling and refresh behaviour. You still own data modelling, ingest pipelines, retention and alert design. Check which settings are not exposed in the managed offering before migrating.

Scenario-based real-time ELK questions

43. Logs show up in Kibana five minutes late. How do you find the delay?

Compare @timestamp, event.ingested and the shipper's own timestamps to find which stage adds the delay. Check Filebeat or Agent output errors, Kafka consumer lag, Logstash queue size and event rates, Elasticsearch write rejections and the index refresh interval. Also check time zone parsing: a wrong offset looks like lag.

44. Kibana Discover shows nothing for the last minutes, but Filebeat says it is sending. What next?

Confirm documents are reaching the right index or data stream with a count query, check for mapping errors in Filebeat or Logstash logs, check that the data view pattern matches the target, and confirm @timestamp is parsed correctly. Events with a future or past timestamp fall outside the time picker.

45. Elasticsearch returns 429 errors during a traffic spike. What do you do now and later?

Now: let shippers back off, make sure buffers (persistent queue, Kafka) have room, and temporarily raise refresh interval on the busiest streams. Later: add hot nodes or primaries, reduce expensive processing, review shard balance, and alert on write thread pool rejections before users notice.

46. A Logstash node restarts and you fear lost events. How should it have been configured?

With a persistent queue sized for expected outage time, so in-flight events survive the restart, and with Kafka or Beats backpressure upstream. Without them, the in-memory queue's contents are gone and you can only resend from source files if they still exist.

47. An alert for "more than 50 errors in 5 minutes" fires late or not at all. Why?

The rule's lookback window may be shorter than ingest delay, so late documents fall outside it. Fix by widening the window, querying on event.ingested where appropriate, or reducing lag. Also verify the rule's schedule, the query filter and that the connector works.

48. A new deployment changed the log format and parsing failures spike. How do you respond?

Route failures to an errors index or DLQ so no data is lost, update the grok or dissect pattern and test it with the simulate API, deploy the fix, and reprocess the failed events. Add a check in the application's CI that validates sample logs against the pipeline.

49. Disk on hot nodes fills up quickly and indices go read-only. What happened and how do you fix it?

Elasticsearch blocks writes when a node passes the flood-stage disk watermark. Free space or add capacity, check that ILM rollover and phase moves are running (GET <index>/_ilm/explain), shorten hot phase retention, and the write block is released automatically once usage drops below the high watermark.

50. Security wants authentication failures visible within one minute. Sketch the design.

Collect auth logs with Elastic Agent's system integration, parse them in its ingest pipeline into ECS fields such as event.outcome, write to a dedicated data stream with a 1-second refresh, and create a rule that runs every minute grouped by user.name and source.ip, sending to the on-call channel.

51. A Kibana dashboard on live data is slow to load. How do you speed it up?

Shorten the default time range, reduce the number of panels and high-cardinality terms aggregations, use runtime fields sparingly, and consider transforms or downsampling for long-range views. Check whether searches hit warm or cold tiers unnecessarily.

52. Two Filebeat instances read the same file and you see duplicates. How do you prevent them?

Each file should be read by one Filebeat with its own registry. For safety, generate a deterministic document ID (for example with the fingerprint processor) so Elasticsearch rejects duplicates on create. Duplicates also appear after registry loss, which the same ID strategy covers.

How to prepare

  • Run a local stack, ship one log file with Filebeat, and measure the delay between writing a line and seeing it in Discover.
  • Change refresh_interval, enable a persistent queue, and break a mapping on purpose to see the dead letter queue fill.
  • Build one Elasticsearch query rule and one ES|QL query in Discover.
  • Read the main ELK interview preparation guide for a study plan and the real-time logging tools roundup to compare ELK with alternatives.

Our ELK Stack training covers Beats, Logstash, Elasticsearch and Kibana alerting with hands-on labs, in the Kharadi, Pune classroom or live online. For the wider picture, see the observability tools hub.

Frequently asked questions

Is Elasticsearch real time?

It is near real time. Documents become searchable after the next refresh, which happens every second by default on actively searched indices.

How can I reduce the delay before logs appear in Kibana?

Find which stage adds delay by comparing @timestamp with event.ingested, then fix that stage: shipper backpressure, Kafka lag, slow Logstash filters, Elasticsearch rejections or a long refresh interval.

What is the difference between refresh and flush?

Refresh makes recent writes visible to search. Flush commits segments to disk and trims the translog for durability and faster recovery.

Should I use Logstash or ingest pipelines?

Use ingest pipelines for straightforward parsing and enrichment. Use Logstash when you need many input or output types, disk buffering or complex routing.

Why use Kafka with the ELK Stack?

Kafka buffers bursts, lets you replay data and lets several consumers read the same stream, so producers keep running when Elasticsearch slows down.

How do Kibana alerts work?

A rule checks a condition on a schedule, such as an Elasticsearch query or threshold, and when it matches it sends actions through connectors such as email, Slack or webhooks.

How is this page different from the main ELK interview guide?

This page covers only the real-time path: shipping, buffering, processing, refresh, data streams and alerting. The main 80+ guide covers the whole ELK Stack at every level.

Official references

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.