20 Log Aggregation Tools by Pipeline Layer

20 log aggregation and analysis tools mapped to pipeline layers: collectors, processors, storage, query and SaaS, with three reference architectures.

Dec 16, 2025 - 16:23
Nov 9, 2026 - 07:37
 0  89
20 Log Aggregation Tools by Pipeline Layer

Log aggregation tools fall into four layers: collectors and shippers (Fluent Bit, Vector, OpenTelemetry Collector), processors (Logstash, Fluentd), storage and query engines (Elasticsearch, OpenSearch, Grafana Loki, ClickHouse) and analysis front ends or SaaS platforms (Kibana, Grafana, Splunk, Datadog). Choose one tool per layer based on log volume, query style and who will run it.

This guide is about building the pipeline: which of 20 tools fits which layer, and three reference architectures you can copy. If you want a feature-by-feature table of logging products instead, read 20 DevOps logging tools compared feature-wise. For cloud-provider options in particular, see top 10 logging tools for cloud environments.

Key takeaways

  • Aggregation is a pipeline: collect, parse and enrich, buffer, store, then query. Most problems come from skipping the buffer or the parsing step.
  • Fluent Bit and Vector are light agents for every node; Logstash and Fluentd are heavier processors you run centrally.
  • Elasticsearch and OpenSearch index the full text of logs; Grafana Loki indexes only labels, which costs less but makes ad hoc text search slower.
  • The OpenTelemetry Collector lets you collect logs with the same agent used for metrics and traces.
  • SaaS platforms (Splunk, Datadog, Sumo Logic, cloud-native logging) remove storage operations but charge by ingest or usage, so filter before you ship.
  • Decide retention and labels before you pick tools; they drive cost more than the product does.

The log aggregation pipeline

Every centralized logging setup, from a three-server startup to a Kubernetes platform, has the same stages:

  1. Collect. Read files, container stdout, journald or syslog on each host.
  2. Parse and enrich. Turn text lines into fields (timestamp, level, service) and add metadata such as Kubernetes pod and namespace.
  3. Buffer. Hold data on disk or in a queue such as Kafka so a storage outage does not lose logs.
  4. Store and index. Write to a search engine, a label-indexed store, a columnar database or a SaaS backend.
  5. Query, visualise, alert. Search, build dashboards and trigger alerts on patterns.

Our explainer on why teams should adopt centralized logging early covers the reasons; this post covers the parts.

#ToolLayerLicence or model
1Fluent BitCollector / shipperOpen source (CNCF graduated, under Fluentd)
2FluentdCollector / aggregatorOpen source (CNCF graduated)
3VectorCollector / processorOpen source (maintained by Datadog)
4OpenTelemetry CollectorCollector / processorOpen source (CNCF)
5Filebeat / Elastic AgentCollector / shipperElastic
6rsyslogCollector / forwarderOpen source
7LogstashProcessorElastic
8Apache KafkaBuffer / transportOpen source (Apache)
9ElasticsearchStorage and searchElastic (includes an AGPL option)
10OpenSearchStorage and searchOpen source (Apache 2.0)
11Grafana LokiLabel-indexed storageOpen source (AGPLv3)
12ClickHouseColumnar storageOpen source (Apache 2.0)
13GraylogStorage + UI platformOpen and commercial editions
14KibanaQuery and visualisationElastic
15GrafanaQuery and visualisationOpen source (AGPLv3)
16SplunkSaaS / self-managed platformCommercial
17Datadog Log ManagementSaaS platformCommercial
18Sumo LogicSaaS platformCommercial
19Amazon CloudWatch LogsCloud-native serviceAWS, pay per use
20Google Cloud Logging / Azure Monitor LogsCloud-native servicesGCP / Azure, pay per use

Item 20 groups the Google Cloud and Azure services because they play the same role as CloudWatch Logs on their own clouds.

Layer 1: collectors and shippers

1. Fluent Bit

Fluent Bit is a lightweight log processor and forwarder written in C, part of the Fluentd project under the CNCF. It is the usual choice as a Kubernetes DaemonSet: its tail input reads container logs and the kubernetes filter adds pod, namespace and label metadata. It can output to Elasticsearch, OpenSearch, Loki, Kafka, Splunk HEC, CloudWatch and many others.

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  docker, cri
    Tag               kube.*

[FILTER]
    Name    kubernetes
    Match   kube.*

[OUTPUT]
    Name    loki
    Match   kube.*
    Host    loki.logging.svc
    Port    3100
    Labels  job=fluent-bit

2. Fluentd

Fluentd is the older, Ruby-based member of the same project, with a large plugin ecosystem. Teams often run Fluent Bit on every node and forward to a smaller pool of Fluentd aggregators that do heavy routing, buffering and fan-out to several destinations.

3. Vector

Vector is a high-performance observability data pipeline written in Rust. Datadog acquired the company behind it, Timber, in 2021, and Vector remains open source. It runs as an agent or aggregator, and its Vector Remap Language (VRL) handles parsing and field changes in one place.

sources:
  app:
    type: file
    include: ["/var/log/app/*.log"]
transforms:
  parse:
    type: remap
    inputs: ["app"]
    source: |
      . = parse_json!(.message)
sinks:
  os:
    type: elasticsearch
    inputs: ["parse"]
    endpoints: ["https://opensearch.example.internal:9200"]

4. OpenTelemetry Collector

The OpenTelemetry Collector receives, processes and exports telemetry. With the filelog receiver (in the contrib distribution) it reads log files, and it can export to OTLP backends, Loki, Elasticsearch, Splunk and SaaS vendors. Using one collector for logs, metrics and traces means fewer agents per node and consistent resource attributes across signals.

5. Filebeat and Elastic Agent

Filebeat is Elastic's lightweight shipper for log files, with modules for common formats such as Nginx and system logs. Elastic Agent is Elastic's newer unified agent, managed centrally through Fleet, and is Elastic's recommended path for new deployments. Both ship directly to Elasticsearch or through Logstash.

6. rsyslog

rsyslog is the default syslog daemon on many Linux distributions, including RHEL. It can forward logs over TCP or RELP with disk-assisted queues, which makes it a dependable shipper for servers and network devices without installing anything new. See our RHEL 10 log management guide for journald and rsyslog settings.

Layer 2: processing and buffering

7. Logstash

Logstash is the processing stage of the classic ELK stack. Its pipelines have inputs, filters and outputs; the grok filter parses unstructured text with named patterns, and persistent queues protect against data loss on restart. It uses more memory than node agents, so run it centrally.

input  { beats { port => 5044 } }
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output { elasticsearch { hosts => ["https://es01:9200"] } }

8. Apache Kafka

Kafka is not a logging tool by itself, but in large pipelines it sits between shippers and storage as a durable buffer. If Elasticsearch is slow or down, logs wait in Kafka topics instead of filling agent buffers. It also lets several consumers (search, security analytics, archive to object storage) read the same stream.

Layer 3: storage and query engines

9. Elasticsearch

Elasticsearch is a distributed search engine built on Apache Lucene that indexes the full content of each log event. It is strong for free-text search and aggregations. Index lifecycle management (ILM) moves data through hot, warm, cold and delete phases to control cost. In 2024 Elastic added AGPLv3 as a licence option for the source code alongside its existing licences.

10. OpenSearch

OpenSearch is the Apache 2.0 fork of Elasticsearch and Kibana created in 2021, now governed by the OpenSearch Software Foundation under the Linux Foundation. It offers similar search and dashboards (OpenSearch Dashboards) and index state management for retention. Amazon OpenSearch Service is the managed AWS offering.

11. Grafana Loki

Loki stores log lines compressed in object storage and indexes only a small set of labels, such as namespace, app and level. That keeps storage and index costs low. Queries in LogQL first select streams by label and then filter text:

{namespace="payments", app="api"} |= "timeout" | json | duration > 2s

The trade-off is that high-cardinality labels (user IDs, request IDs) hurt performance, and searching all logs for a random string is slower than in a full-text index.

12. ClickHouse

ClickHouse is a column-oriented SQL database. Some teams store logs in ClickHouse tables because columnar compression and SQL aggregation work well for very large, structured log volumes. You design the schema yourself, so it suits teams with data engineering skills.

13. Graylog

Graylog is a log management platform that bundles ingestion (GELF, syslog, Beats inputs), processing rules, search, dashboards and alerts. It uses OpenSearch (or its own Graylog Data Node) for storage and MongoDB for configuration. It suits teams that want one product instead of assembling ELK pieces.

Layer 4: analysis and visualisation

14. Kibana

Kibana is the front end for Elasticsearch: Discover for searching, Lens and dashboards for charts, and alerting rules. Most ELK interview questions focus on Kibana index patterns (now called data views) and KQL queries.

15. Grafana

Grafana queries many data sources, including Loki, Elasticsearch, OpenSearch, ClickHouse and CloudWatch. Its main advantage for logs is correlation: a panel showing a latency spike from Prometheus can link to the Loki logs for the same time range and labels. Our Prometheus and Grafana guide covers the metrics side.

SaaS and cloud-native platforms

16. Splunk

Splunk indexes machine data and is queried with SPL (Search Processing Language). Universal forwarders send data to indexers, and the HTTP Event Collector (HEC) accepts events over HTTPS. Splunk is available as Splunk Cloud Platform or Splunk Enterprise. Cisco completed its acquisition of Splunk in March 2024.

17. Datadog Log Management

Datadog collects logs through its Agent or integrations, parses them with pipelines and processors, and links them to traces and metrics through unified service tags (env, service, version). Datadog separates ingesting from indexing: you can ingest everything, index only what you search often, and send the rest to archives in your own storage.

18. Sumo Logic

Sumo Logic is a cloud-native log analytics and SIEM platform. Collectors and the Sumo Logic Kubernetes collection (based on OpenTelemetry) send data in, and a query language with operators such as parse, where and count by analyses it.

19. Amazon CloudWatch Logs

CloudWatch Logs stores logs from AWS services, Lambda and the CloudWatch agent in log groups with configurable retention. CloudWatch Logs Insights runs queries with commands like fields, filter and stats. Subscription filters stream logs to Kinesis, Lambda or OpenSearch. Read our CloudWatch Logs Insights tutorial for query examples.

20. Google Cloud Logging and Azure Monitor Logs

Google Cloud Logging collects logs from GCP services and GKE, with log buckets for retention, the Logs Explorer for queries and log sinks to route data to BigQuery, Cloud Storage or Pub/Sub. Azure Monitor Logs stores data in Log Analytics workspaces and is queried with KQL (Kusto Query Language). Both are the low-effort option when your workloads live on that cloud.

Three reference architectures

ScenarioPipelineWhy
Kubernetes, cost-sensitiveFluent Bit DaemonSet → Loki (object storage) → GrafanaLow storage cost, label queries match Kubernetes metadata
Search-heavy, security or auditElastic Agent or Fluent Bit → Kafka → Logstash → Elasticsearch/OpenSearch → Kibana/DashboardsFull-text search, buffering for spikes, rich parsing
Small team, no ops capacityVendor agent or OpenTelemetry Collector → Datadog, Splunk Cloud or cloud-native loggingNo storage cluster to run; pay per ingest

For Kubernetes-specific choices, our 12 Kubernetes logging tools post goes deeper.

How to choose

  • Volume. Estimate GB per day. Full-text indexes grow storage several times faster than label indexes.
  • Query style. Mostly "show logs for service X around time T"? Loki fits. Frequent free-text hunts across everything? Elasticsearch or OpenSearch.
  • Retention and compliance. Audit logs may need a year or more; debug logs rarely need more than a few weeks. Split them.
  • Team skills. Running Elasticsearch clusters needs JVM, shard and index skills. SaaS shifts that to the vendor for a fee.
  • Format discipline. Structured JSON logs make every tool cheaper and faster. Fix the application logging format before buying anything.

Common mistakes

  • Sending every debug line to a paid backend instead of filtering at the collector.
  • Putting request IDs or user IDs into Loki labels, which explodes stream count.
  • No disk buffer on agents, so a backend outage loses logs.
  • Logging secrets or personal data. Redact at the collector with filters or VRL.
  • Clocks out of sync, which scrambles event order. Run NTP or chrony everywhere.

Learn it hands-on

Our ELK Stack training in Pune builds a full pipeline with Beats, Logstash, Elasticsearch and Kibana, in the Kharadi classroom or live online. If you are preparing for interviews, the 80+ ELK Stack interview questions post is a good next step.

FAQ

What is log aggregation in DevOps?

Log aggregation is collecting logs from all servers, containers and services into one central system where they are parsed, stored, searched and used for dashboards and alerts.

What is the difference between Fluent Bit and Fluentd?

Both belong to the same CNCF project. Fluent Bit is written in C and is lightweight, so it runs on every node. Fluentd is Ruby-based with a larger plugin set and is often used as a central aggregator.

Is Loki a replacement for Elasticsearch?

For many Kubernetes teams, yes, because Loki is cheaper to run. But Loki indexes only labels, so free-text search across all logs is slower than in Elasticsearch or OpenSearch.

Do I need Kafka in a logging pipeline?

Not for small setups. Kafka helps at high volume, when you need a durable buffer during storage outages or want several consumers to read the same log stream.

What is the ELK stack versus the EFK stack?

ELK is Elasticsearch, Logstash and Kibana. EFK replaces Logstash with Fluentd or Fluent Bit, which is common on Kubernetes because the agents are lighter.

Can the OpenTelemetry Collector handle logs?

Yes. The filelog receiver in the contrib distribution reads log files, and the collector can export logs to OTLP backends and many vendors, alongside metrics and traces.

How do I control logging costs?

Filter and sample at the collector, drop debug logs in production, set retention per log type, avoid high-cardinality labels in Loki and, on SaaS platforms, index only the logs you search often.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.