20 Log Aggregation Tools by Pipeline Layer
20 log aggregation and analysis tools mapped to pipeline layers: collectors, processors, storage, query and SaaS, with three reference architectures.
Log aggregation tools fall into four layers: collectors and shippers (Fluent Bit, Vector, OpenTelemetry Collector), processors (Logstash, Fluentd), storage and query engines (Elasticsearch, OpenSearch, Grafana Loki, ClickHouse) and analysis front ends or SaaS platforms (Kibana, Grafana, Splunk, Datadog). Choose one tool per layer based on log volume, query style and who will run it.
This guide is about building the pipeline: which of 20 tools fits which layer, and three reference architectures you can copy. If you want a feature-by-feature table of logging products instead, read 20 DevOps logging tools compared feature-wise. For cloud-provider options in particular, see top 10 logging tools for cloud environments.
Key takeaways
- Aggregation is a pipeline: collect, parse and enrich, buffer, store, then query. Most problems come from skipping the buffer or the parsing step.
- Fluent Bit and Vector are light agents for every node; Logstash and Fluentd are heavier processors you run centrally.
- Elasticsearch and OpenSearch index the full text of logs; Grafana Loki indexes only labels, which costs less but makes ad hoc text search slower.
- The OpenTelemetry Collector lets you collect logs with the same agent used for metrics and traces.
- SaaS platforms (Splunk, Datadog, Sumo Logic, cloud-native logging) remove storage operations but charge by ingest or usage, so filter before you ship.
- Decide retention and labels before you pick tools; they drive cost more than the product does.
The log aggregation pipeline
Every centralized logging setup, from a three-server startup to a Kubernetes platform, has the same stages:
- Collect. Read files, container stdout, journald or syslog on each host.
- Parse and enrich. Turn text lines into fields (timestamp, level, service) and add metadata such as Kubernetes pod and namespace.
- Buffer. Hold data on disk or in a queue such as Kafka so a storage outage does not lose logs.
- Store and index. Write to a search engine, a label-indexed store, a columnar database or a SaaS backend.
- Query, visualise, alert. Search, build dashboards and trigger alerts on patterns.
Our explainer on why teams should adopt centralized logging early covers the reasons; this post covers the parts.
| # | Tool | Layer | Licence or model |
|---|---|---|---|
| 1 | Fluent Bit | Collector / shipper | Open source (CNCF graduated, under Fluentd) |
| 2 | Fluentd | Collector / aggregator | Open source (CNCF graduated) |
| 3 | Vector | Collector / processor | Open source (maintained by Datadog) |
| 4 | OpenTelemetry Collector | Collector / processor | Open source (CNCF) |
| 5 | Filebeat / Elastic Agent | Collector / shipper | Elastic |
| 6 | rsyslog | Collector / forwarder | Open source |
| 7 | Logstash | Processor | Elastic |
| 8 | Apache Kafka | Buffer / transport | Open source (Apache) |
| 9 | Elasticsearch | Storage and search | Elastic (includes an AGPL option) |
| 10 | OpenSearch | Storage and search | Open source (Apache 2.0) |
| 11 | Grafana Loki | Label-indexed storage | Open source (AGPLv3) |
| 12 | ClickHouse | Columnar storage | Open source (Apache 2.0) |
| 13 | Graylog | Storage + UI platform | Open and commercial editions |
| 14 | Kibana | Query and visualisation | Elastic |
| 15 | Grafana | Query and visualisation | Open source (AGPLv3) |
| 16 | Splunk | SaaS / self-managed platform | Commercial |
| 17 | Datadog Log Management | SaaS platform | Commercial |
| 18 | Sumo Logic | SaaS platform | Commercial |
| 19 | Amazon CloudWatch Logs | Cloud-native service | AWS, pay per use |
| 20 | Google Cloud Logging / Azure Monitor Logs | Cloud-native services | GCP / Azure, pay per use |
Item 20 groups the Google Cloud and Azure services because they play the same role as CloudWatch Logs on their own clouds.
Layer 1: collectors and shippers
1. Fluent Bit
Fluent Bit is a lightweight log processor and forwarder written in C, part of the Fluentd project under the CNCF. It is the usual choice as a Kubernetes DaemonSet: its tail input reads container logs and the kubernetes filter adds pod, namespace and label metadata. It can output to Elasticsearch, OpenSearch, Loki, Kafka, Splunk HEC, CloudWatch and many others.
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
[FILTER]
Name kubernetes
Match kube.*
[OUTPUT]
Name loki
Match kube.*
Host loki.logging.svc
Port 3100
Labels job=fluent-bit
2. Fluentd
Fluentd is the older, Ruby-based member of the same project, with a large plugin ecosystem. Teams often run Fluent Bit on every node and forward to a smaller pool of Fluentd aggregators that do heavy routing, buffering and fan-out to several destinations.
3. Vector
Vector is a high-performance observability data pipeline written in Rust. Datadog acquired the company behind it, Timber, in 2021, and Vector remains open source. It runs as an agent or aggregator, and its Vector Remap Language (VRL) handles parsing and field changes in one place.
sources:
app:
type: file
include: ["/var/log/app/*.log"]
transforms:
parse:
type: remap
inputs: ["app"]
source: |
. = parse_json!(.message)
sinks:
os:
type: elasticsearch
inputs: ["parse"]
endpoints: ["https://opensearch.example.internal:9200"]
4. OpenTelemetry Collector
The OpenTelemetry Collector receives, processes and exports telemetry. With the filelog receiver (in the contrib distribution) it reads log files, and it can export to OTLP backends, Loki, Elasticsearch, Splunk and SaaS vendors. Using one collector for logs, metrics and traces means fewer agents per node and consistent resource attributes across signals.
5. Filebeat and Elastic Agent
Filebeat is Elastic's lightweight shipper for log files, with modules for common formats such as Nginx and system logs. Elastic Agent is Elastic's newer unified agent, managed centrally through Fleet, and is Elastic's recommended path for new deployments. Both ship directly to Elasticsearch or through Logstash.
6. rsyslog
rsyslog is the default syslog daemon on many Linux distributions, including RHEL. It can forward logs over TCP or RELP with disk-assisted queues, which makes it a dependable shipper for servers and network devices without installing anything new. See our RHEL 10 log management guide for journald and rsyslog settings.
Layer 2: processing and buffering
7. Logstash
Logstash is the processing stage of the classic ELK stack. Its pipelines have inputs, filters and outputs; the grok filter parses unstructured text with named patterns, and persistent queues protect against data loss on restart. It uses more memory than node agents, so run it centrally.
input { beats { port => 5044 } }
filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output { elasticsearch { hosts => ["https://es01:9200"] } }
8. Apache Kafka
Kafka is not a logging tool by itself, but in large pipelines it sits between shippers and storage as a durable buffer. If Elasticsearch is slow or down, logs wait in Kafka topics instead of filling agent buffers. It also lets several consumers (search, security analytics, archive to object storage) read the same stream.
Layer 3: storage and query engines
9. Elasticsearch
Elasticsearch is a distributed search engine built on Apache Lucene that indexes the full content of each log event. It is strong for free-text search and aggregations. Index lifecycle management (ILM) moves data through hot, warm, cold and delete phases to control cost. In 2024 Elastic added AGPLv3 as a licence option for the source code alongside its existing licences.
10. OpenSearch
OpenSearch is the Apache 2.0 fork of Elasticsearch and Kibana created in 2021, now governed by the OpenSearch Software Foundation under the Linux Foundation. It offers similar search and dashboards (OpenSearch Dashboards) and index state management for retention. Amazon OpenSearch Service is the managed AWS offering.
11. Grafana Loki
Loki stores log lines compressed in object storage and indexes only a small set of labels, such as namespace, app and level. That keeps storage and index costs low. Queries in LogQL first select streams by label and then filter text:
{namespace="payments", app="api"} |= "timeout" | json | duration > 2s
The trade-off is that high-cardinality labels (user IDs, request IDs) hurt performance, and searching all logs for a random string is slower than in a full-text index.
12. ClickHouse
ClickHouse is a column-oriented SQL database. Some teams store logs in ClickHouse tables because columnar compression and SQL aggregation work well for very large, structured log volumes. You design the schema yourself, so it suits teams with data engineering skills.
13. Graylog
Graylog is a log management platform that bundles ingestion (GELF, syslog, Beats inputs), processing rules, search, dashboards and alerts. It uses OpenSearch (or its own Graylog Data Node) for storage and MongoDB for configuration. It suits teams that want one product instead of assembling ELK pieces.
Layer 4: analysis and visualisation
14. Kibana
Kibana is the front end for Elasticsearch: Discover for searching, Lens and dashboards for charts, and alerting rules. Most ELK interview questions focus on Kibana index patterns (now called data views) and KQL queries.
15. Grafana
Grafana queries many data sources, including Loki, Elasticsearch, OpenSearch, ClickHouse and CloudWatch. Its main advantage for logs is correlation: a panel showing a latency spike from Prometheus can link to the Loki logs for the same time range and labels. Our Prometheus and Grafana guide covers the metrics side.
SaaS and cloud-native platforms
16. Splunk
Splunk indexes machine data and is queried with SPL (Search Processing Language). Universal forwarders send data to indexers, and the HTTP Event Collector (HEC) accepts events over HTTPS. Splunk is available as Splunk Cloud Platform or Splunk Enterprise. Cisco completed its acquisition of Splunk in March 2024.
17. Datadog Log Management
Datadog collects logs through its Agent or integrations, parses them with pipelines and processors, and links them to traces and metrics through unified service tags (env, service, version). Datadog separates ingesting from indexing: you can ingest everything, index only what you search often, and send the rest to archives in your own storage.
18. Sumo Logic
Sumo Logic is a cloud-native log analytics and SIEM platform. Collectors and the Sumo Logic Kubernetes collection (based on OpenTelemetry) send data in, and a query language with operators such as parse, where and count by analyses it.
19. Amazon CloudWatch Logs
CloudWatch Logs stores logs from AWS services, Lambda and the CloudWatch agent in log groups with configurable retention. CloudWatch Logs Insights runs queries with commands like fields, filter and stats. Subscription filters stream logs to Kinesis, Lambda or OpenSearch. Read our CloudWatch Logs Insights tutorial for query examples.
20. Google Cloud Logging and Azure Monitor Logs
Google Cloud Logging collects logs from GCP services and GKE, with log buckets for retention, the Logs Explorer for queries and log sinks to route data to BigQuery, Cloud Storage or Pub/Sub. Azure Monitor Logs stores data in Log Analytics workspaces and is queried with KQL (Kusto Query Language). Both are the low-effort option when your workloads live on that cloud.
Three reference architectures
| Scenario | Pipeline | Why |
|---|---|---|
| Kubernetes, cost-sensitive | Fluent Bit DaemonSet → Loki (object storage) → Grafana | Low storage cost, label queries match Kubernetes metadata |
| Search-heavy, security or audit | Elastic Agent or Fluent Bit → Kafka → Logstash → Elasticsearch/OpenSearch → Kibana/Dashboards | Full-text search, buffering for spikes, rich parsing |
| Small team, no ops capacity | Vendor agent or OpenTelemetry Collector → Datadog, Splunk Cloud or cloud-native logging | No storage cluster to run; pay per ingest |
For Kubernetes-specific choices, our 12 Kubernetes logging tools post goes deeper.
How to choose
- Volume. Estimate GB per day. Full-text indexes grow storage several times faster than label indexes.
- Query style. Mostly "show logs for service X around time T"? Loki fits. Frequent free-text hunts across everything? Elasticsearch or OpenSearch.
- Retention and compliance. Audit logs may need a year or more; debug logs rarely need more than a few weeks. Split them.
- Team skills. Running Elasticsearch clusters needs JVM, shard and index skills. SaaS shifts that to the vendor for a fee.
- Format discipline. Structured JSON logs make every tool cheaper and faster. Fix the application logging format before buying anything.
Common mistakes
- Sending every debug line to a paid backend instead of filtering at the collector.
- Putting request IDs or user IDs into Loki labels, which explodes stream count.
- No disk buffer on agents, so a backend outage loses logs.
- Logging secrets or personal data. Redact at the collector with filters or VRL.
- Clocks out of sync, which scrambles event order. Run NTP or chrony everywhere.
Learn it hands-on
Our ELK Stack training in Pune builds a full pipeline with Beats, Logstash, Elasticsearch and Kibana, in the Kharadi classroom or live online. If you are preparing for interviews, the 80+ ELK Stack interview questions post is a good next step.
FAQ
What is log aggregation in DevOps?
Log aggregation is collecting logs from all servers, containers and services into one central system where they are parsed, stored, searched and used for dashboards and alerts.
What is the difference between Fluent Bit and Fluentd?
Both belong to the same CNCF project. Fluent Bit is written in C and is lightweight, so it runs on every node. Fluentd is Ruby-based with a larger plugin set and is often used as a central aggregator.
Is Loki a replacement for Elasticsearch?
For many Kubernetes teams, yes, because Loki is cheaper to run. But Loki indexes only labels, so free-text search across all logs is slower than in Elasticsearch or OpenSearch.
Do I need Kafka in a logging pipeline?
Not for small setups. Kafka helps at high volume, when you need a durable buffer during storage outages or want several consumers to read the same log stream.
What is the ELK stack versus the EFK stack?
ELK is Elasticsearch, Logstash and Kibana. EFK replaces Logstash with Fluentd or Fluent Bit, which is common on Kubernetes because the agents are lighter.
Can the OpenTelemetry Collector handle logs?
Yes. The filelog receiver in the contrib distribution reads log files, and the collector can export logs to OTLP backends and many vendors, alongside metrics and traces.
How do I control logging costs?
Filter and sample at the collector, drop debug logs in production, set retention per log type, avoid high-cardinality labels in Loki and, on SaaS platforms, index only the logs you search often.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0