Introducing the FLOOD Logging Stack: Fluent and Open
A centralized logging solution is essential for the efficient implementation of best practices for logging in Kubernetes. We present the FLOOD stack as an open source solution to this requirement.
Logging in Kubernetes
The built-in logging in kubernetes only provides very basic functionality. Developers can only access the logs of one container at a time and there are only logs of the current pod and its direct predecessor available. For more sophisticated needs, an external logging stack is needed where developers can aggregate logs across multiple containers, have access to the full history of logs and can define individual dashboards for easy overview. A centralized solution allows for an easy handling of log rotation, structured logging, the purging of sensitive information, and the inclusion of contextual information.
What does FLOOD stand for?
FLOOD is an acronym where single or a number of subsequent letters point to its parts:
- Fluentbit
- Fluentd
- Logging Operator
- Opensearch Operator
- Opensearch
- Opensearch Dashboards
The following diagram visualizes the acronym.
Why not ELK, Graylog or PLG?
First of all, to the agent and aggregator components: We chose Fluent Bit and Fluentd over Filebeat and Logstash because they are mature cloud native products that are also CNCF graduated since 2019. The more recent competitors of Promtail and Loki follow another approach to processing logs. Instead of indexing full text contents of the logs, Loki indexes a set of labels with metadata for each log stream.
And finally, we have licensing reservations towards the Elastic and Graylog stacks (see table “Overview of Logging Stack Components and Licenses”). The Elastic and SSPL licenses are neither FSF nor OSI approved and protect the commercial interests of the companies in question. Also, Elastic changed their licensing several times over the last years for their distribution binaries and source code (see figure “Comparing Elastic licenses before 2021, 2021–2024, and from 2024 onwards”).
| Stack | Agent | Aggregator | Storage | Dashboard |
|---|---|---|---|---|
| ELK | Filebeat | Logstash | Elasticsearch | Kibana |
| Graylog | Fluent Bit | Graylog | Elastic-/Opensearch, MongoDB | Graylog |
| PLG | Promtail | Loki | Grafana | |
| FLOOD | Fluent Bit | Fluentd | OpenSearch | OpenSearch-Dashboards |
Deploying the FLOOD stack
The FLOOD stack does not have tightly integrated components all the way down but instead can be divided into two main parts:
- Fluent Bit agent and Fluentd aggregator
- OpenSearch for log data storage and OpenSearch Dashboards for visualization and querying
Fluentd and FluentBit can be provisioned by the Logging Operator, reducing the maintainance effort of this part of the stack by automating the deployment and configuration of a Kubernetes logging pipeline. This operator has only recently (2023-09-19) joined up with CNCF Sandbox projects and is expected to see wide adoption, distribution, and, eventually, CNCF graduation. One indicator for this assessment is Rancher’s decision to use it as the integrated logging solution since v2.5, released October 2020.
The Logging Operator is deployed using a helm chart well maintained by the operator developers. The operator is configured to deploy Fluent Bit as a DaemonSet to the cluster and assures that every node has a logging agent tailing incoming container logs. The operator also deploys Fluentd (or Syslog-ng) as a StatefulSet that is set up to aggregate the logs collected by the agents (see diagram “Logging Operator architecture”).
In the proposed setup, the ClusterFlow and ClusterOutput custom resources are used to configure Fluentd to process and forward the logs to OpenSearch. The configuration applied to our service entails
- from which containers logs are collected (ClusterFlow match),
- how the logs are formatted (ClusterFlow filter),
- where the logs are sent to (ClusterOutput opensearch).
Firstly, we implemented an opt-in for logging using a list of namespaces for which we want logs to be collected.
Secondly, a filter that proved to be necessary is dedot. This filter-plugin, which is bundled with the default Fluend docker image, is used to replace dots in the keys of JSON-formatted logs with underscores to prevent type-ambiguities in the OpenSearch index. Without de-dotting, the app=myapp label would be interpreted as a string value with the app key, while the label app.kubernetes.io/component=myapp would be interpreted as a nested object with the same key. This would not be allowed to be ingested into neither Elasticsearch nor OpenSearch indexes. However, the de-dotted version app_kubernetes_io/component=myapp of the latter example is correctly treated as a key with a string value.
Thirdly, the output configuration ensures an encrypted and authenticated forward to OpenSearch and comprises the buffer configuration. Beware that Fluentd defaults to using a file-type buffer whose performance and implementation heavily depends on the characteristics of the local file system. It is not recommended to be used with any kind of remote file systems, e.g. NFS, which is provided in our clusters. Therefore we explicitly configured to use an in-memory buffer instead.
The second part of the stack is provisioned using the OpenSearch Kubernetes Operator. The operator watches for a Custom Resource OpenSearchCluster that defines the deployment of an OpenSearch Cluster and, advisably, Opensearch-Dashboards.
What else?
This brief introduction to the FLOOD stack could not go into detail about security considerations, performance metrics, operational aspects, troubleshooting or even how the actual logs could be utilized. But stay tuned for a follow-up on these topics driven by real-world experience.
Also, consider these further readings that give an idea of how and why we decided as we did:
- Selvam Raju: Kubernetes logging best practices, 2023-07-03.
- Shruti Anekar: How To Get Started With Logging On Kubernetes?, 2022-12-07.
- Joakim Eriksson & Anawil Karavek: A Comparative Analysis Of Log Management Solutions: ELK Stack Versus PLG Stack, 2023-06-08.
- Sematext Group: OpenSearch vs Elasticsearch: Which One Is Better to Use?, before 2025-03-25, most probably early 2022.
