Fluent Bit: Lightweight Log Collection in Production
Actualizado: 2026-05-03
Fluent Bit[1] is the lightweight collector from the Fluentd ecosystem. A graduated CNCF project, written in C, with a ~1.5 MB binary and a memory footprint that rarely exceeds 30 MB under steady load. For anyone weighing Promtail, Vector, or the Datadog agent, it belongs in the conversation as a serious option, especially when nodes are resource-constrained or process density is high. With the 3.x line stable, it has become a reasonable default for Kubernetes fleets.
Key takeaways
- ~1.5 MB C binary without a runtime: overhead multiplies by nodes in a DaemonSet deployment.
- Four-stage architecture: inputs → parsers → filters → outputs, with fan-out to multiple simultaneous destinations.
- The Kubernetes filter is nearly mandatory: it adds namespace, pod, container, and labels to every event.
- Loki, Elasticsearch, S3, Kafka, and generic HTTP are the most common outputs; one instance can serve several at once.
- Label cardinality and poorly anchored regex parsers are the two common sources of production incidents.
Why size matters
A tiny binary is not just a curiosity. When the collector ships as a DaemonSet, the cost multiplies by nodes. I have seen clusters where Filebeat ate several hundred megabytes per node — real money vanishing into a sidecar whose only job was moving text. Fluent Bit, written in C without a runtime underneath, lives in a different order of magnitude.
Density also matters at the edge. An industrial router, a Raspberry Pi, or a small VM has very little headroom, and the gap between a 10 MB and a 100 MB agent decides whether observability fits at all. Performance tracks size: around 100,000 events per second per node without breaking a sweat.
How it is organised internally
The conceptual pipeline has four stages: inputs (file tailing, journald, Kubernetes module, TCP/syslog), parsers (structuring data into typed fields), filters (Kubernetes enrichment, grep for noise reduction, modify for field manipulation, Lua for custom logic), and outputs (Loki, Elasticsearch, Kafka, S3, HTTP endpoints). A single event can go to several outputs at once, enabling fan-out without an intermediate broker.
Installing in practice
In Kubernetes, the official Helm chart installs a DaemonSet with mounts and permissions sorted:
helm install fluent-bit fluent/fluent-bit --namespace logging --create-namespaceConfiguration follows a sectioned INI-style format with SERVICE, INPUT, PARSER, FILTER, and OUTPUT blocks. The learning curve is moderate: day one you struggle to find where each thing lives; within a week the format becomes predictable.
Common outputs and when to pick them
Loki is the natural pairing when Grafana is the visualisation stack. Choose low-cardinality labels: every distinct value creates an index in Loki and the memory bill grows quickly.
Elasticsearch or OpenSearch: the native output handles retries, Logstash formatting, and bulk inserts. If you already run Elastic, Filebeat is better integrated with Kibana, but Fluent Bit wins on footprint and multi-destination flexibility.
S3 with gzip compression: perfect for cheap archival. Send only recent data to Loki and push everything to a bucket that later transitions to Glacier.
Filters and parsers worth mastering
The Kubernetes filter is nearly mandatory: without it, events lose half their value. It needs RBAC permissions to read pod metadata.
Regex parsers: anchor them firmly at start and end of line. A badly written expression slips by unnoticed until CPU on a node starts climbing for no obvious reason. When the application can emit JSON, it’s worth changing the application rather than suffering fragile parsing. The Lua filter is the escape hatch for cases no standard plugin handles.
Conclusion
Fluent Bit is the reasonable default for log collection in Kubernetes and at the edge. Treat it as a lightweight event bus, not a point-to-point pipe. A single DaemonSet collects everything, splits by tag, and fans out to Loki for short-term querying, S3 for archive, and Kafka when security teams want streaming. Hold the line on label cardinality and custom parsers, and Fluent Bit disappears from postmortem meetings for months — the highest compliment you can pay a piece of infrastructure.
