Petabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From Intuit

Intuit leveraged CNCF projects, Fluentd and Fluent Bit, to configure reliable buffering for network outages, flexible configurations for backpressure management, and tips to avoid common mistakes that will save you hours of troubleshooting.

• Fluentd and Fluent Bit are CNCF projects that solve the problem of taking data from point A and sending it to point B, regardless of the data source or destination.
• Challenges for logging at scale include high costs, reliability and buffering, networking issues, event throughput, security, and operationality.
• Fluentd and Fluent Bit solve these challenges through filtering, parsing, and compression, file system and memory buffers, configurable retry mechanisms, back pressure handling, multi-worker configuration, anonymization filters, TLS encryption, and forwarder-aggregator architecture.
• Common architecture patterns for logging at scale include forwarder-only and forwarder-aggregator.
• Intuit's use case demonstrates how they achieved petabyte scale logging with Fluentd and Fluent Bit.

Intuit faced several problems when scaling up logging for their Kubernetes clusters, including decreased throughput, higher resource consumption, and unexpected backpressure. By leveraging Fluentd and Fluent Bit, they were able to configure reliable buffering for network outages, flexible configurations for backpressure management, and avoid common mistakes that saved them hours of troubleshooting.

In financial and software companies like Intuit, it’s mandatory to provide a reliable observability layer so internal end-users like developers from different line-of-businesses can access their application data. In the last few years, we faced several problems when scaling up logging. For all the new applications we deployed in our Kubernetes clusters, the data rate increased, creating decreased throughput, higher resource consumption, and unexpected backpressure. In this session, you will learn how we leveraged CNCF projects, Fluentd and Fluent Bit, to configure reliable buffering for network outages, flexible configurations for backpressure management, and tips to avoid common mistakes that will save you hours of troubleshooting. A secondary title for this talk would be: “How we achieved Petabyte scale logging with Fluentd and Fluent Bit”.