Enhance your Kubernetes logging with the Logging operator 4.0 release

We are happy to announce a new major release of the Logging operator.

It’s been a while since we wrote about the Logging operator. In the background, we were working on awesome new features to make Kubernetes logging as convenient and effortless as possible. Among its new features, one stands out as it brings fundamentally new functionality to the Logging operator: the option to use syslog-ng as the central processing element of the Logging operator.

Why another tool for Kubernetes logging?

Kubernetes logging has plenty of tools already, so you might wonder: why another one? The initial plan for the Logging operator was always to be a collector-independent solution for Kubernetes logging. While many people lock themselves in their Kubernetes dashboard, sometimes you need more robust solutions. 

We chose Fluent Bit and Fluentd as our first approach because at that time these were the most widely used tools and the team had a lot of experience with them. We are happy that the Logging operator is thriving and reaching more and more users. Naturally, this brings new challenges and we need to re-evaluate the way how the operator works.

We got an interesting case from a Logging operator user. They have relatively big clusters with an application set-up that requires around 100-150 flows to handle the different kinds of applications. Our initial routing implementation for Fluentd has its limitations:

• It's a single-threaded Ruby plugin.
• Due to the nature of the function, you can’t do batch processing: you need to make decisions on a per-event basis.
• Consequently, computation-intensive regex match filters of the logging flows are evaluated several times for every incoming message.

We have worked intensively with this customer to optimize their configuration and we managed to triple the performance of the log processing, but it became clear that big clusters produce a lot of log messages. Our optimizations around Fluentd won't be able to keep up with large-scale production traffic. We had to think outside the box of Logging operator, and add a new tool that can route and process log traffic at scale.

Why syslog-ng?

There is an overwhelming number of different tools out there to build a Kubernetes log, so we gathered a list of requirements that we expected from the new tool:

• Small footprint: Resources on the edge can be very limited. We don’t want to impact the endpoint’s performance significantly.
• Extensive configuration options: Logging operator solves complex problems with a simplified configuration layer. That means that a simple rule can be compiled into several small, related configuration units. To achieve that, the underlying language must provide a great degree of freedom. 
• Hot reload: It may seem insignificant to be able to reload the configuration at runtime, but in larger clusters with frequent configuration changes hot reloading saves you from a huge overhead. You need to persist/restore data from the disk.
• Plugins: It is almost impossible to embed everything into one tool. Using plugins allows us to easily extend the features without having to recompile or rebuild the whole toolset. And we can even exclude unnecessary components.
• Responsive and helpful developers: A vibrant community of developers is one of the key values we search for in a 3rd-party component. Quick fixes and up-to-date security patches are essential to keep the stack healthy.
• Open-source: It goes without saying that open-source software has many benefits when it comes to embedding in your Kubernetes dashboard or software stack.

Syslog-ng is a mature open-source log management tool that has been used by large enterprises for over twenty years, with a broad feature set and excellent performance. It is actively maintained and developed, has a large, worldwide user base, and is known to work on practically everything, from e-readers to cars and airplanes.

We worked closely with the syslog-ng developers and decided to roll together. Extending syslog-ng capabilities to run smoothly on top of Kubernetes is a much smaller scope than evaluating brand-new tools and thoroughly testing the basic capabilities. We are not saying that other tools like Vector, Tremor, or OpenTelemetry Collector won’t do the job but we have trust in syslog-ng.

However, syslog-ng didn't support cloud-native Kubernetes use cases out of the box and showed some weird behavior if its configuration was changed frequently. Also, the format of its configuration needs some getting used to. To overcome these problems, we have:

• Added new custom resources to the Logging operator that are specific to syslog-ng. That way you configure these CRs using the familiar Logging operator objects in YAML format, and Logging operator generates the syslog-ng configuration from these CRs automatically.
• Worked with the syslog-ng developers and the community to add Kubernetes support to syslog-ng.
• Filed and fixed bugs and issues that came up when running syslog-ng in Kubernetes, and improved its JSON support.

Results

We have worked with our customers and tested the syslog-ng support in Logging operator in their environment to make sure it meets their requirements. Right now, they are already using it in production, and the solution is stable and has solved their performance issues.

Fluentd and syslog-ng, support, and features

Fluentd remains fully supported in Logging operator, and Logging operator 4.0 is fully compatible with the older 3.x releases.

Logging operator can use Fluentd and syslog-ng as log forwarders to receive, filter, and transform the incoming logs, and to transport them to one or more destination outputs. Fluentd and syslog-ng have their separate capabilities and they support their own destination outputs (of course, there are overlaps). Currently, only a limited number of syslog-ng outputs are available in Logging operator, but we are actively working on extending the list. However, if you're sending logs to Sumo Logic, or a generic HTTP output, you might want to try syslog-ng.

Advanced routing example

Routing with syslog-ng is much more flexible than with Fluentd. You can match strings or regular expressions to any part of the message, for example, to route an exact type of message from a container. Like match only the 4XX codes from an Nginx access log.

match:
    and:
    - regexp:
        value: json.kubernetes.labels.app.kubernetes.io/instance
        pattern: one-eye-log-generator
        type: string
    - regexp:
        value: json.code
        Pattern: 4*
        type: glob

We are planning to make more syslog-ng features available from Logging operator in the future, and also to add new features to Logging operator based on syslog-ng. For details, see the Logging operator documentation and out quick start guides.

Other changes in Logging operator 4.0

A list with all the changes since 3.0.0 would be overwhelming: we released 3.0.0 almost three years ago, even the last minor release (3.17.0) is over a year old now. So to keep things short and manageable, let's see the notable changes and bugfixes since the last patch release (3.17.10):

• Improved Fluentd concat support to handle split containerd/cri logs in the Flow and ClusterFlow manifests
• You can now enable the "include_thread_label" option in the Loki output
• You can deploy an SCC in OpenShift to run Fluent Bit in privileged mode
• You can now set a TTL for Fluent Bit metadata to avoid unneeded restarts
• Logging operator now uses the latest prometheus-operator (version 0.62.0)
• Flush errors are properly handled in cached RESTMapper constructor

Logging operator moves to a new place

If you are watching the Logging operator project, you might have noticed some major changes. The reason for this is that the team behind the Logging operator wanted to spend more time on the project. As a result, Logging operator got its own organization, and the former team (now Axoflow) and Cisco will work together on the success of the project.

What has changed:

• The project repository is now available under a separate GitHub organization at https://github.com/kube-logging/logging-operator.
• The documentation has also moved to its own place at https://kube-logging.github.io/docs/.
• Our Fluentd image repository is available at https://github.com/kube-logging/fluentd-images/.
• Helm chart repository is now at https://kube-logging.github.io/helm-charts.

Kubernetes logging made simpler

Building a Kubernetes log shouldn’t be complicated. Logging operator 4.0 is here to ensure that you can get more work done without adding to the complexity of your logs. But if you want to learn more about how this is changing things, you can check out where Logging operator has appeared before here at Outshift by Cisco.

Contributor acknowledgments

We'd like to thank the syslog-ng developers for their time and help in making syslog-ng ready for the Kubernetes world. (For example, making ARM builds of syslog-ng available.)

Of course, apart from the bigger features, there have been many smaller additions and fine-tunings in the various Logging operator plugins and outputs, and we'd like to express our sincere and utmost gratitude to all our contributors for actively supporting the project.