Many of the most potent features being added to Fluent Bit are only available in the newer YAML format. At the same time Fluent Bit is celebrating its 10th anniversary, so we can be sure that are some established deployments that are being maintained with the classic format.

The transition between the two Fluent Bit formats can be a bit fiddly. We built a tool as described in my Fluent Bit config from classic to YAML post using Java over a year ago. While it addressed many of the needs, I wasn’t entirely happy with the solution, for several reasons:

• Written in Java, which, as a language, is the one I’m most at home with, as an implementation, it does increase the workload for using and implementing – needing a container, etc., ideally. Not unreasonable, but as a small tool project, we’re not working with a nice enterprise build pipeline, so from a code tweak to test is just fiddlier. I’m sure any Java devs reading this will be shouting, ” Setting up Maven, Jenkins, JUnit isn’t difficult – and they’re right, but it’s a time-consuming distraction from working on the real problem.
• While it works, the code didn’t feel as clean as it could or should.

Over the last couple of years, we’ve become more comfortable with Python, and messing around with LLMs and code generation has helped accelerate our development skills. Python offers a potentially better solution: many environments that use Fluentd or Fluent Bit are Linux-based and include Python by default, so you can use the tool without messing with containers if you don’t want to. For a tool intended to support development and maintenance of configurations rather than in production being able to use or modify/extend the code easily is attractive.

We’ve started afresh and built a new implementation that is more extensible and maintainable. The new Python implementation is under the same Apache 2 license, but in a different repo so that existing use and forks of the Java implementation aren’t disrupted.

We can wrap Python with all the production-class build processes, but it’s not necessary to get a tool up and running or to enable people to tinker as needed. So we have focused on that. Over time, we’ll play with Poetry or other packaging mechanisms to make it even easier.

The new implementation can be found at https://github.com/mp3monster/fluent-bit-classic-to-yaml-converter

Design & Implementation Ideals

We have tried to ensure the Python code addresses the following underlying principles:

• Minimizing dependencies as much as possible (ideally, we only want to work with core Python packages), meaning you don’t have to worry about virtual environments, and we don’t need to worry about patching dependencies.
• Avoid binding to the latest version of Python so it can be used in as many environments as possible.
• Make the code easy to follow so you can extend/customize it.
• Drive the mapping through configuration so that the code is unimpacted by minor releases.

Testing

To test the utility, we’ve taken the configurations we produced for the book in classic format, processed them with the utility, visually inspected the content, and used the Fluent Bit dry-run feature to run some (not all), as environment setup can be more time-consuming.

We have attempted to implement a test script that runs the dry-run process across all test configurations. But found a gremlin in the way Fluent Bit works, which meant we couldn’t capture the dry-run output for successful results.

Known current limitations

Currently, there are a couple of constraints to the generated output, specifically:

• The tool can handle inclusions, but it can’t recreate the inclusion files in the new format. This is because understanding what needs to change requires the entire configuration. Which means actually merging all the configurations together first. Of course, once the conversion is complete, the new configuration can be manually decomposed again. We’ll look into
• Because comments can be placed almost anywhere in the configuration, this can complicate processing, so comments aren’t yet carried through.

Future proofing

While we can’t guarantee the solution is future-proof, we don’t know what the future will bring in terms of changes. That said, changes such as adding new plugins and attributes are common advancements. We have made it as easy as possible to address and maintain by using a JSON configuration file to drive the mapping.

We’ll try to maintain the config, but it has to be done around the day job. But we’re happy if people want to raise pull requests to add missing details, etc.

CLI Guide

The GitHub repository includes a README with all the details to use the tool. The Repository also includes instructions for extending the mapping configuration to handle any required transitions for customer and private plugins.

Current Feature Enhancement list:

• Handle comments so they’re carried through to the YAML configuration.
• Automate testing so we can easily rerun any test and scale test coverage.
• Look at reversing the merge process, so that it includes YAML, so that it can work.
• Improve the deployment/installation using Wheel/Poetry.
• Look into how we might be able to extend the logic to also perform validation – so we exploit the configuration file and provide additional validation beyond what is implemented by a dry-run such as:
  • Checking for attributes that aren’t known to a plugin as these can trigger runtime errors.
  • Optionally, check connectivity to the network location or to the ports identified in the configuration.
• Use the same mechanism, but a different configuration to perform Fluentd to Fluent Bit migration.