Tag Archives: YAML

Fluent Bit classic to YAML

16 Friday Jan 2026

Posted by in development, Fluentbit, Fluentd, General, python, Technology

≈ Leave a comment

Tags

, , , ,

Many of the most potent features being added to Fluent Bit are only available in the newer YAML format. At the same time Fluent Bit is celebrating its 10th anniversary, so we can be sure that are some established deployments that are being maintained with the classic format.

The transition between the two Fluent Bit formats can be a bit fiddly. We built a tool as described in my Fluent Bit config from classic to YAML post using Java over a year ago. While it addressed many of the needs, I wasn’t entirely happy with the solution, for several reasons:

  • Written in Java, which, as a language, is the one I’m most at home with, as an implementation, it does increase the workload for using and implementing – needing a container, etc., ideally. Not unreasonable, but as a small tool project, we’re not working with a nice enterprise build pipeline, so from a code tweak to test is just fiddlier. I’m sure any Java devs reading this will be shouting, ” Setting up Maven, Jenkins, JUnit isn’t difficult – and they’re right, but it’s a time-consuming distraction from working on the real problem.
  • While it works, the code didn’t feel as clean as it could or should.

Over the last couple of years, we’ve become more comfortable with Python, and messing around with LLMs and code generation has helped accelerate our development skills. Python offers a potentially better solution: many environments that use Fluentd or Fluent Bit are Linux-based and include Python by default, so you can use the tool without messing with containers if you don’t want to. For a tool intended to support development and maintenance of configurations rather than in production being able to use or modify/extend the code easily is attractive.

We’ve started afresh and built a new implementation that is more extensible and maintainable. The new Python implementation is under the same Apache 2 license, but in a different repo so that existing use and forks of the Java implementation aren’t disrupted.

We can wrap Python with all the production-class build processes, but it’s not necessary to get a tool up and running or to enable people to tinker as needed. So we have focused on that. Over time, we’ll play with Poetry or other packaging mechanisms to make it even easier.

The new implementation can be found at https://github.com/mp3monster/fluent-bit-classic-to-yaml-converter

Continue reading →

Fluent Bit 3.2: YAML Configuration Support Explained

23 Monday Dec 2024

Posted by in Fluentbit, General, Technology

≈ Leave a comment

Tags

, , , , , , , , ,

Among the exciting announcements for Fluent Bit 3.2 is the support for YAML configuration is now complete. Until now, there have been some outliers in the form of details, such as parser and streamer configurations, which hadn’t been made YAML compliant until now.

As a result, the definitions for parsers and streams had to remain separate files. That is no longer the case, and it is possible to incorporate parser definitions within the same configuration file. While separate configuration files for parsers make for easier re-use, it is more troublesome when incorporating the configuration into a Kubernetes deployment configuration, particularly when using a side-car deployment.

Parsers

With this advancement, we can define parsers like this:

Classic Fluent Bit

[PARSER]
    name myNginxOctet1
    format regex
    regex (?<octet1>\d{1,3})

YAML Configuration

parsers:
  - name: myNginxOctet1
    format: regex
    regex: '/(?<octet1>\d{1,3})/'

As the examples show, we swap [PARSER] for a parsers object. Then, each parser is an array of attributes starting with the parser name. The names follow a one-to-one mapping in most cases. This does break down when it comes to parsers where we can define a series of values, which in classic format would just be read in order.

Multiline Parsers

When using multiline parsers, we must provide different regular expressions for different lines. In this situation, we see each set of attributes become a list entry, as we can see here:

Classic Fluent Bit

[MULTILINE_PARSER]
  name multiline_Demo
  type regex
  key_content log
  flush_timeout 1000
  #
  # rule|<state name>|<regex>|<next state>
  rule "start_state" "^[{].*" "cont"
  rule "cont" "^[-].*" "cont"

YAML Configuration

multiline_parsers:
  - name: multiline_Demo
    type: regex
    rules:
    - state: start_state
      regex: '^[{].*'
      next_state: cont
    - state: cont
      regex: "^[-].*"
      next_state: cont

In addition to how the rules are nested, we have moved from several parameters within a single attribute(rule) to each rule having several discrete elements (regex, next_state). In addition to this, we have also changed the use of single and double quote marks.

If you want to keep the configurations for parsers and streams separate, we can continue to do so, referencing the file and name from the main configuration file. While converting the existing conf to a YAML format is the bulk of the work, in all likelihood, you’ll change the file extension to be .YAML will means you must also modify the referencing parsers_file reference in the server section of the main configuration file.

Streams

Streams follow very much the same path as parsers. However, we do have to be a lot more aware of the query syntax to remain within the YAML syntax rules.

Classic Fluent Bit

[STREAM_TASK]
  name selectTaskWithTag
  exec SELECT record_tag(), rand_value FROM STREAM:random.0;

[STREAM_TASK]
  name selectSumTask
  exec SELECT now(), sum(rand_value)   FROM STREAM:random.0;

[STREAM_TASK]
  name selectWhereTask
  exec SELECT unix_timestamp(), count(rand_value) FROM STREAM:random.0 where rand_value > 0;

YAML Configuration 

stream_processor:
  - name: selectTaskWithTag
    exec: "SELECT record_tag(), rand_value FROM STREAM:random.0;"
  - name: selectSumTask
    exec: "SELECT now(), sum(rand_value) FROM STREAM:random.0;"
  - name: selectWhereTask
    exec: "SELECT unix_timestamp(), count(rand_value) FROM STREAM:random.0 where rand_value > 0;"

Note, it is pretty common for Fluent Bit YAML to use the plural form for each of the main blocks, although stream definition is an exception to the case. Additionally, both stream_processor and stream_task are accepted (although stream_task is not recognized in the main configuration file)..

Incorporating Configuration directly into the core configuration file

To support directly incorporating these definitions into a single file, we can lift the YAML file contents and apply them as root elements (i.e., at the same level as the pipeline, and service, for example).

Fluent Bit book examples

Our Fluent Bit book (Manning, Amazon UK, Amazon US, and everywhere else) has several examples of using parsers and streams in its GitHub repo. We’ve added the YAML versions of the configurations illustrating parsers and stream processing to its repository in the Extras folder.

Two weeks of Fluent Bit

30 Tuesday Jul 2024

Posted by in Fluentbit, General

≈ Leave a comment

Tags

, , , , , ,

The last couple of weeks have been pretty exciting. Firstly, we have Fluent Bit 3.1 released, which brings further feature development to Fluent Bit, making it even more capable with Fluent Bit handling of Open Telemetry (OTel).

The full details of the release are available at https://fluentbit.io/announcements/v3.1.0/

Fluent Bit classic to YAML

We’ve been progressing the utility, testing and stabilizing it, and making several releases accordingly. The utility is packaged as a Docker image, and the regression test tool also runs as a Docker image.

Moving forward, we’ll start branching to develop significant changes to keep the trunk stable, including experimenting with the possibility of extending the tool to help port Fluentd to Fluent Bit YAML configurations. The tools won’t be able to do everything, but I hope they will help address the core structural challenges and flag differences needing manual intervention.

Book

The Fluent Bit book has moved into its last phase with the start of copy editing. We have also had a shift in the name to Logs and Telemetry using Fluent Bit, Kubernetes, streaming, and more, or just Logs and Telemetry using Fluent Bit. The book fundamentally hasn’t changed. There is still a lot of Kubernetes-related content, but it helps focus on what Fluent Bit is all about rather than being just another Kubernetes book.

Logs and Telemetry using Fluent Bit
Logs and Telemetry using Fluent Bit, Kubernetes, streaming and more

Fluent Bit config from classic to YAML

02 Tuesday Jul 2024

Posted by in Fluentbit, General, java, Technology

≈ 2 Comments

Tags

, , , , ,

Fluent Bit supports both a classic configuration file format and a YAML format. The support for YAML reflects industry direction. But if you’ve come from Fluentd to Fluent Bit or have been using Fluent Bit from the early days, you’re likely to be using the classic format. The differences can be seen here:

[SERVICE]
    flush 5
    log_level debug
[INPUT]
   name dummy
   dummy {"key" : "value"}
   tag blah
[OUTPUT]
   name stdout
   match *
#
# Classic Format
#
service:
    flush: 1
    log_level: info
pipeline:
    inputs:
        - name: dummy
          dummy: '{"key" : "value"}'
          tag: blah
    outputs:
        - name: stdout
          match: "*"
#
# YAML Format
#

Why migrate to YAML?

Beyond having a consistent file format, the driver is that some new features are not supported by the classic format. Currently, this is predominantly for Processors; it is fair to assume that any other new major features will likely follow suit.

Migrating from classic to YAML

The process for migrating from classic to YAML has two dimensions:

  • Change of formatting
    • YAML indentation and plugins as array elements
    • addressing any quirks such as wildcard (*) being quoted, etc
  • Addressing constraints such as:
    • Using include is more restrictive
    • Ordering of inputs and outputs is more restrictive – therefore match attributes need to be refined.

None of this is too difficult, but doing it by hand can be laborious and easy to make mistakes. So, we’ve just built a utility that can help with the process. At the moment, this solution is in an MVP state. But we hope to have beefed it up over the coming few weeks. What we plan to do and how to use the util are all covered in the GitHub readme.

The repository link (fluent-bit-classic-to-yaml-converter)

Update 4th July 24

A quick update to say that we now have a container configuration in the repository to make the tool very easy to use. All the details will be included in the readme, along with some additional features.

Update 7th July

We’ve progressed past the MVP state now. The detected include statements get incorporated into a proper include block but commented out.

We’ve added an option to convert the attributes to use Kubernetes idiomatic form, i.e., aValue rather than a_value.

The command line has a help option that outputs details such as the control flags.

Update 12th July

In the last couple of days, we pushed a little too quickly to GitHub and discovered we’d broken some cases. We’ve been testing the development a lot more rigorously now, and it helps that we have the regression container image working nicely. The Javadoc is also generating properly.

We have identified some edge cases that need to be sorted, but most scenarios have been correctly handled. Hopefully, we’ll have those edge scenarios fixed tomorrow, so we’ll tag a release version then.

Mastering Puppet Review

09 Tuesday Sep 2014

Posted by in Books, General, Technology

≈ Leave a comment

Tags

, , , , , , , , , , , , ,

Packt’s Mastering Puppet kicks off with substantial first chapter on how to setup Puppet in a manner that can then scale. The core of this is driven by an explanation of the constituent parts of a Puppet solution and where the workload is. In terms of execution this is as much about understanding the configuration of things like Apache, Passenger and Ningx as it is Puppet. As part of the explanation there are indicative numbers in terms of supportable scale which reflects the knowledge of the product.

Looking at configuration distribution for headless deployments with Git is a solid well considered piece and the writing suggests considers all the needs of a solid deployment of a production quality solution such as access control, whilst supporting collaborative working etc. it would be interesting to have seen how that would have stacked against capabilities such as Zookeeper.

As we move through the chapters the books continues with more advanced themes such as using Hiera as a object hierarchical framework for managing configuration and on into leveraging Puppet forge and various Git repositories (and the challenges when linking to git repositories of the latest code vs a release). With the repositories we can draw in additional tooling and how to incorporate these capabilities into a deployment. This includes looking at several modules that practical experience from the author would recommend.

By chapter 6 we’re into writing our own custom modules and facts and deploying them. So you can do things such as create modules to manage your custom solutions.

The next natural step is to look at the reporting aspects of Puppet, orchestration through marionette collective (mCollective). Obviously to report you need to gather the activity information, so the book touches on the out of the box (OOTB) approach and moves onto the idea of using IRC; presentation via Foreman and Puppet Dashboard. Finally then with a reporting view, the next step is to dynamically query the nodes in Puppet environment which uses mcollective to communicate back & forth with the nodes.

So now we have a dynamically configurable set of Nodes, which can report and have dynamic querying against the nodes.  Final chapters cover the use of things like PuppetDB, roles & profiles and developing and debugging your puppet environment.

Reading the book, I get the feeling that a fair grasp of Linux system administration would help (i.e. a bit more than the average developer). There are a few useful touches that I think could have been included, such as external references such as man pages for RPM or site for the Pulp tool mentioned. But, as criticisms go, this as much me being too lazy to Google. The only other refinement would be inclusion of some diagrams to support the words. As they say a picture can tell a 1000 words, even if this was to just show the hierarchy or directory structures involved.

Compared to the recently reviewed Puppet Reporting book, this book isn’t for someone starting out with Puppet (but the Packt site says as much). You atleast need to have got some basic understanding or practical exposure to Puppet,  and exposure to a development environment is an added bonus.  So if you’re setting out with Puppet you might consider starting with the Puppet 3 Beginner’s Guide (Amazon) or Instant Puppet 3 Starter (Amazon).  Having got those under your belt, try this book to to really develop the use of Puppet configuration and deployment.  When it comes to reporting I’d look at this book along with reporting book (reviewed here).  This book feels like more options are on offer, but Puppet Reporting is a lot richer (but you’d expect that given the different book emphasis).

In summary – good solid book, full of practical experience and ideas.  But don’t try to use this as a jumpstart to Puppet.

Below are a few links I thought might be helpful as they aren’t in the book:

  • YAML – human readable serialization format
  • Pulp – software repository management app
  • Ruby – Open Source OO programming language
  • Foreman – tool capable of extending puppet to deliver PXE capabilities along with capabilities such as reporting
  • Splunk – BigData style analytics on log files etc
  • Elasticsearch / Logstash / Kibana (ELK) – set of tools to provide analytics against log files
  • ActiveMQ – Apache implementation of a JMS compliant messaging solution used my mcollective

Mastering Puppet at Amazon.