Earlier in the year, I made a utility available that supported the migration from Fluent Bit classic configuration format to YAML. I also mentioned I would explore the migration of Fluentd to Fluent Bit. I say explore because while both tools have a common conceptual foundation, there are many differences in the structure of the configuration.
We discussed the bigger ones in the Logs and Telemetry book. But as we’ve been experimenting with creating a Fluentd migration tool, it is worth exploring the fine details and discussing how we’ve approached it as part of a utility to help the transformation.
Routing
Many of the challenges come from the key difference in terms of routing and consumption of events from the buffer. Fluentd assumes that an event is consumed by a single output; if you want to direct the output to more than one output, you need to copy the event. Fluent Bit looks at things very differently, with every output plugin having the potential to output every event – the determination of output is controlled by the match attribute. These two approaches put a different emphasis on the ordering of declarations. Fluent Bit focuses on routing and the use of tags and match declarations to control the rounding of output.
<match *>
@type copy
<store>
@type file
path ./Chapter5/label-pipeline-file-output
<buffer>
delayed_commit_timeout 10
flush_at_shutdown true
chunk_limit_records 50
flush_interval 15
flush_mode interval
</buffer>
<format>
@type out_file
delimiter comma
output_tag true
</format>
</store>
<store>
@type relabel
@label common
</store>
</match>Hierarchical
We can also see that Fluentd’s directives are more hierarchical (e.g., buffer, and format are within the store) than the structures used by Fluentd Bit, so we need to be able to ‘flatten’ the hierarchy. As a result, it makes sense that where the copy occurs, we’ll define both outputs in the copy declaration as having their own output plugins.
Buffering
There is a notable difference between the outputs’ buffer configurations: in Fluent Bit, the output can only control how much storage in the filesystem can be used. As you can see in the preceding example, we can set the flushing frequency, control the number of chunks involved (regardless of storage type).
Pipelines
Fluentd allows us to implicitly define multiple pipelines of sources and destinations, as ordering of declarations and event consumption is key. ~In addition to this, we can group plugin behavior with the use of the Fluentd label attribute. But the YAML representation of a Fluent Bit doesn’t support this idea.
<source>
@type dummy
tag dummy
auto_increment_key counter
dummy {"hello":"me"}
rate 1
</source>
<filter dummy>
@type stdout
</filter>
<match dummy>
@id redisTarget
@type redislist
port 6379
</match>
<source>
@id redisSource
@type redislist
tag redisSource
run_interval 1
</source>
<match *>
@type stdout
</match>Secondary outputs
Fluentd also supports the idea of a secondary output as the following fragment illustrates. If the primary output failed, you could write the event to an alternate location. Fluent Bit doesn’t have an equivalent mechanism. To create a mapping tool, we’ve taken the view we should create a separate output.
<match *>
@type roundrobin
<store>
@type forward
buffer_type memory
flush_interval 1s
weight 50
<server>
host 127.0.0.1
port 28080
</server>
</store>
<store>
@type forward
buffer_type memory
flush_interval 1s
weight 50
<server>
host 127.0.0.1
port 38080
</server>
</store>
<secondary>
@type stdout
</secondary>
</match>The reworked structure requires consideration for the matching configuration, which isn’t so easily automated and can require manual intervention. To help with this, we’ve included an option to add comments to link the new output to the original configuration.
Configuration differences
While the plugins have a degree of consistency, a closer look shows that there are also attributes and, as a result, features of plugins that don’t translate. To address this, we have commented out the attribute so that the translated configuration can be seen in the new configuration to allow manual modification.
Conclusion
While the tool we’re slowly piecing together will do a lot of the work in converting Fluentd to Fluent Bit, there aren’t exact correlations for all attributes and plugins. So the utility will only be able to perform the simplest of mappings without developer involvement. But we can at least help show where the input is needed.
Phil (aka MP3Monster)'s Blog 