Tags

, , , , , , , , , , , , , , ,

While the typical norm is for applications to write their logs to file or to stdout (console), this isn’t the most efficient way to handle logs (particularly given I/O performance for the storage devices). Many logging frameworks have addressed this by providing more direct outputs to commonly used services such as ElasticSearch and OpenSearch. This is fine, but the only downside is that there is no means for an intermediary layer to preprocess, filter, and route (potentially to multiple services). These constraints can be overcome by using an intermediary service such as Fluent Bit or Fluentd.

Many logging frameworks can work with Fluentd by supporting the HTTP or Forward protocols Fluentd supports out of the box. But as both Fluent Bit and Fluentd are interchangeable with these protocols and logging framework that supports Fluentd, by implication also supports Fluent Bit, not to mention Fluent Bit supports OpenTelemetry.

The following table identifies a range of frameworks that can support communicating directly with Fluent Bit. It is not exhaustive but does provide broad coverage. We’ll update the table as we discover new frameworks that can communicate directly.

LanguageFramework / LibraryProtocol(s)Commentary
JavaLog4J2HTTP AppenderSend JSON payloads over HTTP (use HTTP input plugin)
Javafluent-logger-javaForward
Pythoncore languageHTTP HandlerProvides the means to send logs over HTTP – means Fluent Bit input handler can manage
Pythonfluent-logger-pythonForward
PythonFluent LoggerForward
Node.jsfluent-logger-nodeForward
Node.jsWinstonHTTP
Go (Golang)fluent-logger-golangFoward
.Net (C# VB.Net etc)NLog (NLog.Targets.Fluentd)
Rubyfluent-logger-rubyForward
PHPfluent-logger-phpForward
Perlfluent-logger-perlForward
Scalafluent-logger-scalaForward
Erlangfluent-logger-erlangForward
OCAMLfluent-logger-ocamlForward