Tags

, , , , , , , , , , , ,

When we looked at Fluent Bit 5.1, the configuration extensions were particularly interesting from a management perspective. With 5.2, there is more to explore around how we process telemetry and how the pipeline behaves when the destination struggles to keep up.

For me, the value is in the operational questions these changes help address. Can we make data useful before paying to store it? Can we apply processing without damaging the information we need for investigation? And can our collectors give an overloaded backend a chance to recover?

Lua processing across logs, metrics and traces

The headline addition is Fluent Typed Scripting (FTS), a native processor that uses Lua to express policies across all three telemetry signals. It provides typed access to values and OpenTelemetry context, including resources, scopes and attributes.

The “typed” part allows us to ensure data values such as identifiers represented as large integers need to survive a transformation unchanged. Rounding or truncation while modifying a field is likely to break the correlation of data values. FTS ensures that exact integers, nulls and binary values aren’t accidentally disrupted.

Strong typing also offers security benefits, as it can make it harder to induce memory overruns or underwrites.

We have previously touched upon the power of Lua in the Fluent Bit pipeline with use cases such as:

  • Redact named sensitive fields before forwarding telemetry.
  • Bring inconsistent attributes into a common form.
  • Reduce unnecessary events and metric dimensions. Reducing payload size can improve performance and lower storage costs.

FTS still uses Lua, so it doesn’t require learning a statically typed language. Existing Lua filters remain supported, although moving them to FTS requires adapting their API. This gives us a reason to evaluate FTS for new requirements without rewriting every working filter.

Reducing metric dimensions needs care.

A useful example in the FTS reference is deliberate metric aggregation. Removing a label can make previously distinct series identical. FTS provides an aggregation helper for combining compatible series, rather than assuming that deleting an attribute is sufficient.

Consider request counts broken down by individual customer identifiers. If our operational dashboard only needs route and response status, retaining customer-level detail may create a great deal of storage and query work without helping that dashboard.

The opportunity is to decide which detail serves an operational purpose before forwarding it. We still need to preserve the distinctions required for investigation and avoid combining measurements with incompatible meaning.

There is a similar decision around processing failures. By default, a failed FTS policy rolls back its changes and drops the affected item. The alternative keep mode forwards the original. For redaction, that could mean forwarding the very information we intended to remove. Error handling therefore belongs in the policy design and its tests.

Making system logs easier to use with OpenTelemetry

The new journald_otel processor maps journal records into OpenTelemetry logs, including severity, host and process information.

This addresses a familiar integration problem. Collecting a message is only part of making it useful; its fields also need to mean something to the tools querying it. Native mapping can reduce the custom transformations needed to bring Linux system logs alongside application telemetry.

We saw in the Moving Open Source Observability backend closer to turn-key solutions the value of normalising attribute names in telemetry.

Giving a busy destination time to recover

Fluent Bit 5.2 adds destination-aware output throttling for HTTP, OpenTelemetry, Elasticsearch and OpenSearch. Supported responses and retry hints can trigger a cooldown for that destination, and waiting through the cooldown does not consume retry attempts.

For Elasticsearch and OpenSearch bulk delivery, retries also retain unresolved records without replaying items already acknowledged as successful.

If a backend is overloaded, repeatedly sending it more work can prolong the problem. Responding to its request to slow down makes recovery more cooperative and avoids unnecessary traffic.

We still need sufficient buffering for the interruption. A cooldown changes delivery behaviour; it does not create unlimited capacity to absorb incoming data.

Performance that could translate into useful headroom

The published 5.2 benchmarks show substantial gains in particular workloads. In the memory-buffered OTLP/HTTP JSON test that adds one log attribute, throughput reaches 1.50 million records per second, around 7.2 times the 5.1 result when using native content_modifier processing.

The separate FTS log-policy performance reports 18–45% less CPU than equivalent Lua policies on 5.1.0. Given the additional overhead, this is an impressive performance boost.

The experimental browser SDK is another interesting direction: it runs Fluent Bit through WebAssembly. It deserves exploration, with its separate build requirements and experimental status in mind.

The buffering results are interesting as well. One sparse-stream test reports a 92.2% reduction in allocated disk space. An estate with many quiet streams may benefit differently from one dominated by a few busy streams.

We do have to treat these performance figures with care – simply assuming we see an existing deployment become seven times faster is foolhardy, as the figures are context-specific. But it is reasonable to assume that the new features will not adversely impact performance.

Smaller changes that help day-to-day operation

The release notes also describe YAML generation from command-line options and configuration-relative path handling for supported assets.

Plugins and Processor

As with all releases, there are a number of plugin upgrades and additions for the core, such as the output throttling mentioned, but we also have a new processor: journald_otel. The processor makes it quick and easy to map journald records directly to the OpenTelemetry structures (the alternative is a more complex mapping exercise).

Resources