• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Journald

Fluent Bit 5.2 announced – what’s new and how it can help

18 Wednesday Nov 2026

Posted by mp3monster in Fluent Observability, Fluentbit, General, Technology

≈ Leave a comment

Tags

benchmarks, Elasticsearch, Fluent Bit, HTTP, Journald, Lua, OpenSearch, OpenTelemetry, OTel, OTLP, performance, release, WebAssembly

When we looked at Fluent Bit 5.1, the configuration extensions were particularly interesting from a management perspective. With 5.2, there is more to explore around how we process telemetry and how the pipeline behaves when the destination struggles to keep up.

For me, the value is in the operational questions these changes help address. Can we make data useful before paying to store it? Can we apply processing without damaging the information we need for investigation? And can our collectors give an overloaded backend a chance to recover?

Lua processing across logs, metrics and traces

The headline addition is Fluent Typed Scripting (FTS), a native processor that uses Lua to express policies across all three telemetry signals. It provides typed access to values and OpenTelemetry context, including resources, scopes and attributes.

The “typed” part allows us to ensure data values such as identifiers represented as large integers need to survive a transformation unchanged. Rounding or truncation while modifying a field is likely to break the correlation of data values. FTS ensures that exact integers, nulls and binary values aren’t accidentally disrupted.

Strong typing also offers security benefits, as it can make it harder to induce memory overruns or underwrites.

We have previously touched upon the power of Lua in the Fluent Bit pipeline with use cases such as:

  • Redact named sensitive fields before forwarding telemetry.
  • Bring inconsistent attributes into a common form.
  • Reduce unnecessary events and metric dimensions. Reducing payload size can improve performance and lower storage costs.

FTS still uses Lua, so it doesn’t require learning a statically typed language. Existing Lua filters remain supported, although moving them to FTS requires adapting their API. This gives us a reason to evaluate FTS for new requirements without rewriting every working filter.

Reducing metric dimensions needs care.

A useful example in the FTS reference is deliberate metric aggregation. Removing a label can make previously distinct series identical. FTS provides an aggregation helper for combining compatible series, rather than assuming that deleting an attribute is sufficient.

Consider request counts broken down by individual customer identifiers. If our operational dashboard only needs route and response status, retaining customer-level detail may create a great deal of storage and query work without helping that dashboard.

The opportunity is to decide which detail serves an operational purpose before forwarding it. We still need to preserve the distinctions required for investigation and avoid combining measurements with incompatible meaning.

There is a similar decision around processing failures. By default, a failed FTS policy rolls back its changes and drops the affected item. The alternative keep mode forwards the original. For redaction, that could mean forwarding the very information we intended to remove. Error handling therefore belongs in the policy design and its tests.

Making system logs easier to use with OpenTelemetry

The new journald_otel processor maps journal records into OpenTelemetry logs, including severity, host and process information.

This addresses a familiar integration problem. Collecting a message is only part of making it useful; its fields also need to mean something to the tools querying it. Native mapping can reduce the custom transformations needed to bring Linux system logs alongside application telemetry.

We saw in the Moving Open Source Observability backend closer to turn-key solutions the value of normalising attribute names in telemetry.

Giving a busy destination time to recover

Fluent Bit 5.2 adds destination-aware output throttling for HTTP, OpenTelemetry, Elasticsearch and OpenSearch. Supported responses and retry hints can trigger a cooldown for that destination, and waiting through the cooldown does not consume retry attempts.

For Elasticsearch and OpenSearch bulk delivery, retries also retain unresolved records without replaying items already acknowledged as successful.

If a backend is overloaded, repeatedly sending it more work can prolong the problem. Responding to its request to slow down makes recovery more cooperative and avoids unnecessary traffic.

We still need sufficient buffering for the interruption. A cooldown changes delivery behaviour; it does not create unlimited capacity to absorb incoming data.

Performance that could translate into useful headroom

The published 5.2 benchmarks show substantial gains in particular workloads. In the memory-buffered OTLP/HTTP JSON test that adds one log attribute, throughput reaches 1.50 million records per second, around 7.2 times the 5.1 result when using native content_modifier processing.

The separate FTS log-policy performance reports 18–45% less CPU than equivalent Lua policies on 5.1.0. Given the additional overhead, this is an impressive performance boost.

The experimental browser SDK is another interesting direction: it runs Fluent Bit through WebAssembly. It deserves exploration, with its separate build requirements and experimental status in mind.

The buffering results are interesting as well. One sparse-stream test reports a 92.2% reduction in allocated disk space. An estate with many quiet streams may benefit differently from one dominated by a few busy streams.

We do have to treat these performance figures with care – simply assuming we see an existing deployment become seven times faster is foolhardy, as the figures are context-specific. But it is reasonable to assume that the new features will not adversely impact performance.

Smaller changes that help day-to-day operation

The release notes also describe YAML generation from command-line options and configuration-relative path handling for supported assets.

Plugins and Processor

As with all releases, there are a number of plugin upgrades and additions for the core, such as the output throttling mentioned, but we also have a new processor: journald_otel. The processor makes it quick and easy to map journald records directly to the OpenTelemetry structures (the alternative is a more complex mapping exercise).

Resources

  • Fluent Bit 5.2 release notes
  • Fluent Typed Scripting overview.
  • FTS examples and API reference
  • Fluent Bit 5.2 benchmark methodology and results

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d