• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Lua

Moving Open Source Observability backend closer to turn-key solutions

09 Monday Nov 2026

Posted by mp3monster in AI, Cloud Native, Fluent Observability, General, Technology

≈ Leave a comment

Tags

agentic, agents, AI, collector, Fluent Bit, Fluentd, Lua, observability, open-source, OpenTelemetry, OTel, OTTL, Transformation, Vector

In our first post on Moving Open-Source Observability closer to turn-key solutions, we discussed how we can automate the source-to-pipeline. But turn-key solutions also flow data to the backend and provide logs and metrics that are contextually aware of the systems being monitored.

While detecting sources and building the capture mechanism may seem more challenging at first glance, it is easier – we have a defined technology, and our observability agent defines the output structure. So what makes the pipeline to the backend more complex?

  • We’re more likely to have a variety of products, and some products may have functional overlaps.
  • Greater configuration diversity in how tools can be configured.
  • Greater contextual awareness is needed.

Let’s break these down in more detail.

Greater variety of tools means we need a common way to set up transformations; otherwise, the mechanics of a transformation process in our pipeline will explode in complexity.

One benefit of going open source is that you can choose the tools that fit your needs for log analytics, metrics, alert routing, and so on, but this can lead to functional overlaps. For example, you may use Prometheus for infrastructure-based metrics, but want to use Elastic APM for Application Performance Management (APM) as you’re using an Elastic Java stack for app development.

The different approaches to configuring tools, or configuration diversity, is probably the single largest challenge, and the number of approaches multiplies the number of configuration permutations that must be achievable. While there appears to be some initia behind the support the Prometheus mixin framework which provides templates for different scenarios such as infrastructure dashboards and alerts. It is far from universal adoption today. For example, Prometheus and Grafana support its use. But other solution stacks such as Elastic do not. If you align on a common technology stack, the chances of a harmonised configuration process rise.

We need contextual awareness from the source, as well as the collected data, to map data to the table(s). For efficiency, if we can avoid putting all metrics into a single table, queries could have one less dimension to filter. Context awareness will inform how we render the dashboards and set alert thresholds. For example, general filesystem storage capacity thresholds would be hugely different to database storage thresholds.

All of this means that if we want a deterministic, transformation-based process, we’ll end up with an unsustainable set of combinations that need to be mappable, testable, and maintainable as the different solutions grow, evolve, and occasionally get archived as contributions dry up.

As turnkey solutions, bring an opinionated answer and either build their own backends from scratch (e.g. Oracle’s Enterprise Manager) and leveraging open source as building blocks – they effectively eliminate the challenges of functional overlaps and configuration diversity – at the same time removing that flexibility. To underline how messy things could get, we experimented in a branch of the repo to see how messy it could get, while trying to keep the divergence as narrow as possible. This diagram illustrates the point visually.

The complexity and variability of pipeline tools and observability back ends

Solution

If a deterministic approach is unsustainable, are there options to ease the challenge or achieve a solution?

Simplification option

One option that can simplify, but certainly doesn’t solve, the challenge is to leverage OpenTelemetry’s work on semantic conventions, where they have been cultivating naming conventions that define data elements likely to be transmitted. When you manually map data, the process becomes more intuitive (and therefore quicker) and improves the chances of an AI agent mapping successfully. In some respects, this is like middleware-driven efforts to build data ontologies, so you need to avoid the problems global ontologies had.

Bringing AI to the game

Because configuration is linguistic and well documented, you can apply an LLM-based approach to the problem. The process of mapping from the ingestion part of the pipeline to the back end needs to be flow-driven so we can:

  • Perform a validation process by selecting and using validation tools to verify the configuration(s).
  • When validations fail, the process can be pushed back to the configuration stage again.
  • When users are in the loop, it becomes possible to drive a conversation for more clarity.
  • With a conversational process, it becomes possible to create in-pipeline transforms as a result of user conversation, along with simpler transforms to handle possible data formats and names.

What we’re describing here isn’t trivial, and could arguably be offered as a commercial product in its own right. The config management feature of our OpAMP solution also provides meaningful inputs, as it defines valid data types for observability agents like Fluent Bit where a concrete schema is not provided.

Using AI – agent graph

Agent descriptions

The agentic process illustrated in the preceding diagram is not trivial. It will be pretty context-heavy, as much of the information to work with can’t be easily addressed through targeted RAG or semantic search. In most cases, these core agents will need tools such as web search, since their training isn’t as up to date as the latest features of the different tools.

  • Input evaluator – We need a variety of inputs covering the target systems and, if there is overlap in capability, guidance on which ones to use. We need to tell the agent what tooling the pipeline needs. We’ll likely need metadata on the sources. For example, if we detect a database, we need to know which type so we can inject the correct queries. The input evaluator will use this to decide whether the detail is sufficient. The result of this analysis could be one of several things:
    • Use a client conversational agent that chats with the user to get clarifications; there is some overlap here, as the conversation will need to know what answers are needed, and some answers may open up new questions. Once this agent believes it has compiled the details for the answers. it returns to the Input Evaluator to reassess the picture.
    • We can ask the evaluator to pause the process and generate a report showing the gaps identified. This can be done using the Clarifications Report generator.
    • The Input evaluator may determine that it reports the gaps and presses on with a best effort.
  • Semantic normalizer – To simplify the flow, mapping source names to the semantic definitions from OpenTelemetry would need to understand the source and the OTel Semantic Naming model.
  • Pipeline generator – Once we confirm the details and have simplified semantics, we can have the AI build out the rest of the pipeline. The agent may need to be handed back to the agent for correction.
  • Config validator – once the pipeline generator hands off the configurations, the validator agent determines the best tools to verify whether the configurations are valid. If the tool returns an error, it hands the cause back to the pipeline generator. Ideally, tool calls will return meaningful error descriptions. But if not, we’d need to create a simple knowledge base to extract details from.
  • Clarifications Report generator – The clarifications report generator’s role is to take the gaps identified and generate and share them via a document repository or email in a user-friendly format
  • User clarification – The conversational interface with the user to obtain missing details, according to the Input evaluator. One difference between this and the Input evaluator is that this agent will need to consider conversational history to build the required additional detail. The input evaluator shouldn’t need prompting.

Controlling complexity

We’ve seen that allowing flexibility and removing the opinionated constraints of turnkey solutions increases complexity very quickly. The AI solution will become much more reliable if we strip back the options we allow agents to choose from, or if we push more organizations to standardize on consistent configuration strategies that enable a move toward templatization.

A Unified pipeline syntax

The other path that could radically simplify things is the Open Telemetry Transformation Language (OTTL), which has just reached the version 1 milestone. While it is not a complete solution, it offers a way to advance the standardized pipelines. We’ll explore OTTL in a separate blog in more depth. But essentially, if you look at the collector as two parts- the wire-level mechanism to capture or emit data and a second part for routing and manipulation- then OTTL is about the routing and manipulation. OTTL has both a formal syntax and an implementation as a Go library. While the library may not help if you’re using Vector, Fluentd, Fluent Bit, etc., the formal notation will work with ANTLR4; the resources are available through the OTLery OTTL-Lang repo. This allows us to generate, across many languages (C, Java, Ruby, Lua, etc.), a parser for the OTTL syntax into a tree structure that can then be walked. This still needs the intelligence to translate data references into references for the observability tool, and the functions, editors, and transformers to be implemented. Still, most of these can be mapped onto language-native libraries.

With a formal notation and a growing catalogue of prepared OTTL scripts and utilities, such as those offered by the OTLery repos, there is the ability to leverage AI to generate the transformation logic. As many observability tools such as Fluent Bit and Vector include support for scripting and plugins, incorporating a complete parser becomes a practical option. With both Vector and Fluent Bit, for example, supporting Lua being incorporated into the pipeline, and ANTLR4 supporting Lua as well, it should be possible to implement OTTL in those pipelines.

There is one downside to using OTTL: we lower our observability agents to the lowest common denominator. For example, Fluent Bit provides basic stream analytics. Transformations are now going through a parser phase rather than going directly to native binary processes.

Conclusion

The more we can restrict options, the closer we can get to being able to automate the creation of the observability pipeline quickly, and efficiently, but at the price of potentially not maximizing feature exploitation (the downside of using OTTL across different observability agents). The problem is to provide a solution that is very flexible, which will require a fair bit of effort. That flexibility also creates the possibility of exploiting the full feature set of the available tools.

Useful links

  • Prometheus Monitoring Mixins
  • OpenTelemetry Semantic Conventions
  • Elastic APM for Application Performance Management (APM)
  • Open Telemetry Transformation Language (OTTL)
  • OTLery OTTL-Lang
  • OTTL Go library
  • ANTLR4
  • Configuration Generator main repository

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit 4.0.4

21 Monday Jul 2025

Posted by mp3monster in development, Fluentbit, General, Technology

≈ 1 Comment

Tags

characters, encoding, filters, Fluent Bit, fstat, GBK, Lua, Open Telemetry, ShiftJIS, tail, Technology, WhatWG

The latest release of Fluent Bit is only considered a patch release (based on SemVer naming). But given the enhancements included it would be reasonable to have called it a minor change. There are some really good enhancements here.

Character Encoding

As all mainstream programming languages have syntaxes that lend themselves to English or Western-based languages, it is easy to forget that a lot of the global population use languages that don’t have this heritage, and therefore can’t be encoded using UTF-8. For example, according to the World Factbook, 13.8% speak Mandarin Chinese. While this doesn’t immediately translate into written communication or language use with computers, it is a clear indicator that when logging, we need to support log files that can be encoded to support idiomatic languages, such as Simplified Chinese, and recognized extensions, such as GSK and BIG5. However, internally, Fluent Bit transmits the payload as JSON, so the encoding needs to be handled. This means log file ingestion with the Tail plugin ideally needs to support such encodings. To achieve this, the plugin features a native character encoding engine that can be directed using a new attribute called generic. encoding, which is used to specify the encoding the file is using.

service:
  flush: 1

pipeline:
  inputs:
    - name: tail
      path: ./basic-file.gsk
      read_from_head: true
      exit_on_eof: true
      tag: basic-file
      generic.encoding: gsk

  outputs:
    - name: stdout
      match: "*"

The encoders supported out of the box, and the recognized names (in italics are) are:

  • GB18030 (earlier Simplifed Chinese Standard from the Chinese government called Information Technology – Chinese coded character set)
  • GBK (standard that extends the GB18030 standard for Simplified Chinese)
  • UHC (Unified Hangul Code also known as Extended Wandung – for Korean)
  • ShiftJIS (Japanese characters)
  • Big5 (for Chinese as used in Taiwan, Hong Kong, Macau)
  • Win866 (Cyrillic Russian)
  • Win874 (Thai)
  • Win1250 (Latin 2 & Central European languages)
  • Win1251 (Cyrillic)
  • Win1252 (Latin 1 & Western Europe)
  • Win1254 (Turkish)
  • Win1255 (also known as cp1255 and supports Hebrew)
  • Win1256 (Arabic)
  • Win2513 (suspect this should be Win1253, which covers the Greek language)

These standards are governed by the WhatWG specification (Web Hypertext Application Technology Group), not a well-known name, but have an agreement with the well-known W3C for various HTML and related standards.

The Win**** encodings are Windows-based formats that predate the adoption of UTF-8 by Microsoft.

Log Rotation handling

The Tail plugin, has also seen another improvement. Working with remote file mounts has been challenging, as it is necessary to ensure that file rotation is properly recognized. To improve the file rotation recognition, Fluent Bit has been modified to take full advantage of fstat. From a configuration perspective, we’ll not see any changes, but from the viewpoint of handling edge cases the plugin is far more robust.

Lua scripting for OpenTelemetry

In my opinion, the Lua plugin has been an underappreciated filter. It provides the means to create customized filtering and transformers with minimal overhead and effort. Until now, Lua has been limited in its ability to interact with OpenTelemetry payloads. This has been rectified by introducing a new callback signature with an additional parameter, which allows access to the OLTP attributes, enabling examination and, if necessary, return of a modified set. The new signature does not invalidate existing Lua scripts with the older three or four parameters. So backward compatibility is retained.

The most challenging aspect of using Lua scripts with OpenTelemetry is understanding the attribute values. Given this, let’s just see an example of the updated Lua callback. We’ll explore this feature further in future blogs.

function cb(tag, ts, group, metadata, record)
  if group['resource']['attributes']['service.name'] then
    record['service_name'] = group['resource']['attributes']['service.name']
  end

  if metadata['otlp']['severity_number'] == 9 then
    metadata['otlp']['severity_number'] = 13
    metadata['otlp']['severity_text'] = 'WARN'
  end

  return 1, ts, metadata, record
end

Other enhancements

With nearly every release of Fluent Bit, you can find plugin enhancements to improve performance (e.g., OpenTelemetry) or leverage the latest platform enhancements, such as AWS services.

Links

  • Fluent Bit
  • Fluent Bit on GitHub
  • Logs and Telemetry (Using Fluent Bit)

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit – using Lua script to split up events into multiple records

20 Friday Sep 2024

Posted by mp3monster in Fluentbit, General, Technology

≈ Leave a comment

Tags

FluentBit, Lua, plugins

One of the really advanced features of Fluent Bit’s use of Lua scripts is the ability to split a single log event so downstream processing can process multiple log events. In the Logging and Telemetry book, we didn’t have the space to explore this possibility. Here, we’ll build upon our understanding of how to use Lua in a filter. Before we look at how it can be done, let’s consider why it might be done.

Why Split Fluent Bit events

This case primarily focuses on the handling of log events. There are several reasons that could drive us to perform the split. Such as:

  • Log events contain metrics data (particularly application or business metrics). Older systems can emit some metrics through logging such as the time to complete a particular process within the code. When data like this is generated, ideally, we expose it to tools most suited to measuring and reporting on metrics, such as Prometheus and Grafana. But doing this has several factors to consider:
    • A log record with metrics data is unlikely to generate the data in a format that can be directed straight to Prometheus.
    • We could simply transform the log to use a metrics structure, but it is a good principle to retain a copy of the logs as they’re generated so we don’t lose any additional meaning, which points to creating a second event with a metrics structure. We may wish to monitor for the absence of such metrics being generated, for example.
  • When transactional errors occur, the logs can sometimes contain sensitive details such as PII (Personally Identifiable Information). We really don’t want PII data being unnecessarily propagated as it creates additional security risks – so we mask the PII data for the event to go downstream. But, at the same time, we want to know the PII ID to make it easier to identify records that may need to be checked for accuracy and integrity. We can solve this by:
    • Copying the event and performing the masking with a one-way hash
    • Create a second event with the PII data, which is limited in its propagation and is written to a data store that is sufficiently secured for PII data, such as a dedicated database

In both scenarios provided, the underlying theme is creating a version of the event to make things downstream easier to handle.

Implementing the solution

The key to this is understanding how the record construct is processed as it gets passed back and forth. When the Lua script receives an event, it arrives in our script as a table construct (Java developers, this approximates a HashMap), with the root elements of the record representing the event payload.

Typically, we’d manipulate the record and return it with a flag saying the structure has changed, but it is still a table. But we could return an array of tables. Now each element (array entry) will be processed as its own log event.

A Note on how Lua executes copying

When splitting up the record, we need to understand how Lua handles its data. if we tried to create the array with the code:

record1 = record
record2 = record
newRecord[record1, record2] 

Then we manipulated newRecord[1] We would still impact both records; this is because Lua, like its C underpinning, always uses shallow references rather than deep copies of objects. So we need to ensure we perform a deep copy before manipulating the records. You can see this in our example configuration (here on GitHub), or look at the following Lua code fragment:

function copy(obj)
  if type(obj) ~= 'table' then return obj end
  local res = {}
  for k, v in pairs(obj) do res[copy(k)] = copy(v) end
  return res
end

The proof

To illustrate the behavior, we have created a configuration with a single dummy plugin that only emits a single event. That event is then picked up by a Filter with our Lua script. After the filter, we have a simple output plugin. As a result of creating two records, we should see two output entries. To make it easy to compare, in the Lua script, we have a flag called deepCopy; when set to true – we’ll clone the records and modify payload values; when set to true – we then perform the split.

[SERVICE]
  flush 1

[INPUT]
    name dummy
    dummy {   "time": "12/May/2023:08:05:52 +0000",   "remote_ip": "10.4.72.163",   "remoteuser": "-",   "request": {     "verb": "GET",     "path": " /downloads/product_2",     "protocol": "HTTP",     "version": "1.1"   },   "response": 304}
    samples 1
    tag dummy1

[FILTER]
    name lua
    match *
    script ./advanced.lua
    call cb_advanced
    protected_mode true

[OUTPUT]
    name stdout
    match *

Limitations and solutions

While we can easily split events up and return multiple records, we can’t use different tags or timestamps. Using the same timestamp is pretty sensible, but different tags may be more helpful if we want to route the different records in other ways.

As long as the record contains the value we want to use as a tag, we can add to the pipeline a tag-write plugin and point it to the attribute to parse with a REGEX. To keep things efficient, if we create an element that is just the tag when creating the new record, then the REGEX becomes a very simple expression to match the value.

Conclusion

We’ve seen a couple of practical examples of why we might want to spin out new observability events based on what we get from our system. An important aspect of the process is how Lua handles memory.

Resources

  • Logging and Telemetry with Fluent Bit book
  • GitHub example
  • Tech resources

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d