When I wrote Logging In Action, I found that to help with setting up and testing logging pipelines, having a utility that could just take existing logs and play them so it looked like a real application was generating the logs was helpful – not only for the book, but in everyday work.
The natural evolution was: can I also create and run synthetic logs through observability systems so we can ensure log thresholds, filters, alerts, etc. worked as expected. Not to mention taking synthetic or real logs and applying some basic reformatting so we could mimic Apache or Nginx web servers, for example.
So the LogGenerator (sometimes called Log Simulator) was born and made available. At the time, my development skills were centred on Java (and I loved Groovy for its convenience and development speed). So we built the solution using Groovy and Java -the net result is that you could create a deployable JAR or run it without the need to first establish a Maven pipeline (without sacrificing the ability to keep polymorphism and abstract layers).
The LogGenerator has had a number of enhancements, so we could use it to check not only tools like Fluentd and Fluent Bit, but also cloud observability tooling such as OCI. This was made to be a pluggable framework so you could add custom targets easily.
Time moves on …
Python continues to grow in popularity and is the default language bundled with Linux. I’m comfortable developing in Python now, and we have the tools I enjoyed with Groovy and Java (no need to mess with build pipelines for quick changes, but it can also be packaged and easily deployed for mainstream adoption).
But it isn’t programming language evolution that has triggered change. OpenTelemetry standards and the golden signals of Logs, Traces and Metrics are the norm. So applying the same principles to the Traces and metrics that the LogGenerator used for logs is a natural evolution.
Deployment
It was pointed out to me that, even if people don’t take the full OpAMP solution we built, the simulator component may be attractive in its own right. This got us to thinking about whether there was value in pairing the SignalGenerator with the OpAMP simulator.
As a result, we created an optional feature that lets us deploy SignalGenerator as an extension of the OpAMP simulator. This gives us multiple deployment options.
A free-standing solution in its own right, with no need for anything else. Ideal for dropping into test environments to exercise your Observability configuration and tuning without needing start-up applications that require large deployments with complex inputs to generate unhappy path signals.
Deploy in a supervised manner using OpAMP – ideal if you want to drive synthetic Observability exercises in a distributed manner.
Plug into the OpAMP simulator – fewer processes running, easier to create custom commands through the OpAMP framework to get the generator to do things from pushing out new data sources to be replayed, to having signal generators simulate Observability data that reflects a distributed cascading failure as means to evaluate smarter AI-driven processes that should be looking for correlations and cause and effect.
The last part with custom messaging over OpAMP is not yet in place, but it’s a candidate for enhancement.
New home
As the Logging in Action book uses the LogGenerator, we didn’t want to make this level of change in the current repository. So we have a new repository within the OpAMP-Observe GitHub organization which has feature parity, but also has a mechanism for simulating metrics (pushing, or allowing the data to be scraped).
The documentation to link the repositories and the GitHub documentation together is a little bit behind, and we need to figure out how to create a manageable playback structure for Traces before we enable that feature.
In our first post on Moving Open-Source Observability closer to turn-key solutions, we discussed how we can automate the source-to-pipeline. But turn-key solutions also flow data to the backend and provide logs and metrics that are contextually aware of the systems being monitored.
While detecting sources and building the capture mechanism may seem more challenging at first glance, it is easier – we have a defined technology, and our observability agent defines the output structure. So what makes the pipeline to the backend more complex?
We’re more likely to have a variety of products, and some products may have functional overlaps.
Greater configuration diversity in how tools can be configured.
Greater contextual awareness is needed.
Let’s break these down in more detail.
Greater variety of tools means we need a common way to set up transformations; otherwise, the mechanics of a transformation process in our pipeline will explode in complexity.
One benefit of going open source is that you can choose the tools that fit your needs for log analytics, metrics, alert routing, and so on, but this can lead to functional overlaps. For example, you may use Prometheus for infrastructure-based metrics, but want to use Elastic APM for Application Performance Management (APM) as you’re using an Elastic Java stack for app development.
The different approaches to configuring tools, or configuration diversity, is probably the single largest challenge, and the number of approaches multiplies the number of configuration permutations that must be achievable. While there appears to be some initia behind the support the Prometheus mixin framework which provides templates for different scenarios such as infrastructure dashboards and alerts. It is far from universal adoption today. For example, Prometheus and Grafana support its use. But other solution stacks such as Elastic do not. If you align on a common technology stack, the chances of a harmonised configuration process rise.
We need contextual awareness from the source, as well as the collected data, to map data to the table(s). For efficiency, if we can avoid putting all metrics into a single table, queries could have one less dimension to filter. Context awareness will inform how we render the dashboards and set alert thresholds. For example, general filesystem storage capacity thresholds would be hugely different to database storage thresholds.
All of this means that if we want a deterministic, transformation-based process, we’ll end up with an unsustainable set of combinations that need to be mappable, testable, and maintainable as the different solutions grow, evolve, and occasionally get archived as contributions dry up.
As turnkey solutions, bring an opinionated answer and either build their own backends from scratch (e.g. Oracle’s Enterprise Manager) and leveraging open source as building blocks – they effectively eliminate the challenges of functional overlaps and configuration diversity – at the same time removing that flexibility. To underline how messy things could get, we experimented in a branch of the repo to see how messy it could get, while trying to keep the divergence as narrow as possible. This diagram illustrates the point visually.
Solution
If a deterministic approach is unsustainable, are there options to ease the challenge or achieve a solution?
Simplification option
One option that can simplify, but certainly doesn’t solve, the challenge is to leverage OpenTelemetry’s work on semantic conventions, where they have been cultivating naming conventions that define data elements likely to be transmitted. When you manually map data, the process becomes more intuitive (and therefore quicker) and improves the chances of an AI agent mapping successfully. In some respects, this is like middleware-driven efforts to build data ontologies, so you need to avoid the problems global ontologies had.
Bringing AI to the game
Because configuration is linguistic and well documented, you can apply an LLM-based approach to the problem. The process of mapping from the ingestion part of the pipeline to the back end needs to be flow-driven so we can:
Perform a validation process by selecting and using validation tools to verify the configuration(s).
When validations fail, the process can be pushed back to the configuration stage again.
When users are in the loop, it becomes possible to drive a conversation for more clarity.
With a conversational process, it becomes possible to create in-pipeline transforms as a result of user conversation, along with simpler transforms to handle possible data formats and names.
What we’re describing here isn’t trivial, and could arguably be offered as a commercial product in its own right. The config management feature of our OpAMP solution also provides meaningful inputs, as it defines valid data types for observability agents like Fluent Bit where a concrete schema is not provided.
Using AI – agent graph
Agent descriptions
The agentic process illustrated in the preceding diagram is not trivial. It will be pretty context-heavy, as much of the information to work with can’t be easily addressed through targeted RAG or semantic search. In most cases, these core agents will need tools such as web search, since their training isn’t as up to date as the latest features of the different tools.
Input evaluator – We need a variety of inputs covering the target systems and, if there is overlap in capability, guidance on which ones to use. We need to tell the agent what tooling the pipeline needs. We’ll likely need metadata on the sources. For example, if we detect a database, we need to know which type so we can inject the correct queries. The input evaluator will use this to decide whether the detail is sufficient. The result of this analysis could be one of several things:
Use a client conversational agent that chats with the user to get clarifications; there is some overlap here, as the conversation will need to know what answers are needed, and some answers may open up new questions. Once this agent believes it has compiled the details for the answers. it returns to the Input Evaluator to reassess the picture.
We can ask the evaluator to pause the process and generate a report showing the gaps identified. This can be done using the Clarifications Report generator.
The Input evaluator may determine that it reports the gaps and presses on with a best effort.
Semantic normalizer – To simplify the flow, mapping source names to the semantic definitions from OpenTelemetry would need to understand the source and the OTel Semantic Naming model.
Pipeline generator – Once we confirm the details and have simplified semantics, we can have the AI build out the rest of the pipeline. The agent may need to be handed back to the agent for correction.
Config validator – once the pipeline generator hands off the configurations, the validator agent determines the best tools to verify whether the configurations are valid. If the tool returns an error, it hands the cause back to the pipeline generator. Ideally, tool calls will return meaningful error descriptions. But if not, we’d need to create a simple knowledge base to extract details from.
Clarifications Report generator – The clarifications report generator’s role is to take the gaps identified and generate and share them via a document repository or email in a user-friendly format
User clarification – The conversational interface with the user to obtain missing details, according to the Input evaluator. One difference between this and the Input evaluator is that this agent will need to consider conversational history to build the required additional detail. The input evaluator shouldn’t need prompting.
Controlling complexity
We’ve seen that allowing flexibility and removing the opinionated constraints of turnkey solutions increases complexity very quickly. The AI solution will become much more reliable if we strip back the options we allow agents to choose from, or if we push more organizations to standardize on consistent configuration strategies that enable a move toward templatization.
A Unified pipeline syntax
The other path that could radically simplify things is the Open Telemetry Transformation Language (OTTL), which has just reached the version 1 milestone. While it is not a complete solution, it offers a way to advance the standardized pipelines. We’ll explore OTTL in a separate blog in more depth. But essentially, if you look at the collector as two parts- the wire-level mechanism to capture or emit data and a second part for routing and manipulation- then OTTL is about the routing and manipulation. OTTL has both a formal syntax and an implementation as a Go library. While the library may not help if you’re using Vector, Fluentd, Fluent Bit, etc., the formal notation will work with ANTLR4; the resources are available through the OTLery OTTL-Lang repo. This allows us to generate, across many languages (C, Java, Ruby, Lua, etc.), a parser for the OTTL syntax into a tree structure that can then be walked. This still needs the intelligence to translate data references into references for the observability tool, and the functions, editors, and transformers to be implemented. Still, most of these can be mapped onto language-native libraries.
With a formal notation and a growing catalogue of prepared OTTL scripts and utilities, such as those offered by the OTLery repos, there is the ability to leverage AI to generate the transformation logic. As many observability tools such as Fluent Bit and Vector include support for scripting and plugins, incorporating a complete parser becomes a practical option. With both Vector and Fluent Bit, for example, supporting Lua being incorporated into the pipeline, and ANTLR4 supporting Lua as well, it should be possible to implement OTTL in those pipelines.
There is one downside to using OTTL: we lower our observability agents to the lowest common denominator. For example, Fluent Bit provides basic stream analytics. Transformations are now going through a parser phase rather than going directly to native binary processes.
Conclusion
The more we can restrict options, the closer we can get to being able to automate the creation of the observability pipeline quickly, and efficiently, but at the price of potentially not maximizing feature exploitation (the downside of using OTTL across different observability agents). The problem is to provide a solution that is very flexible, which will require a fair bit of effort. That flexibility also creates the possibility of exploiting the full feature set of the available tools.
Last week saw the Open Source Observability Day (OSOD) during which I presented on the subject of the Open Agent Management Protocol, covering the underlying protocol that forms the foundation of a wider capability we have been building along with a demo of what is available.
If you want to see the presentation, and hear the full story not only of the core protocol but how it can help with the ideas around ChatOps and AIOps head over to the OSOD website (and previous years can be seen on YouTube with playlists for each year e.g. OSOD 2025). Of course if you have been following my blog posts you’ll know this.
The demo is recorded against a version prior to the 0.6 release.
Running the demo yourself
If you want to try this out for yourself, you can do so. You’ll need to:
There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:
Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
Deployed services and applications with default monitoring configuration
Prebuilt dashboards
While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.
This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.
Detecting and Generating Configuration
The process we need has the following steps:
find the deployed solution(s)
Understand how they are deployed
Generate a configuration based on the detection
Push the configuration out to agents
Detect and capture config
One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.
While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …
As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.
Templates make the result reusable
By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.
There are limits
So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.
Convenience as a weakness?
While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.
Backend Bootstrapping
Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.
A FIPS safe mode to ensure things are safe during start and restart.
Ensuring timestamps will work beyond 2038.
TLS handling refinements such as certificate reloading
Management of the internal buffers can be made more dynamic now.
Improvements across a number of plugins for input and output.
Dynamic flushing to allow dynamic optimization within configured boundaries.
Packaging improvements for Windows Nano and Debian deployments.
New root section called Extensions.
A lot of these gains are great, and will really pay off for high volume deployments, where small savings, cumulative can really payoff economically. It is the new extensions feature that I’m most interested in as it open up some capabilities when managing the observability domain.
Extensions
The extension structure (first identified as a useful feature with issue 11863) doesn’t impact Fluent Bit behavior but is retained and accepted as YAML configuration values. This means that any tools being used to manage Fluent Bit configuration can now add YAML key value pairs (including nested values) without causing Fluent Bit a problem, and can therefore be used by management tools. For example we can track what version of a configuration is being handled by Fluent Bit by injecting a Git version identifier (just as you see in the Fluent Bit output at startup). This simple change means that it is now possible to understand which version of configuration is being executed. When you’re rolling out to a large estate (particularly outside of containerized environments) it is easy to know what has been rolled out.
extensions:
opamp:
endpoint: 127.0.0.1:4318
insecure: true
deployment:
config_version: 12345
If you’re using Fleet Management capabilities such as OpAMP (a specification developed as part of the Open Telemetry project) you can embed the client side controller/supervisor configuration within the Fluent Bit configuration (as long as the controller supervisor understand where to get the values from within a YAML file).
It also means that you’re allowing more dynamic changes, it becomes easy to notate such dynamic changes., for example:
extensions:
deployment:
changeDTG: 2026.08.05.16.25
notes:
- changed flush interval
- moved plugin x output
Personally I’d not advocate changes without using configuration control tooling, but it certainly is better than no record.
Next steps
If you follow this blog regularly, they’ll be aware of the OpAMP project which implements the protocol, and provides a server that includes fleet management and configuration editing and validation for Fluent Bit server side, and a client supervisor/observer on client client/consumer side which can manage Fluent Bit instances. These changes mean that the project will need:
New configuration and validation settings (the UI is entirely meta-data driven) for the new properties.
The configuration catalogue viewer will need to be updated to exploit the extensions feature for showing versioning.
The client/consumer side can be extended to become Fluent Bit aware and exploit the embeddable extensions capability.
We may also offer the dynamic change recording possibility as an optional feature (from a product view – we shouldn’t force specific ways of working, but from a personal view it isn’t a way I’d recommend working)
The Elastic toolset for monitoring and observability with the Beats tools (heartbeat, filebeat etc) and more recently an integrated OTel compliant agent have been attractive because of the strength of the analysis capabilities of Elasticsearch, Logstash and Kibana. All the products are available as open source and can be extended (although with 3rd party constraints as a result of licensing) and with enterprise extensions.
Kibana while focused on data visualisation, also provides the fleet management layer to control the remote agents. Kibana’s communication with the agents now makes use of protobuf. With clear commenting discouraging the use.
This is great if your entire ecosystem is aligned to the Elastic stack, but that is rarely the case. If we’re not already in the era of polyglot, then pervasive use of AI to power development, both at departmental level (shadow/gray IT) and even citizen development.
Managing distributed instances of the Elastic Agent outside of Kibana has to be done using the agent’s command line interface. The only publicly documented web interface allows the retrieval of status information. This does feel rather poor as a means to drive the adoption of Kibana (and encourage the use of Elastic Cloud).
This leads us to the question of whether to use the Supervisor or Observer model of using the OpAMP standard, or should we try to embed the OpAMP client directly into the Elastic Agent. Having studied the documentation and some of the code base it feels like these sort of customisations are not encouraged, and emphasis for extensions are about adding the means to monitor different protocols or products. Incorporating socket or HTTP handling that also needs to interact with lifecycle logic would be very invasive.
Further more, invasive changes may prove to be more problematic if you switch from a forked open-source version to an enterprise licensed version, where you’d want both the benefits of OpAMP and the licensed extras.
The way we have designed and implemented our client so that it is easy to implement specific logic for different observability tool, and the development of the elastic agent logic, meant we took the final step of adopting a fully pluggable mechanism – we’ll come back to those details shortly..
Our implementation of the Elastic Agent management supports both models of supervisor or observer, but we would err towards using it in an observer mode. The control aspects of the agent are mapped onto using creating commands and using the CLI, as with Fluent Bit basic health can be retrieved via the agents REST endpoint.
The command line does allow for more diagnostic information. But we’ll look at that later.
What’s new as a result of the modifications
The improvements are being incorporated into a branch in the repo while we do regression testing, once we’re happy we’ll merge into main, and label.
Starting with the simple things:
We have a simple validation setup that can be run to ensure you have the elastic agent and a containerised Logstash that can talk to each other. The agent is configured to monitor itself to generate traffic to Logstash.
That configuration has been mapped into the example setup of the agent being managed by the OpAMP consumer.
We’ve enhanced the OpAMP-CLI tool so it can launch the containers using Docker or Podman. This makes it quick and easy to configure and launch demos and end-to-end regression tests. There are also some convenience tweaks to the way the CLI works.
New documentation explaining how to implement and deploy the management of your own Observability product.
We’ve refactored part of the consumer code to explicitly separate the specific code for the agent being managed be that Fluent Bit, Fluentd, Elastic Agent etc. the test code reflects the same structure.
The linkage to the Observability tools is dynamic now. Rather than have a central package which imports all the plugins.
There are a few things we haven’t done,..
We’ve only provided support for operations such as start and stop, so the server side of the product only works with the core server features. There is no configuration editor.
The MCP broker hasn’t been enhanced to understand differences between the client types. This is also true with the Slack interfacing. We will return these areas, but the most valuable thing that can be done is to make the OpAMP solution cover a variety of Observability agents.
We still need to improve the automated regression testing.
Extend to cover the individual beats components.
All these improvements will start arriving in the GitHub repo in the next couple of days. Once we think everything is good, we’ll up issue the release number – sdo watch the github repo 🙂
Another aspect of OpenTelemetry’s OpAMP protocol is the ability to share connections details, not just between the server side of the OpAMP protocol, but with other connections a collector like Fluent Bit could be using such as an OTel consumer for logs, traces and metrics that are being collected that could be a managed Grafana service or another Fluent Bit if you’re using a concentrator network.
This is includes Observability data generated by the collector I.e. Fluent Bit in our case.
Yes, but Fluent Bit doesn’t push its own OTel signals
It is true that out of the box Fluent Bit doesn’t push OTel signals for itself. But this is easily fixed. We can control logging, and metrics that are exposed (and today no traces are produced). The solution is simple, we use Fluent Bit itself to collect the logs and metrics and route them to using an OTel output plugin.
While creating a configuration for sharing seems onerous, once you’ve done it once, you can simply include the config file in every deployment. Further more, if you backend for any of the signals is not Fluent Bit conversant, just use a different Fluent Bit plugin.
Config walkthrough
Our configuration consists of several files, which we’ll explain. We’ve adopted a multi-file setup, which works through the use of Fluent Bit’s includes capability so we can separate out the application-level observability configuration (master.yaml), the configuration to allow us to observe Fluent Bit itself (otel.yaml), and the configuration for the connection parameters (otel-config.yaml).
Master.yaml
The master.yaml contains our application observability settings and is the configuration file we pass to Fluent Bit. In this case, we’re simply using a dummy input that’s fed to a file output; we’ve also got a wildcard stdout configuration to make it easy to observe what’s going on from the command line.
There are at this level, two telltale signs of self-monitoring at this level:
The includes declaration picks up otel.yaml, which includes additional inputs and outputs, as well as service configurations.
The service settings that switch on the HTTP server. It is controlled via environment variables, so you could switch this off easily. We have to implement these settings here, as within the full set of file inclusions, there can only be a single service block.
The service declaration also specifies where we want the log file that the Fluent Bit setup needs to go.
otel.yaml
We’ve separated the pipeline definitions for Fluent Bit observability to reuse the configuration across many deployments. As different deployments may need to talk to different instances of the observability backend (for example, when operating in a multi-cloud arrangement we have defined the environment variables that provide the credentials and target address).
We have defined additional environment variables at this level, so that anyone needing to understand master settings and the service values needs only look here. How this layer (or layers) is established is, to an extent, concealed. It also means we can use OpAMP to generate a configuration file, but we’ll come back to that.
The environment variables that we define to be used by the master.yaml switch the HTTP server on, and indicate which ports to use. Making this easy to check for the person or people defining the master.yaml makes it easier to avoid port collisions.
We also tell Fluent Bit where to write its logs through the service setting. To consolidate the configuration, that value references a different environment variable, so we can put all of our configurations together. Even if we didn’t have this level of redirection, it is worthwhile setting the value using an environment variable as the file location is needed both as our output for the service and the input for a plugin.. As you can see, we’ve just named the file. So the file created will be relative to wherever we run Fluent Bit from.
Let’s look at the inputs and outputs being used, and what they do:
Tail – this is the input that reads Fluent Bit’s logs, as we set Fluent Bit to log at debug level, we’ll see plenty of activity. Having Fluent Bit write to the file system, only for a thread in the same process to read it, is obviously inefficient. It can be optimised by using a RAMFS-like filesystem on Linux and a RAMDisk-like filesystem on Windows, so everything is in memory.
Fluentbit_metrics – this is the active collection of metrics, which we simply point to the server part of Fluent Bit. If you’re using Fluent Bit as a sidecar in a pod, this has to be done carefully because of how networking for containers within a pod is handled.
opentelemetry – This takes our different input signals we’ve been collecting and pushes them to the OTel-compliant service. If we had different backend products for different types of signals, then we’d need different or multiple plugins defined. Although OpAMP does infer an OTel-compliant backend.
That is the hard bit done. As a result of these inputs, we get the metrics and logs (and no traces are available). We now just need an OpenTelemetry output plugin to direct all the OTLP-represented signals. For simplicity, we’ve made use of a free account on Grafana Cloud.
otel-config.yaml
This configuration file is simple, which is key to making it easy for us to manage the connections setup using OpAMP. It simply defines the connection-based attributes needed, as shown:
While populating credentials in a file isn’t ideal, it makes it pretty straightforward for an OpAMP Client (observer or supervisor) to receive the credentials for the management side of the protocol, make them available to Fluent Bit quickly, and have Fluent Bit pick them up dynamically via hot-deploy functionality. This is precisely what we have implemented in our OpAMP. This approach works for both Fluent Bit and Fluentd configurations and we can even apply a similar approach for driving clients such as Elastic Beats.
This configuration file is simple enough that we can rewrite it whenever the configuration values are amended; there is no complexity in inserting such settings into the broader configuration. Furthermore, if Fluent Bit were running as an observed process rather than a supervised (i.e., child) process, setting environment variables could become problematic.
We can mitigate credential-related challenges in the file system by restricting filesystem permissions to the file being written, so only certain users and processes can access it. When working with containers, we can also keep this part of the file system within the image by using a transient storage layer.
Alternative strategy for securing credentials
An alternative approach would be for the client to receive the credentials and store them in an encrypted method, such as a credentials vault, and then the launcher for the process would read and decrypt the file as it launches Fluent Bit or another collector. The way we have implemented the client for our OpAMP certainly makes this possible, but it also complicates the process and suggests that the mechanics of credential management would be better handled by a distributed vault solution.
OpAMP’s assumption
The only niggle, is that when passing the connections, the protocol assumes that connection details are only needed for the collector. That’s fine if the collector includes the client side of the protocol. But as we know, the collector may be managed with a supervisor or observer process. We could use the connection for the collector and our supervisor, but that may not be what is wanted.
Visualising the configuration
We can visualise the relationships, like this:
Unpacking the server-side aspects
As you start to think about the server aspect of this, it gets potentially very complex. In simple scenarios, sending everything to a single Observability backend isn’t an issue.
But if you’re managing a multi-cloud, hybrid, or managed client setup, you’re likely to want separate backend instances for different collectors. In our setup, that means passing the different connection details to the client to populate otel-config.yaml. Understanding this deployment will require understanding the distribution. We will also need to manage potentially a concentrated catalogue of credentials, which will need to map to the collector nodes, and a means for the user to define the mapping. Creating a means to visualise the mapping, and ideally to allow the server to apply rules or infer assignments for more complex use cases such as multi-cloud, isn’t going to be simple.
Conclusion
We have a simple means to deploy configuration and observe our collector and the supervisor/observer process. Making server-side management in a large-scale environment easy to work with will require some consideration and potentially additional optional metadata. But the protocol supports all of this, but doesn’t mandate specifics. Which does mean that either the server has to be smart enough to know when what to do if it interacts with a client that has been built or configured to work with this server – which is an essential requirement, otherwise we lose the value of the
So we’ve been busy working on our OpAMP solution. We’ve made a number of enhancements since we labelled the code v0.4 back in April. For this post, we’ll look at the features added and where we’re looking next.
There has been a lot of feature development, particularly in support of working with Fluent Bit (and to a degree Fluentd).
Standalone or OpAMP server plugin for editor
We started building the Configuration Editor as a standalone capability. The thinking was that we would refactor it into the OpAMP server once we were happy with the functionality and had progressed far enough. But, as we saw this come together, it occurred to me that both deployments are good, as part of the OpAMP server, seeing the configuration being used is handy, but having a freestanding editor (without worrying about the connectivity to communicate with agents) is also a real use case.
So we have created the setup, where if the OpAMP is told to look for the Editor via the definition of Python endpoints in the configuration and it is deployed, it will be incorporated into the server. If the editor isn’t provided (deployed, or not identified in the configuration it isn’t offered in the navigation.
The editor is completely configuration-driven through JSON, so it only requires extending the JSON to support custom plugins or to enhance the validation rules (e.g., adding a REGEX to how a particular parameter is set). This far outweighs the current Dry run checks.
The configuration is considerable, so we refactored the structure to make it far easier to work with, and created some code that can mine Fluent Bit’s GitHub to generate an initial clean set of JSON docs. Of course these need an eyeballing to ensure they’re correct.
Catalog viewer
The catalogue viewer is a natural extension of the editor and leverages the way we tag configuration files with version details in the editor. The catalogue viewer has a configuration which tells it where to look for candidate files. These are then listed with the metadata.
The catalogue viewer presents all the identified files, which, when selected, if it knows about the config editor, will open the editor with the file. Otherwise, it opens a simple view of the file.
The catalogue viewer works in the same way as the editor in terms of authentication.
CLI
A Command Line tool maybe and odd choice of feature, but we found ourselves creating more and more scripts to support Windows and Bash shells with commonality. So we elected to leverage some frameworks to help reduce the ongoing effort required to maintain them. The utility can be used to generate the command, if you wanted embed the process of starting or stopping a process into the host OS.
Agent as Supervisor or observer
The OpAMP documentation suggests that the agent logic is embedded, or wrapped with a supervisor, with the inference that the agent forks the application process. This means that introducing the OpAMP would be invasive. There is a noninvasive variant of the Supervisor that we’ve called Observer. Here, the agent know how to identify the key process by examining the host’s processes. Then tasks such as restarting require an understanding of how to get the OS to trigger, for example, if the service is known to init.d in Linux, we can use the service command.
Architecture view
MCP and Slack for ChatOps
We’ve got the Slack foundations progressed, so we can use natural language VIA langgraph to then work with MCP, which exposes a subset of API capabilities. We’ve developed both the main server and the Broker to support the MCP endpoints, with the Broker acting as a proxy to the regular Server APIs. This means if we want the MCP to be usable from outside our network, then we can separate the Broker and Agent into separate networks so that the wider set of endpoints the server provides aren’t exposed.
But if you don’t want that the MCP endpoint can be switched on.
What next …
Config deployment
We have started to address the foundations of managing configuration deployment (for example, if our Catalogue Service is used with the Server, it can be used to select the configuration files to deploy.
Part of config deployment is understanding what is deployed, so we have developed some strategies for versioning Fluent Bit and Fluentd, and I think it will translate to other configuration files (at least into the Observability space).
What we haven’t done is fully implement the process. So we’ll focus on that
E2E Testing
There is a lot of functionality here now, including tests with Playwright, but we really need to extend it to provide end-to-end tests in a clean environment that is set up from the various pip and wheel files.
Deployment Artefact access
We also want to start making these artefacts easier to retrieve, such as pulling the Wheel or PIP files from GitHub or PyPi.
Fluent Bit’s documentation is spread across many static web pages; in some cases, attributes allowed for a plugin are documented across several pages. There is absolutely nothing wrong with this. Having written a book on Fluent Bit, I can say it makes sense, and if the standard docs went into further detail, it would increase the spread of content.
The problem is, once you’ve got a grip on Fluent Bit, you want a quick reference just to check the attribute names or confirm that an attribute does what you expect.
Each quick reference page has a section at the top that provides a comma-separated list of all the plugins for the different operations, inputs, outputs, etc., with anchor links to that section. Each plugin has a table that lists all configuration attributes, including those inherited from the Fluent Bit kernel and those introduced by extending another plugin.
Why and how …
The “why” may prove even more appealing. As part of our OpAMP project, we wanted to make it easy to edit and validate Fluent Bit and Fluentd configurations before deployment using the OpAMP tooling. That tool became the config-service part of the OpAMP repo, and can be independently deployed as well as function as part of the main OpAMP service. As a result, the UI offers the same authentication and authorisation options, ranging from running without authentication to using OAuth.
All the different editable components collapsedPlugins part of the editor
The key to both the UI and documentation is the use of JSON Schemas, as they contain all the information needed to create documentation just as easily as they power the UI. We have therefore generated a markdown page using a Python script. This means the docs are easy to check (compared to an HTML file) and can be rendered within GitHub. We could extend the script to generate HTML. But rather than trying to remember to keep both document types in sync (and double-check formatting), we found a JavaScript tool (marked.js) that performs an on-the-fly transformation that, as long as we stay within the core Markdown syntax, won’t cause any issues. We’ve then enriched that output a bit by applying stylesheets.
To come…
At the moment, we have only covered a subset of Fluent Bit versions, so we will, in due course, expand to cover more versions. Today, this is just the last version from each major release. We haven’t done every version to date as it does take a lot of effort to go through the documentation to generate and check the schema, and we’re still refining things as we enhance the UI.
We are also looking to do the same for Fluentd, though this is much trickier, as the portfolio of plugins that make up the core is smaller than those incorporated into the core of Fluent Bit, and the leveraging of Ruby’s dynamic behaviour makes it very easy for people to offer plugins separately. Then, of course, there is the task of collating all that information.
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{
"type":"object",
"title":"Dummy",
"properties":{
"name":{
"type":"string",
"const":"dummy",
"title":"Plugin"
},
"copies":{
"type":"integer",
"description":"Number of messages to generate each time messages are generated.",
The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
Environment variables
Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
You must be logged in to post a comment.