When I wrote Logging In Action, I found that to help with setting up and testing logging pipelines, having a utility that could just take existing logs and play them so it looked like a real application was generating the logs was helpful – not only for the book, but in everyday work.
The natural evolution was: can I also create and run synthetic logs through observability systems so we can ensure log thresholds, filters, alerts, etc. worked as expected. Not to mention taking synthetic or real logs and applying some basic reformatting so we could mimic Apache or Nginx web servers, for example.
So the LogGenerator (sometimes called Log Simulator) was born and made available. At the time, my development skills were centred on Java (and I loved Groovy for its convenience and development speed). So we built the solution using Groovy and Java -the net result is that you could create a deployable JAR or run it without the need to first establish a Maven pipeline (without sacrificing the ability to keep polymorphism and abstract layers).
The LogGenerator has had a number of enhancements, so we could use it to check not only tools like Fluentd and Fluent Bit, but also cloud observability tooling such as OCI. This was made to be a pluggable framework so you could add custom targets easily.
Time moves on …
Python continues to grow in popularity and is the default language bundled with Linux. I’m comfortable developing in Python now, and we have the tools I enjoyed with Groovy and Java (no need to mess with build pipelines for quick changes, but it can also be packaged and easily deployed for mainstream adoption).
But it isn’t programming language evolution that has triggered change. OpenTelemetry standards and the golden signals of Logs, Traces and Metrics are the norm. So applying the same principles to the Traces and metrics that the LogGenerator used for logs is a natural evolution.
Deployment
It was pointed out to me that, even if people don’t take the full OpAMP solution we built, the simulator component may be attractive in its own right. This got us to thinking about whether there was value in pairing the SignalGenerator with the OpAMP simulator.
As a result, we created an optional feature that lets us deploy SignalGenerator as an extension of the OpAMP simulator. This gives us multiple deployment options.
A free-standing solution in its own right, with no need for anything else. Ideal for dropping into test environments to exercise your Observability configuration and tuning without needing start-up applications that require large deployments with complex inputs to generate unhappy path signals.
Deploy in a supervised manner using OpAMP – ideal if you want to drive synthetic Observability exercises in a distributed manner.
Plug into the OpAMP simulator – fewer processes running, easier to create custom commands through the OpAMP framework to get the generator to do things from pushing out new data sources to be replayed, to having signal generators simulate Observability data that reflects a distributed cascading failure as means to evaluate smarter AI-driven processes that should be looking for correlations and cause and effect.
The last part with custom messaging over OpAMP is not yet in place, but it’s a candidate for enhancement.
New home
As the Logging in Action book uses the LogGenerator, we didn’t want to make this level of change in the current repository. So we have a new repository within the OpAMP-Observe GitHub organization which has feature parity, but also has a mechanism for simulating metrics (pushing, or allowing the data to be scraped).
The documentation to link the repositories and the GitHub documentation together is a little bit behind, and we need to figure out how to create a manageable playback structure for Traces before we enable that feature.
In our first post on Moving Open-Source Observability closer to turn-key solutions, we discussed how we can automate the source-to-pipeline. But turn-key solutions also flow data to the backend and provide logs and metrics that are contextually aware of the systems being monitored.
While detecting sources and building the capture mechanism may seem more challenging at first glance, it is easier – we have a defined technology, and our observability agent defines the output structure. So what makes the pipeline to the backend more complex?
We’re more likely to have a variety of products, and some products may have functional overlaps.
Greater configuration diversity in how tools can be configured.
Greater contextual awareness is needed.
Let’s break these down in more detail.
Greater variety of tools means we need a common way to set up transformations; otherwise, the mechanics of a transformation process in our pipeline will explode in complexity.
One benefit of going open source is that you can choose the tools that fit your needs for log analytics, metrics, alert routing, and so on, but this can lead to functional overlaps. For example, you may use Prometheus for infrastructure-based metrics, but want to use Elastic APM for Application Performance Management (APM) as you’re using an Elastic Java stack for app development.
The different approaches to configuring tools, or configuration diversity, is probably the single largest challenge, and the number of approaches multiplies the number of configuration permutations that must be achievable. While there appears to be some initia behind the support the Prometheus mixin framework which provides templates for different scenarios such as infrastructure dashboards and alerts. It is far from universal adoption today. For example, Prometheus and Grafana support its use. But other solution stacks such as Elastic do not. If you align on a common technology stack, the chances of a harmonised configuration process rise.
We need contextual awareness from the source, as well as the collected data, to map data to the table(s). For efficiency, if we can avoid putting all metrics into a single table, queries could have one less dimension to filter. Context awareness will inform how we render the dashboards and set alert thresholds. For example, general filesystem storage capacity thresholds would be hugely different to database storage thresholds.
All of this means that if we want a deterministic, transformation-based process, we’ll end up with an unsustainable set of combinations that need to be mappable, testable, and maintainable as the different solutions grow, evolve, and occasionally get archived as contributions dry up.
As turnkey solutions, bring an opinionated answer and either build their own backends from scratch (e.g. Oracle’s Enterprise Manager) and leveraging open source as building blocks – they effectively eliminate the challenges of functional overlaps and configuration diversity – at the same time removing that flexibility. To underline how messy things could get, we experimented in a branch of the repo to see how messy it could get, while trying to keep the divergence as narrow as possible. This diagram illustrates the point visually.
Solution
If a deterministic approach is unsustainable, are there options to ease the challenge or achieve a solution?
Simplification option
One option that can simplify, but certainly doesn’t solve, the challenge is to leverage OpenTelemetry’s work on semantic conventions, where they have been cultivating naming conventions that define data elements likely to be transmitted. When you manually map data, the process becomes more intuitive (and therefore quicker) and improves the chances of an AI agent mapping successfully. In some respects, this is like middleware-driven efforts to build data ontologies, so you need to avoid the problems global ontologies had.
Bringing AI to the game
Because configuration is linguistic and well documented, you can apply an LLM-based approach to the problem. The process of mapping from the ingestion part of the pipeline to the back end needs to be flow-driven so we can:
Perform a validation process by selecting and using validation tools to verify the configuration(s).
When validations fail, the process can be pushed back to the configuration stage again.
When users are in the loop, it becomes possible to drive a conversation for more clarity.
With a conversational process, it becomes possible to create in-pipeline transforms as a result of user conversation, along with simpler transforms to handle possible data formats and names.
What we’re describing here isn’t trivial, and could arguably be offered as a commercial product in its own right. The config management feature of our OpAMP solution also provides meaningful inputs, as it defines valid data types for observability agents like Fluent Bit where a concrete schema is not provided.
Using AI – agent graph
Agent descriptions
The agentic process illustrated in the preceding diagram is not trivial. It will be pretty context-heavy, as much of the information to work with can’t be easily addressed through targeted RAG or semantic search. In most cases, these core agents will need tools such as web search, since their training isn’t as up to date as the latest features of the different tools.
Input evaluator – We need a variety of inputs covering the target systems and, if there is overlap in capability, guidance on which ones to use. We need to tell the agent what tooling the pipeline needs. We’ll likely need metadata on the sources. For example, if we detect a database, we need to know which type so we can inject the correct queries. The input evaluator will use this to decide whether the detail is sufficient. The result of this analysis could be one of several things:
Use a client conversational agent that chats with the user to get clarifications; there is some overlap here, as the conversation will need to know what answers are needed, and some answers may open up new questions. Once this agent believes it has compiled the details for the answers. it returns to the Input Evaluator to reassess the picture.
We can ask the evaluator to pause the process and generate a report showing the gaps identified. This can be done using the Clarifications Report generator.
The Input evaluator may determine that it reports the gaps and presses on with a best effort.
Semantic normalizer – To simplify the flow, mapping source names to the semantic definitions from OpenTelemetry would need to understand the source and the OTel Semantic Naming model.
Pipeline generator – Once we confirm the details and have simplified semantics, we can have the AI build out the rest of the pipeline. The agent may need to be handed back to the agent for correction.
Config validator – once the pipeline generator hands off the configurations, the validator agent determines the best tools to verify whether the configurations are valid. If the tool returns an error, it hands the cause back to the pipeline generator. Ideally, tool calls will return meaningful error descriptions. But if not, we’d need to create a simple knowledge base to extract details from.
Clarifications Report generator – The clarifications report generator’s role is to take the gaps identified and generate and share them via a document repository or email in a user-friendly format
User clarification – The conversational interface with the user to obtain missing details, according to the Input evaluator. One difference between this and the Input evaluator is that this agent will need to consider conversational history to build the required additional detail. The input evaluator shouldn’t need prompting.
Controlling complexity
We’ve seen that allowing flexibility and removing the opinionated constraints of turnkey solutions increases complexity very quickly. The AI solution will become much more reliable if we strip back the options we allow agents to choose from, or if we push more organizations to standardize on consistent configuration strategies that enable a move toward templatization.
A Unified pipeline syntax
The other path that could radically simplify things is the Open Telemetry Transformation Language (OTTL), which has just reached the version 1 milestone. While it is not a complete solution, it offers a way to advance the standardized pipelines. We’ll explore OTTL in a separate blog in more depth. But essentially, if you look at the collector as two parts- the wire-level mechanism to capture or emit data and a second part for routing and manipulation- then OTTL is about the routing and manipulation. OTTL has both a formal syntax and an implementation as a Go library. While the library may not help if you’re using Vector, Fluentd, Fluent Bit, etc., the formal notation will work with ANTLR4; the resources are available through the OTLery OTTL-Lang repo. This allows us to generate, across many languages (C, Java, Ruby, Lua, etc.), a parser for the OTTL syntax into a tree structure that can then be walked. This still needs the intelligence to translate data references into references for the observability tool, and the functions, editors, and transformers to be implemented. Still, most of these can be mapped onto language-native libraries.
With a formal notation and a growing catalogue of prepared OTTL scripts and utilities, such as those offered by the OTLery repos, there is the ability to leverage AI to generate the transformation logic. As many observability tools such as Fluent Bit and Vector include support for scripting and plugins, incorporating a complete parser becomes a practical option. With both Vector and Fluent Bit, for example, supporting Lua being incorporated into the pipeline, and ANTLR4 supporting Lua as well, it should be possible to implement OTTL in those pipelines.
There is one downside to using OTTL: we lower our observability agents to the lowest common denominator. For example, Fluent Bit provides basic stream analytics. Transformations are now going through a parser phase rather than going directly to native binary processes.
Conclusion
The more we can restrict options, the closer we can get to being able to automate the creation of the observability pipeline quickly, and efficiently, but at the price of potentially not maximizing feature exploitation (the downside of using OTTL across different observability agents). The problem is to provide a solution that is very flexible, which will require a fair bit of effort. That flexibility also creates the possibility of exploiting the full feature set of the available tools.
Last week saw the Open Source Observability Day (OSOD) during which I presented on the subject of the Open Agent Management Protocol, covering the underlying protocol that forms the foundation of a wider capability we have been building along with a demo of what is available.
If you want to see the presentation, and hear the full story not only of the core protocol but how it can help with the ideas around ChatOps and AIOps head over to the OSOD website (and previous years can be seen on YouTube with playlists for each year e.g. OSOD 2025). Of course if you have been following my blog posts you’ll know this.
The demo is recorded against a version prior to the 0.6 release.
Running the demo yourself
If you want to try this out for yourself, you can do so. You’ll need to:
There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:
Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
Deployed services and applications with default monitoring configuration
Prebuilt dashboards
While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.
This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.
Detecting and Generating Configuration
The process we need has the following steps:
find the deployed solution(s)
Understand how they are deployed
Generate a configuration based on the detection
Push the configuration out to agents
Detect and capture config
One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.
While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …
As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.
Templates make the result reusable
By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.
There are limits
So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.
Convenience as a weakness?
While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.
Backend Bootstrapping
Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.
We’ve made the latest version of our implementation of the OpAMP specification available. It has taken longer to ready than the previous versions, but there is a lot been included, both in the core functionality, but also capabilities supporting the use and setup. Here are the headlines…
Automated regression from clean to deployed and running a series of end-to-end tests as well as producing a guaranteed fresh set of artefacts has been implemented. This can be run against AWS (currently in use) or Azure (needs refining). These are triggered on changes to the main branch.
GitHub pages with links all the documentation are established now.
The Python wheel release artefacts are no incorporated into the release assets.
The client/consumer side has been refactored so it is easy to build your own observability agent controls and independently deploy them (not supported by the config editor).
The CLI has been enhanced so it can help quickly spin up more advanced configurations – helpful for demos and real deployments.
The Dev CLI has been improved.
The server side now has a basic client side (Supervisor/Observer) configuration editor UI. Which can be deployed as a plugin.
Improvements around the use of Podman and Docker for the containerization.
What next?
We’ve added to the documentation pages for the roadmap, initially we’re working to fill out the regression test sets at the moment. Other possibilities:
Expand the client types
Extend the client types to exploit more features.
Perhaps the most interesting idea, we’re thinking about is can we exploit tools like Netdata to discover what solutions are deployed and generate a requirements/configuration definition which can be used to generate a suitable agent configuration file.
Not only will I be going through the what the spec enables, and the value it aims to support we’ll be touching on how OpAMP can be used to support ChatOps and AIOps. While the idea of unleashing AI Agents onto a production may look pretty scared given all the recent news coming from OpenAI, Anthropic and others on how their latest models are escaping and hacking other web sites. I’ll show how among other things how mixing AIOps nd OpAMP can offer the AI smarts while training deterministic controls.
The session isn’t just slideware, we’ll demo our Open Source OpAMP implementation which gives the foundations for using AIOps. If you want to try the demo yourself we’re in the process of making available in GitHub.
A FIPS safe mode to ensure things are safe during start and restart.
Ensuring timestamps will work beyond 2038.
TLS handling refinements such as certificate reloading
Management of the internal buffers can be made more dynamic now.
Improvements across a number of plugins for input and output.
Dynamic flushing to allow dynamic optimization within configured boundaries.
Packaging improvements for Windows Nano and Debian deployments.
New root section called Extensions.
A lot of these gains are great, and will really pay off for high volume deployments, where small savings, cumulative can really payoff economically. It is the new extensions feature that I’m most interested in as it open up some capabilities when managing the observability domain.
Extensions
The extension structure (first identified as a useful feature with issue 11863) doesn’t impact Fluent Bit behavior but is retained and accepted as YAML configuration values. This means that any tools being used to manage Fluent Bit configuration can now add YAML key value pairs (including nested values) without causing Fluent Bit a problem, and can therefore be used by management tools. For example we can track what version of a configuration is being handled by Fluent Bit by injecting a Git version identifier (just as you see in the Fluent Bit output at startup). This simple change means that it is now possible to understand which version of configuration is being executed. When you’re rolling out to a large estate (particularly outside of containerized environments) it is easy to know what has been rolled out.
extensions:
opamp:
endpoint: 127.0.0.1:4318
insecure: true
deployment:
config_version: 12345
If you’re using Fleet Management capabilities such as OpAMP (a specification developed as part of the Open Telemetry project) you can embed the client side controller/supervisor configuration within the Fluent Bit configuration (as long as the controller supervisor understand where to get the values from within a YAML file).
It also means that you’re allowing more dynamic changes, it becomes easy to notate such dynamic changes., for example:
extensions:
deployment:
changeDTG: 2026.08.05.16.25
notes:
- changed flush interval
- moved plugin x output
Personally I’d not advocate changes without using configuration control tooling, but it certainly is better than no record.
Next steps
If you follow this blog regularly, they’ll be aware of the OpAMP project which implements the protocol, and provides a server that includes fleet management and configuration editing and validation for Fluent Bit server side, and a client supervisor/observer on client client/consumer side which can manage Fluent Bit instances. These changes mean that the project will need:
New configuration and validation settings (the UI is entirely meta-data driven) for the new properties.
The configuration catalogue viewer will need to be updated to exploit the extensions feature for showing versioning.
The client/consumer side can be extended to become Fluent Bit aware and exploit the embeddable extensions capability.
We may also offer the dynamic change recording possibility as an optional feature (from a product view – we shouldn’t force specific ways of working, but from a personal view it isn’t a way I’d recommend working)
The Elastic toolset for monitoring and observability with the Beats tools (heartbeat, filebeat etc) and more recently an integrated OTel compliant agent have been attractive because of the strength of the analysis capabilities of Elasticsearch, Logstash and Kibana. All the products are available as open source and can be extended (although with 3rd party constraints as a result of licensing) and with enterprise extensions.
Kibana while focused on data visualisation, also provides the fleet management layer to control the remote agents. Kibana’s communication with the agents now makes use of protobuf. With clear commenting discouraging the use.
This is great if your entire ecosystem is aligned to the Elastic stack, but that is rarely the case. If we’re not already in the era of polyglot, then pervasive use of AI to power development, both at departmental level (shadow/gray IT) and even citizen development.
Managing distributed instances of the Elastic Agent outside of Kibana has to be done using the agent’s command line interface. The only publicly documented web interface allows the retrieval of status information. This does feel rather poor as a means to drive the adoption of Kibana (and encourage the use of Elastic Cloud).
This leads us to the question of whether to use the Supervisor or Observer model of using the OpAMP standard, or should we try to embed the OpAMP client directly into the Elastic Agent. Having studied the documentation and some of the code base it feels like these sort of customisations are not encouraged, and emphasis for extensions are about adding the means to monitor different protocols or products. Incorporating socket or HTTP handling that also needs to interact with lifecycle logic would be very invasive.
Further more, invasive changes may prove to be more problematic if you switch from a forked open-source version to an enterprise licensed version, where you’d want both the benefits of OpAMP and the licensed extras.
The way we have designed and implemented our client so that it is easy to implement specific logic for different observability tool, and the development of the elastic agent logic, meant we took the final step of adopting a fully pluggable mechanism – we’ll come back to those details shortly..
Our implementation of the Elastic Agent management supports both models of supervisor or observer, but we would err towards using it in an observer mode. The control aspects of the agent are mapped onto using creating commands and using the CLI, as with Fluent Bit basic health can be retrieved via the agents REST endpoint.
The command line does allow for more diagnostic information. But we’ll look at that later.
What’s new as a result of the modifications
The improvements are being incorporated into a branch in the repo while we do regression testing, once we’re happy we’ll merge into main, and label.
Starting with the simple things:
We have a simple validation setup that can be run to ensure you have the elastic agent and a containerised Logstash that can talk to each other. The agent is configured to monitor itself to generate traffic to Logstash.
That configuration has been mapped into the example setup of the agent being managed by the OpAMP consumer.
We’ve enhanced the OpAMP-CLI tool so it can launch the containers using Docker or Podman. This makes it quick and easy to configure and launch demos and end-to-end regression tests. There are also some convenience tweaks to the way the CLI works.
New documentation explaining how to implement and deploy the management of your own Observability product.
We’ve refactored part of the consumer code to explicitly separate the specific code for the agent being managed be that Fluent Bit, Fluentd, Elastic Agent etc. the test code reflects the same structure.
The linkage to the Observability tools is dynamic now. Rather than have a central package which imports all the plugins.
There are a few things we haven’t done,..
We’ve only provided support for operations such as start and stop, so the server side of the product only works with the core server features. There is no configuration editor.
The MCP broker hasn’t been enhanced to understand differences between the client types. This is also true with the Slack interfacing. We will return these areas, but the most valuable thing that can be done is to make the OpAMP solution cover a variety of Observability agents.
We still need to improve the automated regression testing.
Extend to cover the individual beats components.
All these improvements will start arriving in the GitHub repo in the next couple of days. Once we think everything is good, we’ll up issue the release number – sdo watch the github repo 🙂
Over the years we’ve created a number of GitHub repos with tools and associated content that can help us in the Observability space, such as log generation and replay, to help validation Fluentd and Fluent Bit configurations (or an OTel collector of choice). This extends to the most recent efforts with imp,eating the OpAMP capabilities and configuration UI.
Tools are good, but documentation that is easy to work with is a key to helping users be successful. To support that we have established a consolidated set of pages using GitBook which can be found at:
The nice thing about GitBook, is that it brings together the various resources such as markdown , HTML etc and creates a friendlier experience. It also keeps in sync with the GitHub repos.
New logos
As we’ve established the GitBook content, and using AI makes us less artistically able people to create graphics and logos, we’ve created a couple of logos for the older repos as well:
Fluent Bit Classic configuration to YAML conversion tool.
Log Generator can create synthic logs, or replay application logs to simulator a real system logging.
You must be logged in to post a comment.