• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: OpenTelemetry

Application Observability: The Database Angle

13 Friday Nov 2026

Posted by mp3monster in development, Fluent Observability, General, Technology

≈ Leave a comment

Tags

database, DB, MariaDB, MySQL, Open Telemetry, OpenTelemetry, Oracle, OTel, OTLP, OTTL, Postgres, queries, sql

When it comes to application observability, it is easy to lose sight of the challenges of tracking how our data flows in and out of the database. Considerations include query performance as the database sees it, when SQL gets held up or fails because of database locks, and database calls that get rejected for various reasons such as privileges (particularly when joining data sets). How impactful these things can be is dependent upon how the application is built, and where the weight of data manipulation resides. For example, a multi-tiered application that uses object-relational mapping frameworks such as Hibernate (Java), Gorm (Go), SQLAlchemy (Python), or tracing methods (even if you don’t instrument the libraries) lets you infer and extrapolate database performance details. However, retrieving the explicit information is always better. But for other solutions, such as ERPs like E-Business Suite and parts of General Dynamics where the business logic is within the database as stored procedures, the ability to extract more detail from the database is going to be needed.

This means our observability agent needs to formulate, execute, and transform the results of SQL queries. This leads us to:

  • Ideally, plugins that understand how to access the relevant database data and retrieve it in a consumable manner will make the process easier. In other words, having pre-canned logic or configuration that understands how to retrieve core metrics like slow queries is desirable.
  • Having the ability to run SQL is also freeing, as it allows for application-specific queries to also be incorporated and retrieve metrics and details from where stored procedures have recorded logs. For example, one of Oracle’s IdM products records its logs in a table (see here).

If we can query the database, caching or pooling connections is highly beneficial, since establishing a connection is costly and we don’t want to incur that cost every time we need to execute the queries. We also don’t want to leak connections or rely on the database’s connection manager to decide that connections have been inactive and to reclaim them.

This leads us to ideally having three capabilities:

  1. Pre-canned queries for mainstream database technologies
  2. Ability to add custom queries
  3. Ability to pool connections

Capability matrix

The following shows which observability tools can work with which databases, the nature of the query capabilities and the ability to cache. For full transparency, I am not an expert in every one of these products, so I used AI to help with the analysis. That said, I have checked what the AI has pointed to.

The language used is explicitly defined below the table if you need to confirm it. The one variation is Fluent Bit, which doesn’t provide an out-of-the-box query-related plugin, but we can use Lua for a script that can work with any database.

Database/ Observability agentOpenTelemetry CollectorTelegrafLogstashMetricbeatGrafana AlloyVectorFluentdFluent
Bit
OracleConfig
Pool ✓
Config
Pool ✓
Config Pool ✓Config
Pool?
Both
Pool ✓
——via Lua
Pool ✓
PostgreSQLBoth
Pool ✓
Both
Pool ✓
Config Pool ✓Both
Pool?
Both
Pool?
Predefined Pool?Limited
Pool?
via Lua
Pool ✓
MySQLBoth
Pool ✓
Both
Pool ✓
Config Pool ✓Both
Pool?
Predefined Pool?—Limited
Pool?
via Lua
Pool ✓
MariaDBConfig
Pool ✓
Both
Pool ✓
Config Pool ✓MySQL-compatible Pool?MySQL-compatible Pool?—Limited
Pool?
via Lua
Pool ✓
SQL ServerBoth
Pool ✓
Both
Pool ✓
Config Pool ✓Both
Pool?
Both
Pool?
——via Lua
Pool ✓
ClickHouseConfig
Pool ✓
Config
Pool ✓
Config, JDBC driver
Pool ✓
————via Lua
Pool ✓
SAP HANAConfig
Pool ✓
Config
Pool ✓
Config, JDBC driver
Pool ✓
————via Lua ODBC
Pool ✓
MongoDBPredefined
Pool?
Predefined Pool?Limited, JDBC adapter Pool ✓*Predefined Pool?Predefined Pool?Predefined Pool?—via Lua
Pool ✓
OtherSnowflake, SAP ASE Config, Pool ✓SQLite, CockroachDB, TiDB, Vertica, Netezza, FlightSQL Config,
Pool ✓
Other JDBC databases Config, Pool ✓CockroachDB
Config,
Pool?
Redis, Snowflake Predefined, Pool?Other specialised sources Pool?Other ActiveRecord adapters: Limited
Pool?
External commands Pool: n/a

Legend

  • Both — predefined monitoring queries and configurable SQL queries are available, potentially through separate plugins.
  • Config — supports user-defined SQL queries.
  • Predefined — built-in database metrics or fixed queries, without needing general-purpose SQL configuration.
  • Limited — partial functionality, such as incremental table polling.
  • — — no native database-querying capability established.
  • Pool ✓ — connection reuse/pooling is explicitly documented.
  • Pool? — connection reuse is not sufficiently established by documentation.
  • Pool n/a — no native database query connection.

Important qualifications

  • OpenTelemetry: A Both designation can represent the combination of the generic sqlquery receiver and a separate database-specific receiver. Connection pooling is documented, but controls differ between receivers. The generic SQL receiver exposes max_open_conn; certain specialized receivers offer more extensive pooling controls.
  • Telegraf: Its generic SQL input supports configurable pooling using maximum open and idle connections, maximum idle time and connection lifetime. Its database-specific plugins provide additional predefined monitoring capabilities.
  • Grafana Alloy: Oracle is a particularly strong option because its exporter explicitly documents max_idle_conns, max_open_conns, predefined metrics and custom metric definitions. Equivalent pooling guarantees have not been established for all other embedded exporters.
  • Logstash: JDBC connection pooling is supported, but you generally need to install the database-specific JDBC driver separately. Its SQL input produces events rather than directly modelling each result as an OpenTelemetry metric.
  • Metricbeat: Both means its database-specific metric sets can be combined with its generic SQL module. This does not mean that the generic SQL module itself provides predefined slow-query statements.
  • Vector, Fluentd and Fluent Bit: Their narrower entries are significant. Vector has native database-specific metrics inputs, Fluentd’s SQL plugin focuses on incremental record ingestion, and Fluent Bit requires database connectivity through a Lua script.

Additional resources

What is clear is that the number of observability agents that provide built-in understanding of databases is limited, and that you need to provide configuration for the queries – which is understandable; if you’re able to at least enable user-defined queries, the user can research how to retrieve data like slow queries. To that end, we’ve started compiling some documents that identify queries that will help for that database. You can find the documentation for the tooling we’re putting together to simplify creating an end-to-end pipeline in GitHub (https://github.com/opamp-observe/config-generator/tree/main/docs/db-query-analysis).

Alternate Strategies

Obviously, we’ve taken an observability agent lead approach, as this means that the data is immediately being routed onwards to where it needs to be. But this polling, while some metrics need to be counts per interval, not all data needs to be that way, and as any integration person will tell you, being event-driven or pushing (as long as we’re not creating unnecessary additional latency in a process) is desirable. This points to: do the databases have built-in push mechanisms?

Built-in in OLTP emitters

In a perfect world, databases would make all their information available through OTLP, including allowing applications to provide additional metadata for application-level logs, metrics and traces. Today, there is varying support for built-in OTLP payload emission, as the following table shows:

DatabaseWhat can be sent to an OTLP endpoint? What can be tailored?
OracleOracle AI Database 26ai can export database observability logs associated with traces. For metrics, Oracle’s separate Oracle AI Database Metrics Exporter publishes metrics over OTLP/gRPC. It can also export the alert log as JSON for a log aggregator. docs.oracle.comThe exporter lets you define custom metrics in YAML or TOML and set collection intervals and query timeouts. Native observability has configurable endpoints and service options. oracle.github.io
PostgreSQLPostgreSQL has no built-in OTLP metrics exporter in the sources reviewed. The open-source pg_otel module can send server log messages to an OTLP endpoint. For metrics, the OpenTelemetry Collector can query PostgreSQL’s statistics. github.compg_otel lets you set the endpoint and resource attributes. The Collector receiver lets you select metrics; its SQL Query Receiver can turn custom query results into metrics or logs. github.com
MySQLMySQL’s telemetry component supports OTLP metrics export. Its supplied log exporter sends OTLP logs, but requires MySQL Enterprise Edition or HeatWave; the logging interface alone does not provide an exporter. dev.mysql.com
Other vendor-supported variants of MySQL also support OTLP, such as Percona Server for MySQL 9.7
Configure the signal endpoints and protocols, then enable or disable meters and loggers. Logging support depends on edition. dev.mysql.com
MariaDBNo built-in server OTLP exporter was identified. The OpenTelemetry Collector’s MySQL receiver can collect metrics from MySQL and MariaDB servers. You can collect them separately or map query results to log records. github.comThe Collector can select metrics or use custom SQL queries and map their results to metrics or logs. github.com
SQL ServerNo built-in server OTLP exporter was identified. The OpenTelemetry Collector’s SQL Server receiver connects to the database to collect metrics; log files or selected query results can be collected separately. github.comConfigure which receiver metrics are emitted, or use custom SQL queries and signal mappings in the Collector. github.com
ClickHouseNo built-in server exporter for OTLP metrics or logs was identified. A Collector can query ClickHouse, read its log files, and forward the results to an OTLP endpoint. github.comThe Collector’s SQL Query Receiver supports custom queries, collection intervals, and mappings from query results to metrics or logs. github.com
MongoDBMongoDB Ops Manager’s Agent can push monitored metrics to an OTLP/HTTP endpoint. MongoDB Atlas can export selected system and audit logs to OTLP endpoints. These exports are provided by MongoDB’s management services, rather than by the database process itself. www.mongodb.comOps Manager lets you configure the endpoint and export cadence. Atlas lets you choose log types and configure up to 10 destinations. For self-managed MongoDB, the Collector can query metrics and collect query-related events. www.mongodb.com

While some databases don’t yet provide support for this, it is likely to become the norm. For example, key members of the PostgreSQL ecosystem, Craig Ringer and Gabriele Bartolini, are actively advocating the native adoption of OTel (see here). Other database solutions we’ve found that include OTLP support include Databricks Lakebase (a managed DB built on Postgres) and SurrealDB.

File Staging

Where there aren’t built-in OTLP capabilities, another approach is to incorporate stored procedures that can write to an externally visible temporary file location. Then the observability tool can use file tailing plugins to ingest the data into the observability pipeline. This means we should take a look at what databases offer in this direction:

DatabaseStored procedure ability to write to an external filesystem? Can it run SQL on a schedule?
OracleYes. PL/SQL can use UTL_FILE to write files in permitted directories. For a remote host, DBMS_SCHEDULER.PUT_FILE can transfer files using a Scheduler agent.Yes. DBMS_SCHEDULER schedules PL/SQL jobs, among other job types. oracle.hydrogen.sagittarius.connect.product.adaptavist.com
PostgreSQLYes, with elevated access. A procedure written in an untrusted language such as PL/Python can access operating-system resources. SQL COPY can also write a server-side file, but requires superuser rights or the pg_write_server_files role.Yes, with an extension or external scheduler. PostgreSQL has no built-in general job scheduler; pg_cron is a widely used extension that runs scheduled SQL inside the database. Python Procedural Language
MySQLYes. A stored program can use SELECT ... INTO OUTFILE to write query results on the server host. The FILE privilege is required, and secure_file_priv can restrict the permitted directory.Yes. The built-in Event Scheduler runs scheduled SQL events; an event can also call a stored procedure. dev.mysql.com
MariaDBYes. SELECT ... INTO OUTFILE writes query results to a server-side file. As with MySQL, the database service account must be able to write to the destination.Yes. MariaDB’s built-in Event Scheduler runs scheduled SQL, including calls to stored procedures. mariadb.com
SQL ServerYes, through privileged features. A stored procedure can invoke xp_cmdshell to run an operating-system command, or use CLR code with external access. xp_cmdshell is disabled by default.Yes. SQL Server Agent schedules jobs with T-SQL steps, including calls to stored procedures. The Agent service is disabled by default on a new installation. Microsoft Learn
ClickHouseNo stored-procedure mechanism in the usual sense. ClickHouse does support executable user-defined functions: a query can invoke an external script, which could write files if the script and server account have permission. That is external code execution, not a stored procedure.For a specific use case, yes. A refreshable materialized view reruns its query on a schedule and stores the result. It is not a general scheduler for arbitrary SQL jobs. ClickHouse Documentation
MongoDBNo general database-side stored procedure with access to the database host’s filesystem. MongoDB server-side JavaScript features are not equivalent to operating-system procedures and are deprecated starting in MongoDB 8.0. Atlas Functions run in a separate managed runtime, not in the database process.Not SQL: MongoDB Atlas Scheduled Triggers can run server-side functions on a schedule. For self-managed MongoDB, you generally handle scheduling externally.

Conclusion – Understanding what to retrieve

So, there are several different strategies that can be pursued to monitor a database: several different strategies can

  • Observability plugin
    • DB-variety aware.
    • Generic connector options, but configuration must provide context for the DB type.
  • Databases can push or be scraped for telemetry.
  • The database can externalize data that can then be picked up by the observability tool using mechanisms such as file tailing.

What is clear is that most options require a good understanding of what to retrieve from the database. As a result, to support our work on automating pipeline creation, we have captured documents that bring together recommended practices for what data to retrieve and how to do so, and compiled the information as documents in the repository.

Useful resources

  • Telegraf – SQL Input Plugin · Supported SQL drivers · PostgreSQL Extensible
  • Grafana Alloy – prometheus.exporter.mysql · prometheus.exporter.oracledb · prometheus.exporter.mssql prometheus.exporter.mongodb · prometheus.exporter.snowflake · prometheus.exporter.postgres ·
  • OpenTelemetry Contrib component registry
  • Open Telemetry – postgresql · sqlquery · oracledb · mongodb · mongodbatlas · sqlserver
  • Metricbeat SQL module
  • Logstash JDBC input documentation
  • Grafana Alloy PostgreSQL exporter · PostgreSQL Prometheus exporter
  • Fluentd SQL plugin
  • Vector PostgreSQL metrics source
  • docs.influxdata.com
  • DB Queries for monitoring
Database Observability - SQL and Telemetry

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

LogGenerator becomes SignalGenerator

11 Wednesday Nov 2026

Posted by mp3monster in development, Fluent Observability, General, logsimulator, OpAMP, Technology

≈ Leave a comment

Tags

Fluent Bit, FluentBit, Fluentd, log simulator, loggenerator, logs, metrics, observability, Open Telemetry, OpenTelemetry, OTel, python, signal generator, signals, Technology, testing, tools, Traces, Tracing, unified logging

When I wrote Logging In Action, I found that to help with setting up and testing logging pipelines, having a utility that could just take existing logs and play them so it looked like a real application was generating the logs was helpful – not only for the book, but in everyday work.

The natural evolution was: can I also create and run synthetic logs through observability systems so we can ensure log thresholds, filters, alerts, etc. worked as expected. Not to mention taking synthetic or real logs and applying some basic reformatting so we could mimic Apache or Nginx web servers, for example.

So the LogGenerator (sometimes called Log Simulator) was born and made available. At the time, my development skills were centred on Java (and I loved Groovy for its convenience and development speed). So we built the solution using Groovy and Java -the net result is that you could create a deployable JAR or run it without the need to first establish a Maven pipeline (without sacrificing the ability to keep polymorphism and abstract layers).

The LogGenerator has had a number of enhancements, so we could use it to check not only tools like Fluentd and Fluent Bit, but also cloud observability tooling such as OCI. This was made to be a pluggable framework so you could add custom targets easily.

Time moves on …

Python continues to grow in popularity and is the default language bundled with Linux. I’m comfortable developing in Python now, and we have the tools I enjoyed with Groovy and Java (no need to mess with build pipelines for quick changes, but it can also be packaged and easily deployed for mainstream adoption).

But it isn’t programming language evolution that has triggered change. OpenTelemetry standards and the golden signals of Logs, Traces and Metrics are the norm. So applying the same principles to the Traces and metrics that the LogGenerator used for logs is a natural evolution.

Deployment

It was pointed out to me that, even if people don’t take the full OpAMP solution we built, the simulator component may be attractive in its own right. This got us to thinking about whether there was value in pairing the SignalGenerator with the OpAMP simulator.

As a result, we created an optional feature that lets us deploy SignalGenerator as an extension of the OpAMP simulator. This gives us multiple deployment options.

  • A free-standing solution in its own right, with no need for anything else. Ideal for dropping into test environments to exercise your Observability configuration and tuning without needing start-up applications that require large deployments with complex inputs to generate unhappy path signals.
  • Deploy in a supervised manner using OpAMP – ideal if you want to drive synthetic Observability exercises in a distributed manner.
  • Plug into the OpAMP simulator – fewer processes running, easier to create custom commands through the OpAMP framework to get the generator to do things from pushing out new data sources to be replayed, to having signal generators simulate Observability data that reflects a distributed cascading failure as means to evaluate smarter AI-driven processes that should be looking for correlations and cause and effect.

The last part with custom messaging over OpAMP is not yet in place, but it’s a candidate for enhancement.

New home

As the Logging in Action book uses the LogGenerator, we didn’t want to make this level of change in the current repository. So we have a new repository within the OpAMP-Observe GitHub organization which has feature parity, but also has a mechanism for simulating metrics (pushing, or allowing the data to be scraped).

The documentation to link the repositories and the GitHub documentation together is a little bit behind, and we need to figure out how to create a manageable playback structure for Traces before we enable that feature.

Resources

  • Logging In Action
  • LogGenerator aka LogSimulator
  • Maven
  • SignalGenerator
  • OpAMP-Observe (repository)
  • OpAMP.info
Signal Generator Logo

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Moving Open Source Observability backend closer to turn-key solutions

09 Monday Nov 2026

Posted by mp3monster in AI, Cloud Native, Fluent Observability, General, Technology

≈ Leave a comment

Tags

agentic, agents, AI, collector, Fluent Bit, Fluentd, Lua, observability, open-source, OpenTelemetry, OTel, OTTL, Transformation, Vector

In our first post on Moving Open-Source Observability closer to turn-key solutions, we discussed how we can automate the source-to-pipeline. But turn-key solutions also flow data to the backend and provide logs and metrics that are contextually aware of the systems being monitored.

While detecting sources and building the capture mechanism may seem more challenging at first glance, it is easier – we have a defined technology, and our observability agent defines the output structure. So what makes the pipeline to the backend more complex?

  • We’re more likely to have a variety of products, and some products may have functional overlaps.
  • Greater configuration diversity in how tools can be configured.
  • Greater contextual awareness is needed.

Let’s break these down in more detail.

Greater variety of tools means we need a common way to set up transformations; otherwise, the mechanics of a transformation process in our pipeline will explode in complexity.

One benefit of going open source is that you can choose the tools that fit your needs for log analytics, metrics, alert routing, and so on, but this can lead to functional overlaps. For example, you may use Prometheus for infrastructure-based metrics, but want to use Elastic APM for Application Performance Management (APM) as you’re using an Elastic Java stack for app development.

The different approaches to configuring tools, or configuration diversity, is probably the single largest challenge, and the number of approaches multiplies the number of configuration permutations that must be achievable. While there appears to be some initia behind the support the Prometheus mixin framework which provides templates for different scenarios such as infrastructure dashboards and alerts. It is far from universal adoption today. For example, Prometheus and Grafana support its use. But other solution stacks such as Elastic do not. If you align on a common technology stack, the chances of a harmonised configuration process rise.

We need contextual awareness from the source, as well as the collected data, to map data to the table(s). For efficiency, if we can avoid putting all metrics into a single table, queries could have one less dimension to filter. Context awareness will inform how we render the dashboards and set alert thresholds. For example, general filesystem storage capacity thresholds would be hugely different to database storage thresholds.

All of this means that if we want a deterministic, transformation-based process, we’ll end up with an unsustainable set of combinations that need to be mappable, testable, and maintainable as the different solutions grow, evolve, and occasionally get archived as contributions dry up.

As turnkey solutions, bring an opinionated answer and either build their own backends from scratch (e.g. Oracle’s Enterprise Manager) and leveraging open source as building blocks – they effectively eliminate the challenges of functional overlaps and configuration diversity – at the same time removing that flexibility. To underline how messy things could get, we experimented in a branch of the repo to see how messy it could get, while trying to keep the divergence as narrow as possible. This diagram illustrates the point visually.

The complexity and variability of pipeline tools and observability back ends

Solution

If a deterministic approach is unsustainable, are there options to ease the challenge or achieve a solution?

Simplification option

One option that can simplify, but certainly doesn’t solve, the challenge is to leverage OpenTelemetry’s work on semantic conventions, where they have been cultivating naming conventions that define data elements likely to be transmitted. When you manually map data, the process becomes more intuitive (and therefore quicker) and improves the chances of an AI agent mapping successfully. In some respects, this is like middleware-driven efforts to build data ontologies, so you need to avoid the problems global ontologies had.

Bringing AI to the game

Because configuration is linguistic and well documented, you can apply an LLM-based approach to the problem. The process of mapping from the ingestion part of the pipeline to the back end needs to be flow-driven so we can:

  • Perform a validation process by selecting and using validation tools to verify the configuration(s).
  • When validations fail, the process can be pushed back to the configuration stage again.
  • When users are in the loop, it becomes possible to drive a conversation for more clarity.
  • With a conversational process, it becomes possible to create in-pipeline transforms as a result of user conversation, along with simpler transforms to handle possible data formats and names.

What we’re describing here isn’t trivial, and could arguably be offered as a commercial product in its own right. The config management feature of our OpAMP solution also provides meaningful inputs, as it defines valid data types for observability agents like Fluent Bit where a concrete schema is not provided.

Using AI – agent graph

Agent descriptions

The agentic process illustrated in the preceding diagram is not trivial. It will be pretty context-heavy, as much of the information to work with can’t be easily addressed through targeted RAG or semantic search. In most cases, these core agents will need tools such as web search, since their training isn’t as up to date as the latest features of the different tools.

  • Input evaluator – We need a variety of inputs covering the target systems and, if there is overlap in capability, guidance on which ones to use. We need to tell the agent what tooling the pipeline needs. We’ll likely need metadata on the sources. For example, if we detect a database, we need to know which type so we can inject the correct queries. The input evaluator will use this to decide whether the detail is sufficient. The result of this analysis could be one of several things:
    • Use a client conversational agent that chats with the user to get clarifications; there is some overlap here, as the conversation will need to know what answers are needed, and some answers may open up new questions. Once this agent believes it has compiled the details for the answers. it returns to the Input Evaluator to reassess the picture.
    • We can ask the evaluator to pause the process and generate a report showing the gaps identified. This can be done using the Clarifications Report generator.
    • The Input evaluator may determine that it reports the gaps and presses on with a best effort.
  • Semantic normalizer – To simplify the flow, mapping source names to the semantic definitions from OpenTelemetry would need to understand the source and the OTel Semantic Naming model.
  • Pipeline generator – Once we confirm the details and have simplified semantics, we can have the AI build out the rest of the pipeline. The agent may need to be handed back to the agent for correction.
  • Config validator – once the pipeline generator hands off the configurations, the validator agent determines the best tools to verify whether the configurations are valid. If the tool returns an error, it hands the cause back to the pipeline generator. Ideally, tool calls will return meaningful error descriptions. But if not, we’d need to create a simple knowledge base to extract details from.
  • Clarifications Report generator – The clarifications report generator’s role is to take the gaps identified and generate and share them via a document repository or email in a user-friendly format
  • User clarification – The conversational interface with the user to obtain missing details, according to the Input evaluator. One difference between this and the Input evaluator is that this agent will need to consider conversational history to build the required additional detail. The input evaluator shouldn’t need prompting.

Controlling complexity

We’ve seen that allowing flexibility and removing the opinionated constraints of turnkey solutions increases complexity very quickly. The AI solution will become much more reliable if we strip back the options we allow agents to choose from, or if we push more organizations to standardize on consistent configuration strategies that enable a move toward templatization.

A Unified pipeline syntax

The other path that could radically simplify things is the Open Telemetry Transformation Language (OTTL), which has just reached the version 1 milestone. While it is not a complete solution, it offers a way to advance the standardized pipelines. We’ll explore OTTL in a separate blog in more depth. But essentially, if you look at the collector as two parts- the wire-level mechanism to capture or emit data and a second part for routing and manipulation- then OTTL is about the routing and manipulation. OTTL has both a formal syntax and an implementation as a Go library. While the library may not help if you’re using Vector, Fluentd, Fluent Bit, etc., the formal notation will work with ANTLR4; the resources are available through the OTLery OTTL-Lang repo. This allows us to generate, across many languages (C, Java, Ruby, Lua, etc.), a parser for the OTTL syntax into a tree structure that can then be walked. This still needs the intelligence to translate data references into references for the observability tool, and the functions, editors, and transformers to be implemented. Still, most of these can be mapped onto language-native libraries.

With a formal notation and a growing catalogue of prepared OTTL scripts and utilities, such as those offered by the OTLery repos, there is the ability to leverage AI to generate the transformation logic. As many observability tools such as Fluent Bit and Vector include support for scripting and plugins, incorporating a complete parser becomes a practical option. With both Vector and Fluent Bit, for example, supporting Lua being incorporated into the pipeline, and ANTLR4 supporting Lua as well, it should be possible to implement OTTL in those pipelines.

There is one downside to using OTTL: we lower our observability agents to the lowest common denominator. For example, Fluent Bit provides basic stream analytics. Transformations are now going through a parser phase rather than going directly to native binary processes.

Conclusion

The more we can restrict options, the closer we can get to being able to automate the creation of the observability pipeline quickly, and efficiently, but at the price of potentially not maximizing feature exploitation (the downside of using OTTL across different observability agents). The problem is to provide a solution that is very flexible, which will require a fair bit of effort. That flexibility also creates the possibility of exploiting the full feature set of the available tools.

Useful links

  • Prometheus Monitoring Mixins
  • OpenTelemetry Semantic Conventions
  • Elastic APM for Application Performance Management (APM)
  • Open Telemetry Transformation Language (OTTL)
  • OTLery OTTL-Lang
  • OTTL Go library
  • ANTLR4
  • Configuration Generator main repository

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Open Source Observability Day Speaker

29 Tuesday Sep 2026

Posted by mp3monster in chatbots, Fluent Observability, General, OpAMP, Technology

≈ 1 Comment

Tags

AIOps, chatops, CNCF, conference, demo, observability, OpAMP, OpenTelemetry, Presentation

I’m excited to share that I will be speaking as part of the Open Source Observability Day virtual conference. I’ll be talking about the Open Telemetry specification OPen Agent Management Protocol (or OpAMP). To register for the event go here..

Not only will I be going through the what the spec enables, and the value it aims to support we’ll be touching on how OpAMP can be used to support ChatOps and AIOps. While the idea of unleashing AI Agents onto a production may look pretty scared given all the recent news coming from OpenAI, Anthropic and others on how their latest models are escaping and hacking other web sites. I’ll show how among other things how mixing AIOps nd OpAMP can offer the AI smarts while training deterministic controls.

The session isn’t just slideware, we’ll demo our Open Source OpAMP implementation which gives the foundations for using AIOps. If you want to try the demo yourself we’re in the process of making available in GitHub.

The way we can use OpAMP with AIOps to get the effectiveness of knowledge, and deterministic behaviour for operational change

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

New DZone article about OpAMP

27 Monday Jul 2026

Posted by mp3monster in Fluent Observability, General, OpAMP

≈ Leave a comment

Tags

blog, development, DZone, FluentBit, observability, OpAMP, OpenTelemetry, OTel, Technology

Just a quick post, we’ve had an article about OpAMP published on DZone check it out at OpenTelemetry’s OpAMP Potential Far Beyond Supporting Collectors

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Quick OpAMP Update

17 Friday Jul 2026

Posted by mp3monster in Fluent Observability, OpAMP, Technology

≈ Leave a comment

Tags

blog, CNCF, OpAMP, OpenTelemetry

Just a quick post: CNCF Ambassador Dotan Horvits posted a blog on the CNCF website discussing the use of OpAMP and referencing our OpAMP project. Check it out at Operating OpenTelemetry at scale with OpAMP. Ignoring the work we’ve been doing, it’s an interesting read.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OpAMP server with MCP – aka conversational Fluent Bit control

14 Tuesday Apr 2026

Posted by mp3monster in AI, chatbots, development, Fluent Observability, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, chatops, Fluent Bit, Fluentd, LangGraph, LLM, MCP, OpAMP, OpenTelemetry, OTel, OTLP

I’ve written a few times about how OpAMP (Open Agent Management Protocol) may emerge from the OpenTelemetry CNCF project, but like OTLP (OpenTelemetry Protocol), it applies to just about any observability agent, not just the OTel Collector. As a side project, giving a real-world use case work on my Python skills, as well as an excuse to work with FastMCP (and LangGraph shortly). But also to bring the evolved idea of ChatOps (see here and here).

One of the goals of ChatOps was to free us from having to actively log into specific tools to mine for information once metrics, traces, and logs reach the aggregating back ends, but being able to. If we leverage a decent LLM with Model Context Protocol tools through an app such as Claude Desktop or ChatGPT (or their mobile variants). Ideally, we have a means to free ourselves to use social collaboration tools, rather than being tied to a specific LLM toolkit.

With a UI and the ability to communicate with Fluentd and Fluent Bit without imposing changes on the agent code base (we use a supervisor model), issue commands, track what is going on, and have the option of authentication. (more improvements in this space to come).

New ChatOps – Phase 1

With the first level of the new ChatOps dynamism being through LLM desktop tooling and MCP, the following are screenshots showing how we’ve exposed part of our OpAMP server via APIs. As you can see in the screenshot within our OpAMP server, we have the concept of commands. What we have done is take some of the commands described in the OpAMP spec, call them standard commands, and then define a construct for Custom Commands (which can be dynamically added to the server and client).

interaction through Claude Desktop which has been configured with our MCP server (part of our OpAMP server) showing us what can be done

The following screenshot illustrates using plain text rather than trying to come up with structured English to get the OpAMP server to shut down a Fluentd node (in this case, as we only had 1 Fluentd node, it worked out which node to stop).

Claude Desktop showing conversation to shutdown a FLuentd node

Interesting considerations

What will be interesting to see is the LLM token consumption changes as the portfolio of managed agents changes, given that, to achieve the shutdown, the LLM will have had to obtain all the Fluent Bit & Fluentd instances being managed. If we provide an endpoint to find an agent instance, would the LLM reason to use that rather than trawl all the information?

Next phase

ChatGPT, Claude Desktop, and others already incorporate some level of collaboration capabilities if the users involved are on a suitable premium account (Team/Enterprise). It would be good to enable greater freedom and potentially lower costs by enabling the capability to operate through collaboration platforms such as Teams and Slack. This means the next steps need to look something along the lines of:

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit v4 the big news

17 Thursday Apr 2025

Posted by mp3monster in Books, Fluentbit, General, Technology

≈ Leave a comment

Tags

book, CNCF, development, eBPF, Fluent Bit, logs, OpenTelemetry, processors, sampling, Security, Trace, zig

With the announcement of Fluent Bit v4 at Kubecon Europe, we thought it worthwhile to take a look at what it means, aside from celebrating 10 years of Fluent Bit.

Firstly, normally using Semantic Versioning would suggest likely breaking (or incompatible changes to use SemVer wording) changes. The good news is that, like all the previous version changes for Fluent Bit the numbering change only reflects the arrival of major new features.

This is good news for me as the author of Logs and Telemetry with Fluent Bit, as it means the book remains entirely relevant. The book obviously won’t address the latest features, but we’ll try to cover those here as supplemental content.

Let’s reflect upon the new features, their benefits, and their implications.

  • New features in Processors allowing:
    • Conditionality to be included.
    • Trace sampling.
  • More flexible support for TLS (v1.3, choosing ciphers to enable)
  • New language for custom plugins in the form of Zig

Security Improvements

While security for many is not something that will get most developers excited about, there are things here that will make a CSO (Chief Security Officer) smile. Any developer who knows implementing security behaviors because it is a good thing, rather than because you have been told to do it, makes a CSO happy, puts them in a good place, to be given some more lianency when there is a need to do something that would get the CSO hot under the collar. Given this, we can now win those points with CSOs by using new Fluent Bit configurations that control TLS versions (1.1 – 1.3) and ciphers to support in use.

But even more fundamental than that are the improvements around basic credentials management. Historically, credentials and tokens had to be explicit in a configuration file or referenced back to an environment variable. Now, such values can come from a file, and as a result, there is no explicitness in the configuration. File security can manage access and visibility of such details. This will also make credentials rotation a lot easier to implement.

Processor Improvements

The processor improvements are probably the most exciting changes. Processors allow us to introduce additional activities within the pipeline as part of a process such as an input, rather than requiring additional buffer fetch and return which we see in standard plugin operations.

Of course, the downside is that if the processor introduces a lot of effort, we can create unexpected problems, such as back pressure, for example, as a result of a processor working hard on an input.

The other factor that extending processors bring is that they are not supported in classic format, meaning that to exploit such formats, you do need to define your configuration using YAML. The only thing I’m not a fan of, is that the configuration for these features does make me think I’m having to read algorithms expressed with Backus Naur form (BNF).

Trace Sampling

Firstly, the processors supporting OpenTelemetry Tracing can now sample. This is probably Fluent Bit’s only weakness in the Open Telemetry domain until now. Sampling is essential here as traces can become significant as you track application executions through many spans. When combined with each new transaction creating a new trace, traces can become voluminous. To control this explosion on telemetry data, we want to sample traces, collecting a percentage of typical traces (performance, latency, no errors, etc) and the outliers, where tracing will show us where a process is suffering, e.g., an end-to-end process is slowing because of a bottleneck. We can dictate how the sampling is applied based on values of existing attributes, the trace status, status codes, latencies, the number of spans, etc.

Conditionality in Processors

Conditionality makes it easier to respond to aspects of logs. For example, only when the logging payload has several attributes with specific values do we want to filter the event out for more attention. For example, an application reporting that it is starting up, and logs are classified as representing an error – then we may want to add a tag to the event so it can be easily filtered and routed to the escalation process.

Plugins with Zig

The enablement of Zig for plugin development (input, output and filters) is strictly an experimental feature. The contributors are confident they have covered all the typical use cases. But the innate flexibility supporting a language always represents potential edge cases never considered and may require some additional work to address.

Let’s be honest: Zig isn’t a well-known language. So, let’s start by looking briefly at it and why the community has adopted it for custom plugin development as an alternative to the existing options with Lua and WASM.

So Zig has a number of characteristics that align with the Fluent Bit ethos better than Lua and WASM, specifically:

  • It is a compiled rather than interpreted language, meaning that we reduce the runtime overheads of an interpreter or JIT compiler such as Lua and the proxy layer of WASM. This aligns to be very fast/minimal compute overhead to do its job, – ideal for IoT and minimising the cost of side-care container deployments.
  • The footprint for the Zig executable is very, very small—smaller than even a C-generated binary! As with the previous point, this lends itself to common Fluent Bit deployments.
  • The language definition is formally defined, compact, and freely available. This means you should be able to take a tool chain from anyone, and it is easy for specialist chip vendors to provide compilers.
  • Based on those who have tried, cross-compiling is far easier to deal with than working with GCC, MSVC, etc. Making it a lot easier to develop with the benefits we want from Go. Unlike Go – to connect to the C binary of Fluent Bit doesn’t require the use of a translation layer.

One of Zig’s characteristics that differs from C is its stronger typing and its approach of, rather than prescribing how edge cases are handled, e.g., null pointers, working to prevent you from entering those conditions.

Zig has been around for a few years (the first pre-release was in 2017, and the first non-pre-release was in August 2023). This is long enough for the supporting tooling to be pretty well fleshed out with package management, important building blocks such as the HTTP server, etc.

While asking a large enterprise with more conservative approaches to development (particularly when IT is seen as an overhead, and source of risk rather than a differentiator/revenue generator) to consider adopting Zig could be challenging compared to adopting, say Go. The different potential values here, make for some interesting potential.

Not Only, but Also

While we have made some significant advancements, each Fluent Bit release brings a variety of improvements in its plugins. For example, working with it with eBPF, HTTP output supports more compression techniques, such as Snappy and ZSTD, and Exit having a configurable delay.

The Plus version of library dependencies is being updated to exploit new capabilities or ensure Fluent Bit isn’t using libraries with vulnerabilities.

Additional resources

    • Chronosphere announcement

    • Announcement YouTube video

    • book
    • My links to technical resources – we’ve extended to include Zig related resources

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Checking your OpenTelemetry pipeline with Telemetrygen

30 Tuesday Apr 2024

Posted by mp3monster in Fluentbit, Fluentd, General, Technology

≈ Leave a comment

Tags

Docker, docker compose, FluentBit, Go, logs, metrics, OpenTelemetry, signals, Telemetrygen, Traces

Testing OpenTelemetry configuration pipelines without resorting to instrumented applications, particularly for traces, can be a bit of a pain. Typically, you just want to validate you can get an exported/generated signal through your pipeline, which may not be the OpenTelemetry Collector (e.g., FluentBit or commercial solutions such as DataDog). This led to the creation of Tracegen, and then the larger Telementrygen.

You can use Tracegen or Telemetrygen by either downloading and running the Go app from GitHub or using the Docker file. But there are a couple of challenges:

  • On initial investigation, these utilities appear wrapped up in the larger opentelemetry-collector-contrib. While potentially useful, shaking out your OTel pipelining is somewhat overkill.
  • We can install the app locally with the following command, but then we need to set up Golang in the environment.
go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
  • Fortunately, there is a Docker image that just contains the tool, but to use it, we need to know what the parameters are to override the container defaults. The only irritant is that you either need to mess about with the container to get at the information (i.e., run the — help options) or install the utility (the parameters are not in the GitHub docs), so we’ve teased out all the options into the following table.

The following table’s Signal column All means it can be applied to Metrics, Traces, or Logs. Otherwise we’ve named the signal type that the parameter can be used with.

SignalParameter/FlagDescription
Trace–batchWhether to batch traces (default true)
All–ca-cert stringTrusted Certificate Authority to verify server certificate
Traces–child-spans intClient certificate file

All
–client-key stringClient private key file
All–duration durationFor how long to run the test
All-h, –help {traces|metrics|logs}help – with give you the basic help if no parameter is passed. Or the signal type help when used with the signal name e.g. telemetrygen traces --help
All–interval durationReporting interval (default 1s)
Traces–marshalWhether to marshal trace context via HTTP headers
All–mtlsWhether to require client authentication for mTLS
All–otlp-attributes map[string]stringCustom resource attributes to use. The value is expected in the format key=”value”. Note you may need to escape the quotes when using the tool from a cli. Flag may be repeated to set multiple attributes (e.g --otlp-attributes key1=\"value1\" --otlp-attributes key2=\"value2\")
All–otlp-endpoint stringDestination endpoint for exporting logs, metrics and traces
All–otlp-header map[string]stringCustom header to be passed along with each OTLP request. The value is expected in the format key=”value”. Note you may need to escape the quotes when using the tool from a cli. Flag may be repeated to set multiple headers (e.g --otlp-header key1=\"value1\" --otlp-header key2=\"value2\")
All–otlp-httpWhether to use HTTP exporter rather than a gRPC one
All–otlp-http-url-path stringWhich URL path to write to (default "/v1/traces")
All–otlp-insecureWhether to enable client transport security for the exporter’s grpc or http connection
All–rate intApproximately how many metrics per second each worker should generate. Zero means no throttling.
Traces–service stringService name to use (default "telemetrygen")
Traces–size intDesired minimum size in MB of string data for each trace generated. This can be used to test traces with large payloads, i.e. when testing the OTLP receiver endpoint max receive size.
Traces–span-duration durationThe duration of each generated span. (default 123µs)
Traces–status-code stringStatus code to use for the spans, one of (Unset, Error, Ok) or the equivalent integer (0,1,2) (default “0”)
All–telemetry-attributes map[string]stringNumber of traces to generate in each worker (ignored if the duration is provided) (default 1)
Traces–traces intNumber of traces to generate in each worker (ignored if duration is provided) (default 1)
All–workers intNumber of workers (goroutines) to run (default of 1)
Metrics–metric-type metricTypeNumber of metrics to generate in each worker (ignored if the duration is provided) (default 1)
Metrics–metrics intNumber of logs to generate in each worker (ignored if the duration is provided) (default 1)
Logs–body stringBody of the log (default “the message”)
Logs–logs intThe severity number of the log ranges from 1 to 24 (inclusive) (default 9)
Logs–severity-number int32The severity number of the log ranges from 1 to 24 (inclusive) (default 9)
Logs–severity-text stringSeverity text of the log (default “Info”)
All the configuration parameters for Telemetrygen

It is worth noting that while Tracegen has similar configuration parameters, they aren’t exactly the same in the CLI, often one dash rather than two in the name for example.

The following is a simple Docker compose file that can help you use the container to conduct local testing of your collector. In this configuration, we’re sending a trace to the host machine with HTTPS disabled.

services:
  web:
    image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest
    network_mode: host
    entrypoint:
      [
        "/telemetrygen",
        "traces",
        "--otlp-insecure",
        "--otlp-endpoint",
        "host.docker.internal:4317",
      ]

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Cloud Observability in Action – Book Review

04 Thursday Jan 2024

Posted by mp3monster in Books, General

≈ Leave a comment

Tags

book, development, FluentBit, Fluentd, manning, Michael Hausenblas, o11y, observability, OpenTelemetry, Prometheus, review

With the Christmas holidays happening, things slowed down enough to sit and catch up on some reading – which included reading Cloud Observability in Action by Michael Hausenblas from Manning. You could ask – why would I read a book about a domain you’ve written about (Logging In Action with Fluentd) and have an active book in development (Fluent Bit with Kubernetes)? The truth is, it’s good to see what others are saying on the subject, not to mention it is worth confirming I’m not overlapping/duplicating content. So what did I find?

Observability in Action by Michael Hausenblas
Cloud Observability in Action by Michael Hausenblas

Cloud Observability In Action has been an easygoing and enjoyable read. Tech books can sometimes get a bit heavy going or dry, not the case here. Firstly, Michael went back to first principles, making the difference between Observability and monitoring – something that often gets muddied (and I’ve been guilty of this, as the latter is a subset of the former). Observability doesn’t roll off the tongue as smoothly as monitoring (although I rather like the trend of using O11y). This distinction, while helpful, particularly if you’re still finding your feet in this space, is good. What is more important is stepping back and asking what should we be observing and why we need to observe it. Plus, one of my pet points when presenting on the subject – we all have different observability needs – as a developer, an ops person, security, or auditors.

Next is Michael’s interesting take on how much O11y code is enough. Historically, I’ve taken the perspective – that enough is a factor of code complexity. More complex code – warrants more O11y or logging as this is where bugs are most likely to manifest themselves; secondly, I’ve looked at transaction and service boundaries. The problem is this approach can sometimes generate chatty code. I’ve certainly had to deal with chatty apps, and had to filter out the wheat from the chaff. So Michael’s approach of cost/benefit and measuring this using his B2I ratio (how much code is addressing the business problems over how much is instrumentation) was a really fresh perspective and presented in a very practical manner, with warnings about using such a measure too rigidly. It’s a really good perspective as well if you’re working on hyperscaling solutions where a couple of percentage point improvements can save tens of thousands of dollars. Pretty good going, and we’re only a couple of chapters into the book.

The book gets into the underlying ideas and concepts that inform OpenTelemetry, such as traces and spans, metrics, and how these relate to Observability. Some of the classic mistakes are called out, such as dimensioning metrics with high cardinality and why this will present real headaches for you.

As the data is understood, particularly metrics you can start to think about how to identify what normal is, what is abnormal, or an outlier. That then leads to developing Service Level Objectives (SLOs), such as an acceptable level of latency in the solution or how many errors can be tolerated.

The book isn’t all theory. The ideas are illustrated with small Go applications, which are instrumented, and the generated metrics, traces, and logs. Rather than using a technology such as Fluentd or Fluent Bit, Michael starts by keeping things simple and directly connecting the gathering of the metrics into tools such as Prometheus, Zipkin, Jaeger, and so on. In later chapters, the complexity of agents, aggregators, and collectors is addressed. Then, the choices and considerations for different backend solutions from cloud vendor-provided services such as OpenSearch, ElasticSearch, Splunk, Instana and so on. Then, the front-end visualization of the data is explored with tools such as Grafana, Kibana, cloud-provided tools, and so on.

As the book progresses, the chapters drill down into more detail, such as the differences and approaches for measuring containerized solutions vs. serverless implementations such as Lambda and the kinds of measures you may want. The book isn’t tied to technologies typically associated with modern Cloud Native solutions, but more traditional things like relational databases are taken into account.

The closing chapters address questions such as how to address alerting, incident management, and implementing SLOs. How to use these techniques and tools can help inform the development processes, not just production.

So I would recommend the book, if you’re trying to understand Observability (regardless of a cloud solution or not). If you’re trying to advance from the more traditional logging to a fuller capability, then this book is a great guide, showing what, why, and how to evaluate the value of doing so.

To come back to my opening question. The books have small points of overlap, but this is no bad thing, as it helps show how the different viewpoints intersect. I would actually say that the Observability in Action shows how the wider landscape fits together, the underlying value propositions that can help make the case for implementing a full observability solution. Then, Logging in Action and the new book, Fluent Bit with Kubernetes, give you some of the common context, and we drill into the details of how and what can be done with Fluent Bit and Fluentd. All Manning needs now is content to deep dive into Prometheus, Grafana, Jaeger, and OpenSearch to provide an end-to-end coverage of first principles to the art of the possible in Observability.

I also have to thank Michael for pointing his readers and sections of Logging in Action that directly relate and provide further depth into an area.

Further reading

  • Michael’s medium blog
  • Michael’s website
  • Return on Investment Driven Observability
  • CNCF Observability Whitepaper
  • My additional resources for Fluent Bit and Fluentd which includes some of the related content

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
← Older posts
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d