• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: AI

Moving Open Source Observability backend closer to turn-key solutions

09 Monday Nov 2026

Posted by mp3monster in AI, Cloud Native, Fluent Observability, General, Technology

≈ Leave a comment

Tags

agentic, agents, AI, collector, Fluent Bit, Fluentd, Lua, observability, open-source, OpenTelemetry, OTel, OTTL, Transformation, Vector

In our first post on Moving Open-Source Observability closer to turn-key solutions, we discussed how we can automate the source-to-pipeline. But turn-key solutions also flow data to the backend and provide logs and metrics that are contextually aware of the systems being monitored.

While detecting sources and building the capture mechanism may seem more challenging at first glance, it is easier – we have a defined technology, and our observability agent defines the output structure. So what makes the pipeline to the backend more complex?

  • We’re more likely to have a variety of products, and some products may have functional overlaps.
  • Greater configuration diversity in how tools can be configured.
  • Greater contextual awareness is needed.

Let’s break these down in more detail.

Greater variety of tools means we need a common way to set up transformations; otherwise, the mechanics of a transformation process in our pipeline will explode in complexity.

One benefit of going open source is that you can choose the tools that fit your needs for log analytics, metrics, alert routing, and so on, but this can lead to functional overlaps. For example, you may use Prometheus for infrastructure-based metrics, but want to use Elastic APM for Application Performance Management (APM) as you’re using an Elastic Java stack for app development.

The different approaches to configuring tools, or configuration diversity, is probably the single largest challenge, and the number of approaches multiplies the number of configuration permutations that must be achievable. While there appears to be some initia behind the support the Prometheus mixin framework which provides templates for different scenarios such as infrastructure dashboards and alerts. It is far from universal adoption today. For example, Prometheus and Grafana support its use. But other solution stacks such as Elastic do not. If you align on a common technology stack, the chances of a harmonised configuration process rise.

We need contextual awareness from the source, as well as the collected data, to map data to the table(s). For efficiency, if we can avoid putting all metrics into a single table, queries could have one less dimension to filter. Context awareness will inform how we render the dashboards and set alert thresholds. For example, general filesystem storage capacity thresholds would be hugely different to database storage thresholds.

All of this means that if we want a deterministic, transformation-based process, we’ll end up with an unsustainable set of combinations that need to be mappable, testable, and maintainable as the different solutions grow, evolve, and occasionally get archived as contributions dry up.

As turnkey solutions, bring an opinionated answer and either build their own backends from scratch (e.g. Oracle’s Enterprise Manager) and leveraging open source as building blocks – they effectively eliminate the challenges of functional overlaps and configuration diversity – at the same time removing that flexibility. To underline how messy things could get, we experimented in a branch of the repo to see how messy it could get, while trying to keep the divergence as narrow as possible. This diagram illustrates the point visually.

The complexity and variability of pipeline tools and observability back ends

Solution

If a deterministic approach is unsustainable, are there options to ease the challenge or achieve a solution?

Simplification option

One option that can simplify, but certainly doesn’t solve, the challenge is to leverage OpenTelemetry’s work on semantic conventions, where they have been cultivating naming conventions that define data elements likely to be transmitted. When you manually map data, the process becomes more intuitive (and therefore quicker) and improves the chances of an AI agent mapping successfully. In some respects, this is like middleware-driven efforts to build data ontologies, so you need to avoid the problems global ontologies had.

Bringing AI to the game

Because configuration is linguistic and well documented, you can apply an LLM-based approach to the problem. The process of mapping from the ingestion part of the pipeline to the back end needs to be flow-driven so we can:

  • Perform a validation process by selecting and using validation tools to verify the configuration(s).
  • When validations fail, the process can be pushed back to the configuration stage again.
  • When users are in the loop, it becomes possible to drive a conversation for more clarity.
  • With a conversational process, it becomes possible to create in-pipeline transforms as a result of user conversation, along with simpler transforms to handle possible data formats and names.

What we’re describing here isn’t trivial, and could arguably be offered as a commercial product in its own right. The config management feature of our OpAMP solution also provides meaningful inputs, as it defines valid data types for observability agents like Fluent Bit where a concrete schema is not provided.

Using AI – agent graph

Agent descriptions

The agentic process illustrated in the preceding diagram is not trivial. It will be pretty context-heavy, as much of the information to work with can’t be easily addressed through targeted RAG or semantic search. In most cases, these core agents will need tools such as web search, since their training isn’t as up to date as the latest features of the different tools.

  • Input evaluator – We need a variety of inputs covering the target systems and, if there is overlap in capability, guidance on which ones to use. We need to tell the agent what tooling the pipeline needs. We’ll likely need metadata on the sources. For example, if we detect a database, we need to know which type so we can inject the correct queries. The input evaluator will use this to decide whether the detail is sufficient. The result of this analysis could be one of several things:
    • Use a client conversational agent that chats with the user to get clarifications; there is some overlap here, as the conversation will need to know what answers are needed, and some answers may open up new questions. Once this agent believes it has compiled the details for the answers. it returns to the Input Evaluator to reassess the picture.
    • We can ask the evaluator to pause the process and generate a report showing the gaps identified. This can be done using the Clarifications Report generator.
    • The Input evaluator may determine that it reports the gaps and presses on with a best effort.
  • Semantic normalizer – To simplify the flow, mapping source names to the semantic definitions from OpenTelemetry would need to understand the source and the OTel Semantic Naming model.
  • Pipeline generator – Once we confirm the details and have simplified semantics, we can have the AI build out the rest of the pipeline. The agent may need to be handed back to the agent for correction.
  • Config validator – once the pipeline generator hands off the configurations, the validator agent determines the best tools to verify whether the configurations are valid. If the tool returns an error, it hands the cause back to the pipeline generator. Ideally, tool calls will return meaningful error descriptions. But if not, we’d need to create a simple knowledge base to extract details from.
  • Clarifications Report generator – The clarifications report generator’s role is to take the gaps identified and generate and share them via a document repository or email in a user-friendly format
  • User clarification – The conversational interface with the user to obtain missing details, according to the Input evaluator. One difference between this and the Input evaluator is that this agent will need to consider conversational history to build the required additional detail. The input evaluator shouldn’t need prompting.

Controlling complexity

We’ve seen that allowing flexibility and removing the opinionated constraints of turnkey solutions increases complexity very quickly. The AI solution will become much more reliable if we strip back the options we allow agents to choose from, or if we push more organizations to standardize on consistent configuration strategies that enable a move toward templatization.

A Unified pipeline syntax

The other path that could radically simplify things is the Open Telemetry Transformation Language (OTTL), which has just reached the version 1 milestone. While it is not a complete solution, it offers a way to advance the standardized pipelines. We’ll explore OTTL in a separate blog in more depth. But essentially, if you look at the collector as two parts- the wire-level mechanism to capture or emit data and a second part for routing and manipulation- then OTTL is about the routing and manipulation. OTTL has both a formal syntax and an implementation as a Go library. While the library may not help if you’re using Vector, Fluentd, Fluent Bit, etc., the formal notation will work with ANTLR4; the resources are available through the OTLery OTTL-Lang repo. This allows us to generate, across many languages (C, Java, Ruby, Lua, etc.), a parser for the OTTL syntax into a tree structure that can then be walked. This still needs the intelligence to translate data references into references for the observability tool, and the functions, editors, and transformers to be implemented. Still, most of these can be mapped onto language-native libraries.

With a formal notation and a growing catalogue of prepared OTTL scripts and utilities, such as those offered by the OTLery repos, there is the ability to leverage AI to generate the transformation logic. As many observability tools such as Fluent Bit and Vector include support for scripting and plugins, incorporating a complete parser becomes a practical option. With both Vector and Fluent Bit, for example, supporting Lua being incorporated into the pipeline, and ANTLR4 supporting Lua as well, it should be possible to implement OTTL in those pipelines.

There is one downside to using OTTL: we lower our observability agents to the lowest common denominator. For example, Fluent Bit provides basic stream analytics. Transformations are now going through a parser phase rather than going directly to native binary processes.

Conclusion

The more we can restrict options, the closer we can get to being able to automate the creation of the observability pipeline quickly, and efficiently, but at the price of potentially not maximizing feature exploitation (the downside of using OTTL across different observability agents). The problem is to provide a solution that is very flexible, which will require a fair bit of effort. That flexibility also creates the possibility of exploiting the full feature set of the available tools.

Useful links

  • Prometheus Monitoring Mixins
  • OpenTelemetry Semantic Conventions
  • Elastic APM for Application Performance Management (APM)
  • Open Telemetry Transformation Language (OTTL)
  • OTLery OTTL-Lang
  • OTTL Go library
  • ANTLR4
  • Configuration Generator main repository

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Are we heading to a Certificate-apocalypse?

29 Thursday Oct 2026

Posted by mp3monster in development, General, Security, Technology

≈ Leave a comment

Tags

ACME, AI, certbot, certificates, CLM, Cloud, CMC, CMP, cybersecurity, SCEP, Security, Technology, TLS

Most engineers are probably aware of the shortening life span of TLS certificates. this year certificate maximum life drops from 398 to 200 days. But what may not have registered is that the roadmap sees the max certificate time will continue to half every year. This means that by 2029 certificate lifecycle will be down to 47 days (31 days (core duration) + core duration/2 days + 1 day). But before 2029, in less than six months we’ll see the current certificate durations half from 200 to 100 days. As with the 2026 change, the new limits come into effect on the 15th March. Which does create an odd situation where certificates issued on the 14th March will outlast certificates issued on the following day.

The addition of half the core duration period is to allow for the recovery from failures. as of 2029 the expectation is every month the certificates should be rotated.

Manual processes are unsustainable

This means that the rotation of certificates has got to be accelerated to be monthly, a velocity that is simply not practical to be done by hand. This means the deployment of certificates really needs to be automated. There are a number of frameworks and standards that support the distribution of certificates.

Perhaps the best known and leading solution has been ACME protocol working in conjunction with Let’s Encrypt CA. The least known is the use of OpAMP also supports certificate distribution. The following table shows the different standards that specifically address certificate management (CLM), some implementations are linked to a Certificate authority , others are not.

StandardSpecificationOpen-source implementationsCommercial implementations
ACMERFC 8555Certbot (client), acme.sh (client), lego (client), step-ca (server), Boulder (server)DigiCert, Sectigo, GlobalSign, EJBCA Enterprise
CMP / CMPv3RFC 9810EJBCA Community, Bouncy Castle (library)EJBCA Enterprise, DigiCert Trust Lifecycle Manager
ESTRFC 7030Cisco libEST (client/server library), EJBCA Community (edition-dependent)EJBCA Enterprise, DigiCert
SCEPRFC 8894MicroMDM SCEP, EJBCA CommunityMicrosoft Intune, EJBCA Enterprise, DigiCert
CMCRFC 5272Dogtag PKICommercial implementations are less prominent; verify explicit CMC support and supported operations with individual PKI vendors

To understand the differences between the different standards, it is worth looking at the standards from the perspective of the operations that are needed to manage certificates.

FunctionACMECMPESTSCEPVendor
CLM API
Certificate issuanceYesYesYesYesUsually
Certificate renewalYesYesYesYesUsually
Certificate revocationYesYesNo native operationLimitedUsually
CA certificate rolloverLimitedYesYesLimitedVaries
Central certificate inventoryNoNoNoNoUsually
Ownership / application mappingNoNoNoNoVaries
Policy managementLimitedLimitedLimitedLimitedUsually
Application deploymentNoNoNoNoVendor-specific
Application activationNoNoNoNoThrough integrations
Deployment verificationNoNoNoNoVaries

You’ll note that the deployment, activation and verification steps fall into the domain of vendor solutions, and are variable. However we can address such considerations in an open-source (or commercial) manner through tooling such as Puppet, Chef and certain payload types provided by OpAMP.

These CLM options aren’t the end of the story as you may wish to look more broadly at no just the request and distribution of certificates, but the mechanisms through which the certificate is stored, and there is an overlap with such management with keys as well as certificates. For example APIs may use keys of certificates to help verify authenticity of the request. This also extends to mTLS operations where the client needs certificate to verify the server requests. For example KMIP provides an interface that can support cryptographic object (keys, certificates (public, private, symmetric), secret data, PGP Keys).

StandardSpecificationOpen-source implementationsCommercial implementations
KMIPOASIS KMIPPyKMIP (Python client/server)HashiCorp Vault Enterprise, Thales CipherTrust
PKCS#11PKCS#11 v3.1SoftHSM, OpenSCHashiCorp Vault Enterprise, Thales Luna HSM
SCIMRFC 7644elimity SCIM (Go library)Microsoft Entra ID, Okta

Managing the impact

There are several impacts that need to be considered:

  • tracking certificate expiry
  • Last mile from client deployer to server folder and any refresh nudge required

Tracking certificate expiry

Traditionally certificate expiry management could be handled as a manual process – after all if you have a log of which systems had certificates then if you simply reviewed the list once a month, you could spot expiries with time to spare. Even if the certificate distribution process was automated across many nodes. This just isn’t going to work, and the contingency time incorporated into the certificate life is contracting. Rather than trusting to a tracker spreadsheet being managed we should really focus on actual certificate data. This points to having a means by which application observability has a reporting metric on when a certificate will expire (date or a number of days). The later will prove to be easier as a metrics measure, but the value needs to be in context of when the metric was collected.

Last mile challenge

While automation frameworks like Certbot and step-ca can drive the certificate issuance. These processes get the certificate to the relevant server. But we still need to move the certificate intro the location the application expects the certificate to be, and do we need to nudge the server to reload the certificate? Certbot for example expects to have a script to trigger for this action. This can get a little messy to deploy and maintain, particularly if there are subtle variances in deployments (as a result of manual application deployment in legacy environments), configuration creep etc. One possible way to simplify this is to leverage the capabilities of environment scanners to locate apps and their configs. Then use that to drive the auto creation of that last mile.

Why the change?

The history for the reduction has largely been driven by browser providers particularly Google and Apple, and their security concerns about the trustworthiness of the web’s public key infrastructure (PKI). Historically, certificates could remain valid for several years, creating a prolonged window of exposure if a private key was compromised, a certificate incorrect certificates being wrongly used (organization or domain information became outdated – fore example domain name changes, acquisition of services etc). Certificate revocation mechanisms, particularly CRLs and OCSP, have not proven to be reliable as a means of ensuring browsers quickly stop trusting compromised certificates. Shorter lifetimes reduce this exposure and encourage the automation of certificate management.

The driving of changes has shifted from browser providers unilaterally declaring they won’t trust certificates that are older than a certain age, to a more formalized process governed primarily by the CA/Browser Forum, (an independent body bringing together Certificate Authorities and browser vendors).

Useful resources

  1. A/Browser Forum — TLS Baseline Requirements, version 2.3.1 (4 October 2026), section 6.3.2. Authoritative current hard limits: 398 days before 15 March 2026; 200 days from that date; 100 days from 15 March 2027; 47 days from 15 March 2029. Section 1.1 and the revision history identify 1 July 2012 as the original effective date. https://cabforum.org/working-groups/server/baseline-requirements/requirements/
  2. CA/Browser Forum — FAQ for Baseline Requirements. Historical summary of the 60-month limit from July 2012 and the 39-month general limit from April 2015. This is a historical source, not the current rule set. https://cabforum.org/faq-about-the-baseline-requirements/
  3. CA/Browser Forum — Ballot 193: 825-day Certificate Lifetimes (17 March 2017). Establishes the March 2018 transition to 825 days; also records the preceding 39-month rule and the qualifying 60-month legacy exception through June 2016. The page contains amendment markup; read the replacement wording in context. https://cabforum.org/2017/03/17/ballot-193-825-day-certificate-lifetimes/
  4. Apple — About upcoming limits on trusted certificates. Announces the 398-day maximum for public TLS server certificates issued on or after 1 September 2020 and explains its scope. This change originated in browser/platform trust policy. https://support.apple.com/en-us/102028
  5. CA/Browser Forum — Ballot SC081v3: Introduce Schedule of Reducing Validity and Data Reuse Periods (11 April 2025). Records the successful vote for the 2026–2029 reductions; source 1 confirms the resulting current requirements. The April milestone is the vote result, not the end of the subsequent intellectual-property review period. https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/
  6. IETF Datatracker — draft-barnes-acme-00 (28 January 2015). First revision of the ACME Internet-Draft, before working-group standardization. https://datatracker.ietf.org/doc/draft-barnes-acme/00/
  7. Let’s Encrypt / ISRG — Entering Public Beta (3 December 2015). Announces public availability without invitations. https://letsencrypt.org/2015/12/03/entering-public-beta
  8. Electronic Frontier Foundation — Announcing Certbot: EFF’s Client for Let’s Encrypt (May 2016). Explains the rename, the EFF home, and ACME-based issuance and configuration automation. https://www.eff.org/deeplinks/2016/05/announcing-certbot-new-tls-robot
  9. Let’s Encrypt community — New name (Certbot) for Python Let’s Encrypt client (12 May 2016). Dated contemporary announcement linking to EFF’s announcement; supports the exact event date. https://community.letsencrypt.org/t/new-name-certbot-for-python-lets-encrypt-client/15688
  10. RFC Editor — RFC 8555: Automatic Certificate Management Environment (ACME), March 2019. Standards-track specification, published as a Proposed Standard. https://www.rfc-editor.org/info/rfc8555/
  11. ACME — RFC 8555: Automated Certificate Management Environment; supports automated certificate issuance, renewal and revocation.
  12. CMP / CMPv3 — RFC 9810: Certificate Management Protocol; supports PKI enrolment and certificate lifecycle operations.
  13. CMP over HTTP — RFC 9811: Defines HTTP transport for CMP messages.
  14. EST — RFC 7030: Enrollment over Secure Transport; supports certificate enrolment and re-enrolment over HTTPS.
  15. SCEP — RFC 8894: Simple Certificate Enrolment Protocol; commonly used for network devices.
  16. CMC — RFC 5272: Certificate Management over CMS; supports certificate requests and management messages. Updated to — RFC 10002: Updated specification superseding RFC 5272.
  17. KMIP 2.1 — OASIS KMIP 2.1 Specification: Key Management Interoperability Protocol; manages cryptographic keys, certificates and other cryptographic objects.
  18. PKCS#11 3.1 — OASIS PKCS#11 3.1 Specification: Cryptographic Token Interface; standard API for accessing cryptographic hardware and software tokens.
  19. SCIM 2.0 — RFC 7644: System for Cross-domain Identity Management; supports identity and account provisioning rather than certificate management.
  20. SPIFFE — SPIFFE Specifications: Secure Production Identity Framework for Everyone; standardizes workload identity across heterogeneous systems.
  21. X.509-SVID — X.509-SVID Specification: Defines X.509 certificates used for workload identity and authentication.
  22. X.509 PKIX — RFC 5280: Defines the Internet X.509 public key certificate and Certificate Revocation List (CRL) profile.
  23. CA/Browser Forum Baseline Requirements — Current Baseline Requirements: Industry requirements governing publicly trusted TLS certificate issuance, domain validation, security practices and maximum certificate validity periods.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Can patches have functional impacts?

27 Tuesday Oct 2026

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, cybersecurity, languages, OS, patches, patching, Security, Technology, vulnerability

With the recent upheaval from the newly discovered PHP vulnerability (often referred to as PHP2Shell), and updates to OpenTelemetry declarations, it struck me that it’s worth highlighting an assumption we all tend to make with patches.

We tend to think patches introduce fixes or defences that mean functional behavior is immutable. It is an understandable reading based on definitions such as Semantic Versioning (semver):

PATCH version when you make backward compatible bug fixes

However, languages and frameworks point towards a different model, where defined API behaviour will be respected.

The bottom line is if you’re exploiting undocumented behaviors, it makes it all the more necessary to test and ideally in an automated manner changes not only in the applications but also for the stack.

If you are taking advantage of undocumented behavior, then it is worth adding an abstraction layer to allow mitigation changes to be added.

The following identifies the statements at a language level relating to what impact a patch can have:

Language / runtimeBehavior changes in patchesBreaking API changes in patchesPolicy, limitations and supporting documentation
GoYesExceptionalExplicit security, unspecified-behavior and specification-correction exceptions. Go compatibility promise
PHPYes, for correctionsGenerally noSecurity and bug fixes permitted; public userland API must remain backward compatible. Internal ABI/API has narrowly defined exceptions. PHP release policy
Python (CPython)Yes, restrictedExceptionalReasonable bug fixes may affect behavior; stronger incompatibilities require exceptional justification. PEP 387
Java (OpenJDK / Oracle JDK)YesGenerally noSecurity corrections can alter runtime behavior; source, binary and behavioral compatibility are evaluated separately. OpenJDK compatibility
C# / .NET runtimeExceptionalExceptionalBreaking changes generally prohibited during servicing, except for security vulnerabilities or other critical issues. .NET release policy
F# / .NET runtimeExceptionalExceptionalShares .NET runtime servicing policies; compiler and language tooling have their own compatibility considerations. .NET compatibility rules
VB.NET / .NET runtimeExceptionalExceptionalSame underlying .NET servicing guarantees. .NET servicing
TypeScriptYes, restrictedNot normallyPatch releases may contain high-priority regression, performance-regression and safe language-service corrections. Ordinary breaking changes are targeted at larger releases. TypeScript release process
JavaScript (ECMAScript)Not applicable at language-specification levelNot applicableECMAScript is a specification, not a separately patched runtime. Behavior depends on implementations such as V8, SpiderMonkey, JavaScriptCore and Node.js. ECMAScript specification
Node.jsYes, for correctionsExceptionalPatch releases accept fixes; semver-major changes are excluded under normal procedures. LTS security exceptions can affect native API/ABI compatibility when unavoidable. Node.js release procedures

The following table, takes the same approach looking at how operating systems view changes for patches:

Operating systemBehavior changes in maintenance patchesBreaking API / ABI changesPolicy, limitations and supporting documentation
Red Hat Enterprise Linux (RHEL)YesGenerally avoidedBackports security and bug fixes into supported versions. Red Hat explicitly tests for unwanted side effects, so this is not general permission to break compatibility. RHEL security backporting
SUSE Linux Enterprise Server (SLES)YesGenerally avoidedMaintenance includes security corrections and defect fixes; individual components have different lifecycle rules. Service packs allow broader changes. SUSE support policy and SLES release notes
Oracle LinuxYesNo general permission establishedOracle distributes security, bug-fix and enhancement errata as package updates. Security hardening can alter functionality independently of major OS releases. Oracle Linux errata policy
Ubuntu LTSYes, cautiouslyGenerally avoidedBackported security patches prioritize compatibility. Exceptional changes and certain feature updates can be accepted under the Stable Release Updates process. Ubuntu security updates
Debian StableYes, restrictedGenerally avoidedImportant bug and security fixes are acceptable; updates are deliberately minimal and require review. Debian Stable update rules
Microsoft Windows 11 / Windows ServerYesNo general permission establishedCumulative security, quality and out-of-band updates may change functionality. Windows 11 may also deliver feature enhancements through monthly servicing. Windows update release policy
Apple macOSPossibleNo general permission establishedSecurity and maintenance releases can affect operating behavior, but there is no general published patch-version exception equivalent to Go’s. Apple security releases

Often when building solutions we don’t see low level frameworks and how they work, so we may not know about any undocumented feature exploitation.

The challenge it can bring

This points to increased testing, which ideally is automated. The reaction is initially is so what. But, when vulnerabilities like PHP come up which had openly triggered systems being disconnected from the net.

So extending testing (even if automated) is going to experience resistance from the commercial parts of an organization who are unlikely to understand the nuances, but will react to headlining risks.

How do we move forward?

There are some strategies we can adopt that can help us. The industry used to talk about security in depth and, more recently, system resilience. Regardless of terminology, we want to introduce control layers that reduce the chance of malicious intent. Often referred to as compensating controls.

For example, if the vulnerability is overloading the runtime by sending an oversized payload, a firewall or proxy layer can have rules that prevent payloads from being too big from hitting the backend.

If the networking allows it, then segmenting the network reduces the risk of a breach affecting other environments.

This doesn’t mean if we can patch, that we shouldn’t. What it does mean is we can be considered and not knee-jerk panic and unwittingly introduce new problems.

Conclusion

Adding additional security and proxy layers and segmenting system environments may seem like overkill, particularly when introducing new systems. But these steps create the means to be far more resilient.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Moving Open Source Observability closer to turn-key solutions

17 Saturday Oct 2026

Posted by mp3monster in Fluent Observability, General, OpAMP, Technology

≈ 1 Comment

Tags

AI, artificial-intelligence, automation, chatgpt, configuration, Fluent Bit, FluentBit, Fluentd, LLM, logging, monitoring, observability, OpAMP, Open Telemetry, OTel, Technology

There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:

  • Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
  • Deployed services and applications with default monitoring configuration
  • Prebuilt dashboards

While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.

This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.

Detecting and Generating Configuration

The process we need has the following steps:

  • find the deployed solution(s)
  • Understand how they are deployed
  • Generate a configuration based on the detection
  • Push the configuration out to agents

Detect and capture config

One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.

While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …

As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.

Templates make the result reusable

By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.

There are limits

So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.

Convenience as a weakness?

While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.

Backend Bootstrapping

Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.

Useful Links

  • OSQuery homepage and GitHub
  • Jinja2 Doc.
  • Primer on Jinja templating

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OpAMP V0.6 implementation released

12 Monday Oct 2026

Posted by mp3monster in Fluent Observability, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, chatgpt, development, LLM, observability, OpAMP, release, Technology

We’ve made the latest version of our implementation of the OpAMP specification available. It has taken longer to ready than the previous versions, but there is a lot been included, both in the core functionality, but also capabilities supporting the use and setup. Here are the headlines…

  • The project has its own organization now (https://github.com/orgs/opamp-observe/) as mentioned here, but also a domain of https://opamp.info/ which will bounce you to the GitHub pages.
  • Automated regression from clean to deployed and running a series of end-to-end tests as well as producing a guaranteed fresh set of artefacts has been implemented. This can be run against AWS (currently in use) or Azure (needs refining). These are triggered on changes to the main branch.
  • GitHub pages with links all the documentation are established now.
  • The Python wheel release artefacts are no incorporated into the release assets.
  • The configuration for the Open Source Observability Day conference demo is available in the https://github.com/opamp-observe/demo-repository repo.
  • We’ve added support for a couple more open-source observability tools:
    • Vector
    • Elastic Agent
    • Elastic Heart beat
  • The client/consumer side has been refactored so it is easy to build your own observability agent controls and independently deploy them (not supported by the config editor).
  • The CLI has been enhanced so it can help quickly spin up more advanced configurations – helpful for demos and real deployments.
  • The Dev CLI has been improved.
  • The server side now has a basic client side (Supervisor/Observer) configuration editor UI. Which can be deployed as a plugin.
  • Improvements around the use of Podman and Docker for the containerization.
OpaMP v0.6 now labelled

What next?

We’ve added to the documentation pages for the roadmap, initially we’re working to fill out the regression test sets at the moment. Other possibilities:

  • Expand the client types
  • Extend the client types to exploit more features.

Perhaps the most interesting idea, we’re thinking about is can we exploit tools like Netdata to discover what solutions are deployed and generate a requirements/configuration definition which can be used to generate a suitable agent configuration file.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Observability as a security sensor using Fluent Bit, OpenTelemetry and AIDE

09 Friday Oct 2026

Posted by mp3monster in Fluent Observability, Fluentbit, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, cyber-security, cybersecurity, Security, Technology

Observability is usually looked at from the perspective of applications and their needs and the cloud native platforms. Nothing wrong with that, but it is also a rather narrow view of the telemetry we are already collecting or are very capable of collecting. Our observability agents (Fluent Bit, Fluentd, Elastic Agent …) can collect signals (logs, metrics and traces) that can contribute to the security operations within an enterprise. More importantly, the same open-source tooling can help answer a deceptively simple question such as has someone changed the system, or changed the tooling that tells us what the system is doing?

When it comes to security, that second part matters. An attacker does not always need to exploit a new vulnerability. If they can alter a service definition, scheduled task, access-control file, logging configuration or telemetry destination, they can prevent us from sensing what damage they maybe doing, including opening up simpler or more invasive routes into your systems. MITRE ATT&CK describes this kind of action as a defence-evasion technique. Research from Palo Alto Networks Unit 42 has also described cloud logging services as an attack surface, including changes that stop, redirect or selectively reduce audit collection. Not only that getting at logs, means they can potentially determine when systems that are running, and find new attack vectors.

As you can see being able to detect configuration and file manipulation is an important factor for security. Many may react to this with concern about introducing cost for expensive security tools, but if there are simple means to listen to OS activities that relate to file manipulation then we can leverage our observability estate for this aspect of security.

The wider open-source stack

That neutrality for our observability agents is valuable for security. A host or cluster can emit operational and security telemetry through the same managed pipeline, while routing data to one or more destinations such as OpenSearch, Loki, ClickHouse, a SIEM, object storage or an OTLP-compatible service. Not to mention if the event is for an application configuration, we can evaluate it against application and service viewpoints such as managing configuration drift.

The roles are complementary:

  • AIDE is a open-source Linux utility that provides the file-integrity check. It builds a known-good database and reports additions, removals and changes to selected files and attributes. Aide achieves this through the use of the iNodes API, which if you preferred could also be tapped into even more directly.
  • Fluent Bit or other OTel compliant tools collect the result close to the workload, adds context, buffers during a network interruption and routes the event. Our backend services which are in all likelihood OTLP capable, means the events can be send nearly anywhere for analysis and alerting
  • The analytical backend correlates the integrity event with logins, process starts, deployments, traces and other activity, then drives an alert or investigation.

AIDE is deliberately focused. It does not need to become a SIEM and Fluent Bit does not need to become a file-integrity engine. Combining small tools through open interfaces lets each component do the job it is good at.

We can use Aide not only to monitor our configurations, but when the deployed binaries are disturbed. We can extend this setup further. By exploiting custom commands with OpAMP (particularly for a supervisor) can be used to not only manage the observability agent, but also manage the deployment of configuration to tools like Aide. Now if we think something is being abused, the option to change our file integrity monitoring is easily in our reach.

Why file integrity belongs in the observability conversation

The available incident statistics do not normally include a neat category called “configuration-file manipulation”. The behaviour is spread across techniques such as creating or modifying system processes, scheduled tasks, startup mechanisms, accounts, policies and logging controls. That makes a single adoption or attack percentage difficult to defend. It does not make the risk rare.

For example, Google Cloud’s M-Trends 2025 reporting included techniques such as Create or Modify System Process (20.6%), Account Manipulation (19.9%), Scheduled Task/Job (14.3%), Registry Run Keys/Startup Folder (11.5%) and Web Shell (7.2%) among observed investigations. These are broader ATT&CK behaviours, not a count of configuration-file changes, but many are enabled or persisted through changes to system configuration.

There is also a compliance driver. PCI DSS v4.0.1 Requirement 11.5.2 explicitly requires a change-detection mechanism, such as file-integrity monitoring, for critical files, with comparisons at least weekly and alerts for unauthorised modification. NIST SP 800-53 Revision 5 control SI-7 requires monitoring for unauthorised changes to software, firmware and information. Other standards may describe the required outcome as configuration control, integrity, audit protection or system monitoring rather than naming a particular product.

FIM helps provide the evidence behind those outcomes:

  • what changed;
  • which host or workload reported it;
  • when the change was detected;
  • which attributes or content hashes changed;
  • whether a change matched an approved deployment window;
  • whether the alert was acknowledged and investigated; and
  • whether the integrity baseline was updated through a controlled process.

The last point is easily overlooked. Automatically accepting every change into the baseline produces a very tidy report and a very weak control.

What should be monitored?

Starting with every file on every host usually creates cost and noise. I would start with the files that can change control flow, identity, collection or evidence retention:

  • Fluent Bit configuration, parser files and included fragments;
  • Vector configuration and transformation definitions;
  • OpenTelemetry Collector configuration;
  • systemd units and drop-ins for telemetry agents;
  • log rotation and retention configuration;
  • TLS trust stores, client certificates and references to credentials;
  • auditd, syslog and journald configuration;
  • scheduled tasks, startup scripts and authorised SSH keys;
  • alert rules, dashboards and detection content managed as files; and
  • the AIDE configuration and baseline database itself.

Runtime state, offset databases, buffers and normal log files should normally be excluded. They are expected to change and monitoring them obscures the files that should remain stable.

There is an architectural wrinkle: if AIDE reports through Fluent Bit, and an attacker changes both AIDE and Fluent Bit, the local signal can be suppressed. The practical response is defence in depth. Restrict write access, ship events off-host quickly, monitor the agent configuration and service unit, protect or sign the AIDE baseline, and alert when expected heartbeats or scheduled check results disappear. Silence is also a signal.

Example: AIDE reporting through Fluent Bit and OTLP

The following is a starting point rather than a universal production configuration. Package paths and AIDE defaults differ between Linux distributions, so it should be merged with the supplied configuration rather than replacing it blindly.

First, add an AIDE rule group for the observability configuration. For example, in /etc/aide/aide.conf.d/observability.conf:

# Permissions, inode, links, ownership, size, timestamps and SHA-256 content.
OBS_CONFIG = p+i+n+u+g+s+m+c+sha256
​
/etc/fluent-bit                         OBS_CONFIG
/etc/vector                             OBS_CONFIG
/etc/otelcol-contrib                   OBS_CONFIG
/etc/systemd/system/fluent-bit.service.d OBS_CONFIG
/etc/systemd/system/vector.service.d     OBS_CONFIG
/etc/systemd/system/otelcol.service.d   OBS_CONFIG
/etc/aide                               OBS_CONFIG
​
# These are runtime data and should not be treated as static configuration.
!/var/lib/fluent-bit
!/var/lib/vector
!/var/lib/otelcol
!/var/log

The main AIDE configuration can direct its report to a file that Fluent Bit is permitted to read:

database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout=true
​
report_url=file:/var/log/aide/aide.log
report_level=changed_attributes

Initialise and approve the baseline using the procedure provided by the operating-system package. The baseline database should be writable only by the account performing the controlled update; consider retaining a signed or read-only copy outside the host.

Run aide --check from a systemd timer or the organisation’s scheduler. An hourly check provides better detection latency than the weekly minimum stated by PCI DSS, although the right frequency depends on estate size, I/O cost and risk. The scheduler should also emit a success or failure event so that a missing check can be detected.

Fluent Bit can then tail the report, add security context and forward the records over OTLP. The Tail input uses a database to retain offsets, and the OpenTelemetry output supports OTLP/HTTP and OTLP/gRPC.

service:
flush: 5
log_level: info
storage.path: /var/lib/fluent-bit/storage
pipeline:
inputs:
- name: tail
tag: security.fim.aide
path: /var/log/aide/aide.log
key: body
path_key: log.file.path
db: /var/lib/fluent-bit/aide-tail.db
db.sync: normal
read_from_head: false
refresh_interval: 5
rotate_wait: 30
skip_empty_lines: true
skip_long_lines: true
storage.type: filesystem
filters:
- name: modify
match: security.fim.aide
Add:
- event.domain file
- event.category file
- event.type change
- security.control file_integrity
- service.name aide
outputs:
- name: opentelemetry
match: security.fim.aide
host: otel-collector.internal
port: 4318
logs_uri: /v1/logs
tls: on
tls.verify: on
compress: gzip

This example emits each AIDE report line as a log record. That is simple and robust, but the backend will need to group lines belonging to the same check. For a more mature deployment I would place a small wrapper around the AIDE invocation that converts each detected change into JSON Lines and adds a check identifier, hostname, baseline version and execution result. Fluent Bit can then use its JSON parser and every changed file becomes a structured event.

Vector can implement the same pattern with its file source and an OTLP-capable destination or intermediate Collector. The OpenTelemetry Collector can also collect files using the contributed filelog receiver. The choice is less important than applying consistent fields, reliable buffering, secure transport and an independently managed destination.

From event collection to a security control

Collecting the AIDE output is only the first half of the solution. To turn it into a control, the backend needs useful detection logic and operational ownership.

I would start with alerts for:

  • a change to an observability agent destination, filter or exclusion rule;
  • a change to a telemetry service unit or executable;
  • deletion or replacement of the AIDE database;
  • changes outside an approved deployment window;
  • a burst of changes across several hosts;
  • a configuration change followed by a fall in log volume; and
  • a missing AIDE run, agent heartbeat or expected integrity summary.

Correlating the FIM event with deployment records is important. A checksum change shortly after an approved GitOps rollout is very different from the same change following an interactive root login at 02:00. Observability provides the surrounding evidence that allows the integrity event to be assessed rather than simply counted.

There is also value in feeding an integrity event back into operational telemetry. A changed collector configuration can annotate dashboards, traces and incident timelines. The question changes from “why did the logs stop?” to “did the logs stop when this configuration changed?” That is a much better starting point for both an SRE and a security analyst.

Final thoughts

The observability and security domains have spent years building separate pipelines for data that frequently starts on the same machine. Open standards and focused open-source tools give us an opportunity to reduce that duplication.

AIDE supplies a mature integrity signal. Fluent Bit and Vector make that signal transportable and enrichable. OpenTelemetry makes the route to the wider platform portable. None of these components replaces access control, endpoint detection or a SIEM, but together they add a useful security capability to infrastructure many organisations already operate.

Perhaps the most important configuration to observe is the configuration that decides what we are allowed to observe.

Resources

  • AIDE project
  • Fluent Bit Tail input
  • Fluent Bit OpenTelemetry output
  • Vector file source
  • OpenTelemetry Collector configuration
  • MITRE ATT&CK: Impair Defences – Disable or Modify Cloud Logs
  • Unit 42: Abusing Cloud Logging Services for Defense Evasion and Visibility
  • NIST SP 800-53 Revision 5
  • PCI Security Standards document library
File Integrity Management using Aide and Fluent Bit

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Could the AI hacks have unintended consequences for AIOps

19 Saturday Sep 2026

Posted by mp3monster in AI, Fluent Observability, General, Technology

≈ Leave a comment

Tags

AI, AIOps, artificial-intelligence, chatgpt, LLM, OpenAI, Technology

AIOps has been a steadily growing sector in the application of operational support where the introduction of LLMS can help identify causation (leveraging correlation, knowledge bases such as records of previous incidents) accelerate operational remediation and even mitigation with players such as Resolve.ai, ObserverIQ, and Keep (Open Source).

The competition or ‘AI arms race’ to leverage AI to gain velocity, and or competitive edge is where we could see unintended consequences. A little bit of back ground first.

OpenAI’s attack on HuggingFace

The OpenAI unintended attack on HuggingFace is well documeented (along with similar cases for Meta and Anthropic). Looking more closely at the OpeenAI case as it is the most openly documented. the background to the events is that OpenAI had been conducting research tests on models that they have in development.

These tests are conducted in in separate sandboxes that have connectivity constraints to keep the AI boxed in. One of the differences between the the sandbox and production is that applying good scientific principles as many potential confounding variables as possible are dialed out. Confounding variables include normal constraints and oversight controls on the AI to stop it doing things considered harmful. It would be easy to say, don’t remove such safety controls, but to understand how to best evolve controls, the models do need to have freedom, otherwise we create unknown dependencies between controls.

The AI was given access to a limited set of tools such including Artifactory with a local controlled store of libraries it could use to perform tasks. Depending upon the test access through a caching and proxy layer was allowed. Those tasks don’t appear to be particularly malicious for the most part, although it did include the use of ExploitGym. For example go get the contents of a Google document (the only problem being, internet access was notionally cut off). Some of what happened is a little eye opening, the headlines, I picked up on, and believe could impact people’s views are:

  • Prior to the cyber testing simpler test scenarios such as getting the model to complete several spreadsheet tasks using Google’s Office tools. Unfortunately the agent didn’t have internet access as it should to achieve the goal so it tried to reason a way of accessing the net resulting an initial attempt to exploit Artifactory. While it failed, it did store content into Artifactory.
  • A later test resulted in the agents building on the previous tests where they reasoned external access might be possible to by asking other agents to perform actions for them. This lead to Artifactory being used as a collaboration channel.
  • Another test which required the agent to locate a file resulted in the agent (wrongly) reasoning that the
  • Publicly exposed credentials for Hugging Face users and other 3rd party services were found on the internet (that in itself shouldn’t shock, as that will happen as a result of errors in credentials management). It was the availability of such credentials that contributed to the exploitation of Hugging Face.
  • The AI reasoned that to access the internet it needed to a way out. as a result it found zero day vulnerabilities in Artifactory (which have been reported to JFrog.

The following timeline – tells the story over time.

How does this impact AIOps?

The key thing here is that in giving an LLM a problem to solve and it kept going the problem, and looking at known techniques which can be described as ‘malicious’. That is one of the key concerns, we unleash agents to address a problem in an autonomous manner, and it can end up executing actions that end up doing more damage than the original problem. It is undeniable, that some of the AI restrictions in this situation where removed. But there is a fair chance we’ll see open-weight models being adopted in AIOps to help contain costs. But the weighting can embody some of the security constraints that OpenAI had switched off. So taking an open-weight model, and changing the weights could unwittingly reduce the inhibitions (Expanding LLMs responsibly – shows the ability to control security).

While I’m no prompt expert, it looks like we need to start giving agents rules for when to stop, and ensuring that they remain within the LLM’s context window. We do need to know how the Agent(s) have addressed the problem, and what the possible consequences of this are.

It is human nature to trust things if the out come looks correct. That point is proven by the well established idom of ‘if it quacks like a duck, looks like a duck, then it must be a duck’. This means we’re at risk of trusting the AI has got the solution correct. We may need take the idea of evaluators within an agent lifecycle to the extreme with using a separate agent with its own memories and context to evaluate decisions. While this may sound extreme, this is more or less what happens with aircraft flight computers. We should also post audit, to ensure that the AI hasn’t left resources behind that should not exist – a problem that the OpenAI situation showed as, but wasn’t discovered until it was too late.

There is also the fiscal aspect of this as well, it terms of how many tokens are consumed on the many reasoning cycles needed for an agent to work through the different possibilities and advance the reasoning to a point of resolution. Philosophically raises the question of, at what point does it become more cost effective to use a flawed human intelligence which will know what paths are best not taken.

Ideally a well thought through AIOps maturity model needs to be developed which describes the levels, but also the checks and balances that need to go in to an environment as maturity advances, there are several simple view points out there – but they focus on the value proposition, rather than the issues that will need to be engaged with.

References

  • OpenAI – HuggingFace teechnical report
  • Expanding LLMs responsibly
  • The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale
AI Ops

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OpAMP implementation update

15 Monday Jun 2026

Posted by mp3monster in Fluent Observability, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, development, Fluent Bit, LLM, MCP, OpAMP, open telementry, Technology

So we’ve been busy working on our OpAMP solution. We’ve made a number of enhancements since we labelled the code v0.4 back in April. For this post, we’ll look at the features added and where we’re looking next.

For a start, we demo’d what has been developed as part of CNCF Webinar about OpAMP

There has been a lot of feature development, particularly in support of working with Fluent Bit (and to a degree Fluentd).

Standalone or OpAMP server plugin for editor

We started building the Configuration Editor as a standalone capability. The thinking was that we would refactor it into the OpAMP server once we were happy with the functionality and had progressed far enough. But, as we saw this come together, it occurred to me that both deployments are good, as part of the OpAMP server, seeing the configuration being used is handy, but having a freestanding editor (without worrying about the connectivity to communicate with agents) is also a real use case.

So we have created the setup, where if the OpAMP is told to look for the Editor via the definition of Python endpoints in the configuration and it is deployed, it will be incorporated into the server. If the editor isn’t provided (deployed, or not identified in the configuration it isn’t offered in the navigation.

Config Editor view with collapsible segments

The editor is completely configuration-driven through JSON, so it only requires extending the JSON to support custom plugins or to enhance the validation rules (e.g., adding a REGEX to how a particular parameter is set). This far outweighs the current Dry run checks.

Configuration that drives the UI and its validation logic for the Editor

The configuration is considerable, so we refactored the structure to make it far easier to work with, and created some code that can mine Fluent Bit’s GitHub to generate an initial clean set of JSON docs. Of course these need an eyeballing to ensure they’re correct.

Catalog viewer

The catalogue viewer is a natural extension of the editor and leverages the way we tag configuration files with version details in the editor. The catalogue viewer has a configuration which tells it where to look for candidate files. These are then listed with the metadata.

The catalogue viewer presents all the identified files, which, when selected, if it knows about the config editor, will open the editor with the file. Otherwise, it opens a simple view of the file.

The catalogue viewer works in the same way as the editor in terms of authentication.

CLI

A Command Line tool maybe and odd choice of feature, but we found ourselves creating more and more scripts to support Windows and Bash shells with commonality. So we elected to leverage some frameworks to help reduce the ongoing effort required to maintain them. The utility can be used to generate the command, if you wanted embed the process of starting or stopping a process into the host OS.

OpAMP CLI with its auto complete capabilities

Agent as Supervisor or observer

The OpAMP documentation suggests that the agent logic is embedded, or wrapped with a supervisor, with the inference that the agent forks the application process. This means that introducing the OpAMP would be invasive. There is a noninvasive variant of the Supervisor that we’ve called Observer. Here, the agent know how to identify the key process by examining the host’s processes. Then tasks such as restarting require an understanding of how to get the OS to trigger, for example, if the service is known to init.d in Linux, we can use the service command.

Architecture view

OpAMP solution - building blocks and their relationships

MCP and Slack for ChatOps

We’ve got the Slack foundations progressed, so we can use natural language VIA langgraph to then work with MCP, which exposes a subset of API capabilities. We’ve developed both the main server and the Broker to support the MCP endpoints, with the Broker acting as a proxy to the regular Server APIs. This means if we want the MCP to be usable from outside our network, then we can separate the Broker and Agent into separate networks so that the wider set of endpoints the server provides aren’t exposed.

But if you don’t want that the MCP endpoint can be switched on.

What next …

Config deployment

We have started to address the foundations of managing configuration deployment (for example, if our Catalogue Service is used with the Server, it can be used to select the configuration files to deploy.

Part of config deployment is understanding what is deployed, so we have developed some strategies for versioning Fluent Bit and Fluentd, and I think it will translate to other configuration files (at least into the Observability space).

What we haven’t done is fully implement the process. So we’ll focus on that

E2E Testing

There is a lot of functionality here now, including tests with Playwright, but we really need to extend it to provide end-to-end tests in a clean environment that is set up from the various pip and wheel files.

Deployment Artefact access

We also want to start making these artefacts easier to retrieve, such as pulling the Wheel or PIP files from GitHub or PyPi.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit configuration quick reference and editor

27 Wednesday May 2026

Posted by mp3monster in Fluent Observability, Fluentbit, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, docs, Fluent Bit, Fluentd, GitHub, MCP, Music, OpAMP, youtube

Fluent Bit’s documentation is spread across many static web pages; in some cases, attributes allowed for a plugin are documented across several pages. There is absolutely nothing wrong with this. Having written a book on Fluent Bit, I can say it makes sense, and if the standard docs went into further detail, it would increase the spread of content.

The problem is, once you’ve got a grip on Fluent Bit, you want a quick reference just to check the attribute names or confirm that an attribute does what you expect.

The good news is that’s what we have created …

  • Quick references
    • 3.2.10 HTML, Markdown
    • 4.2.4 HTML , Markdown
    • 5.0.4 HTML , Markdown

Each quick reference page has a section at the top that provides a comma-separated list of all the plugins for the different operations, inputs, outputs, etc., with anchor links to that section. Each plugin has a table that lists all configuration attributes, including those inherited from the Fluent Bit kernel and those introduced by extending another plugin.

Why and how …

The “why” may prove even more appealing. As part of our OpAMP project, we wanted to make it easy to edit and validate Fluent Bit and Fluentd configurations before deployment using the OpAMP tooling. That tool became the config-service part of the OpAMP repo, and can be independently deployed as well as function as part of the main OpAMP service. As a result, the UI offers the same authentication and authorisation options, ranging from running without authentication to using OAuth.

This shows the editor with all the different configuration sections collapsed.
All the different editable components collapsed
The editor UI showing part of the plugin config feature
Plugins part of the editor

The key to both the UI and documentation is the use of JSON Schemas, as they contain all the information needed to create documentation just as easily as they power the UI. We have therefore generated a markdown page using a Python script. This means the docs are easy to check (compared to an HTML file) and can be rendered within GitHub. We could extend the script to generate HTML. But rather than trying to remember to keep both document types in sync (and double-check formatting), we found a JavaScript tool (marked.js) that performs an on-the-fly transformation that, as long as we stay within the core Markdown syntax, won’t cause any issues. We’ve then enriched that output a bit by applying stylesheets.

To come…

At the moment, we have only covered a subset of Fluent Bit versions, so we will, in due course, expand to cover more versions. Today, this is just the last version from each major release. We haven’t done every version to date as it does take a lot of effort to go through the documentation to generate and check the schema, and we’re still refining things as we enhance the UI.

We are also looking to do the same for Fluentd, though this is much trickier, as the portfolio of plugins that make up the core is smaller than those incorporated into the core of Fluent Bit, and the leveraging of Ruby’s dynamic behaviour makes it very easy for people to offer plugins separately. Then, of course, there is the task of collating all that information.

Resources

  • Quick references
    • 3.2.10 HTML, Markdown
    • 4.2.4 HTML, Markdown
    • 5.0.4 HTML, Markdown
  • OpAMP landing page
  • GitHub home
  • Fluent Bit
  • Fluentd

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Vibe Coding, a book review

16 Saturday May 2026

Posted by mp3monster in AI, Books, development, General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, book, development, LLM, review, Technology, vibe coding, Vibe Engineering

I’ve written a bit about AI in the development process; this has been driven largely by my own experiences, colleagues’ experiences, and blog content from people I trust. So I thought it would be worthwhile to validate my perspectives against those who are more in the know on the subject. So here is my review of the book Vibe Engineering by Tomasz Lelek and Artur Skowroński.

The book opens with a very clear differentiation between vibe coding and vibe engineering (which approximates to what I’ve previously called AI-assisted development). Not only are the key conceptual differences outlined, but the consequences of vibe coding into production are also really driven home …

teams that skip the transition from prototype to engineered artifact consistently report higher defect density, longer incident resolution, and faster architectural decay

The book also shares some real horror stories of blindly trusting LLMs, particularly in operational contexts.

The crucial challenges of vibe coding beyond ideation, PoC, and possibly MVP are brilliantly distilled. Code will do something, but is it right? Is it safe? Will it scale? Can we maintain it?

Tomasz and Artur outline a form of debt called trust debt. Where we have trusted the LLM, and it accumulates issues, particularly with NFRs that are not managed and paid down, it will seriously bite, just as tech debt does. The difference is that tech debt is more readily appreciated and generally easier to understand.

debt is a direct byproduct of the dump-and-review culture. This approach uses AI to generate a large slab of code, opens a pull request, and implicitly offloads responsibility for verification to the reviewer. It’s classic diffusion of responsibility: the presence of the AI (“the model wrote it”) and a reviewer (“someone will check it”) dilutes the author’s ownership of quality

Current approaches to this kind of development can very easily lead to the issues that Human-Machine Interface researchers talk about as automation complacency and the out-of-the-loop problem

The book also highlights interesting parallels, such as those in autonomous vehicle accidents. The consequences may not be as spectacular or as tragic (today), but they can be just as harmful, given that code affects every little aspect of our lives and the decisions we make. It is only a matter of time before it is influenced by vibed code. How long before pressure and a failure to comprehend vibe coding vs vibe engineering creeps into mission-critical development?

Once the consequences and challenges are called out, the book takes us on a journey to illustrate how to better approach vibe development, specifically through defining what a successful outcome should be. The brilliantly simple thing here is that the two approaches are demonstrated with multiple different LLMs using the same prompt.

While the book provides brilliantly illustrated proofs for how to better approach vibing (moving from coding to engineering), Tomasz and Artur point out that this alone is not enough; we need to lean into broader process improvements and leverage good engineering practices.

This first chapter then sets everything up that follows, taking on a journey of re-engineering a solution. Illustrating how to prompt to extract from an existing solution the details that can then be fed as prompts to generate a new solution.

The narrative progresses through considerations such as context compression, then leverages tools to enable the LLM to take on significant tasks, such as UI design, by giving it the information it needs to work out and create React components with a consistent look and feel.

The books reveal some really good ideas that allow things to be developed far more efficiently, for example, rather than expecting the LLM to scan through code, exposing the Language Server, which provides a lot of today’s IDE smarts, such as navigation through the call chain in an application. Exposing the LSP as an MCP tool offers the LLM an efficient and more reliable approach to analysing code.

If you want to follow along and test the points that the book makes, you’re not going to need to fork out masses on LLM tokens; the authors are very clear that the cost to repeat the exercises can be done within free/trial service tiers.

Conclusion

I don’t want to spoil your enjoyment of reading the book by revealing its secrets here. But there is a lot of great content, which means that, with some adjustments to how the LLM is prompted and some setup, it becomes possible to significantly reduce that trust debt.

If you’re heading down the road of vibe-based development, I would highly recommend digging into this book. We’re already making some further refinements to our processes. The changes needed to transition from vibe coding to vibe engineering won’t be shocking to those with a software engineering background. But their adoption is likely to pay back significantly.

Vibe Engineering by Thomasz Lelek & Artur Skowronki cover

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
← Older posts
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d