• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Monthly Archives: October 2026

Are we heading to a Certificate-apocalypse?

29 Thursday Oct 2026

Posted by mp3monster in development, General, Security, Technology

≈ Leave a comment

Tags

ACME, AI, certbot, certificates, CLM, Cloud, CMC, CMP, cybersecurity, SCEP, Security, Technology, TLS

Most engineers are probably aware of the shortening life span of TLS certificates. this year certificate maximum life drops from 398 to 200 days. But what may not have registered is that the roadmap sees the max certificate time will continue to half every year. This means that by 2029 certificate lifecycle will be down to 47 days (31 days (core duration) + core duration/2 days + 1 day). But before 2029, in less than six months we’ll see the current certificate durations half from 200 to 100 days. As with the 2026 change, the new limits come into effect on the 15th March. Which does create an odd situation where certificates issued on the 14th March will outlast certificates issued on the following day.

The addition of half the core duration period is to allow for the recovery from failures. as of 2029 the expectation is every month the certificates should be rotated.

Manual processes are unsustainable

This means that the rotation of certificates has got to be accelerated to be monthly, a velocity that is simply not practical to be done by hand. This means the deployment of certificates really needs to be automated. There are a number of frameworks and standards that support the distribution of certificates.

Perhaps the best known and leading solution has been ACME protocol working in conjunction with Let’s Encrypt CA. The least known is the use of OpAMP also supports certificate distribution. The following table shows the different standards that specifically address certificate management (CLM), some implementations are linked to a Certificate authority , others are not.

StandardSpecificationOpen-source implementationsCommercial implementations
ACMERFC 8555Certbot (client), acme.sh (client), lego (client), step-ca (server), Boulder (server)DigiCert, Sectigo, GlobalSign, EJBCA Enterprise
CMP / CMPv3RFC 9810EJBCA Community, Bouncy Castle (library)EJBCA Enterprise, DigiCert Trust Lifecycle Manager
ESTRFC 7030Cisco libEST (client/server library), EJBCA Community (edition-dependent)EJBCA Enterprise, DigiCert
SCEPRFC 8894MicroMDM SCEP, EJBCA CommunityMicrosoft Intune, EJBCA Enterprise, DigiCert
CMCRFC 5272Dogtag PKICommercial implementations are less prominent; verify explicit CMC support and supported operations with individual PKI vendors

To understand the differences between the different standards, it is worth looking at the standards from the perspective of the operations that are needed to manage certificates.

FunctionACMECMPESTSCEPVendor
CLM API
Certificate issuanceYesYesYesYesUsually
Certificate renewalYesYesYesYesUsually
Certificate revocationYesYesNo native operationLimitedUsually
CA certificate rolloverLimitedYesYesLimitedVaries
Central certificate inventoryNoNoNoNoUsually
Ownership / application mappingNoNoNoNoVaries
Policy managementLimitedLimitedLimitedLimitedUsually
Application deploymentNoNoNoNoVendor-specific
Application activationNoNoNoNoThrough integrations
Deployment verificationNoNoNoNoVaries

You’ll note that the deployment, activation and verification steps fall into the domain of vendor solutions, and are variable. However we can address such considerations in an open-source (or commercial) manner through tooling such as Puppet, Chef and certain payload types provided by OpAMP.

These CLM options aren’t the end of the story as you may wish to look more broadly at no just the request and distribution of certificates, but the mechanisms through which the certificate is stored, and there is an overlap with such management with keys as well as certificates. For example APIs may use keys of certificates to help verify authenticity of the request. This also extends to mTLS operations where the client needs certificate to verify the server requests. For example KMIP provides an interface that can support cryptographic object (keys, certificates (public, private, symmetric), secret data, PGP Keys).

StandardSpecificationOpen-source implementationsCommercial implementations
KMIPOASIS KMIPPyKMIP (Python client/server)HashiCorp Vault Enterprise, Thales CipherTrust
PKCS#11PKCS#11 v3.1SoftHSM, OpenSCHashiCorp Vault Enterprise, Thales Luna HSM
SCIMRFC 7644elimity SCIM (Go library)Microsoft Entra ID, Okta

Managing the impact

There are several impacts that need to be considered:

  • tracking certificate expiry
  • Last mile from client deployer to server folder and any refresh nudge required

Tracking certificate expiry

Traditionally certificate expiry management could be handled as a manual process – after all if you have a log of which systems had certificates then if you simply reviewed the list once a month, you could spot expiries with time to spare. Even if the certificate distribution process was automated across many nodes. This just isn’t going to work, and the contingency time incorporated into the certificate life is contracting. Rather than trusting to a tracker spreadsheet being managed we should really focus on actual certificate data. This points to having a means by which application observability has a reporting metric on when a certificate will expire (date or a number of days). The later will prove to be easier as a metrics measure, but the value needs to be in context of when the metric was collected.

Last mile challenge

While automation frameworks like Certbot and step-ca can drive the certificate issuance. These processes get the certificate to the relevant server. But we still need to move the certificate intro the location the application expects the certificate to be, and do we need to nudge the server to reload the certificate? Certbot for example expects to have a script to trigger for this action. This can get a little messy to deploy and maintain, particularly if there are subtle variances in deployments (as a result of manual application deployment in legacy environments), configuration creep etc. One possible way to simplify this is to leverage the capabilities of environment scanners to locate apps and their configs. Then use that to drive the auto creation of that last mile.

Why the change?

The history for the reduction has largely been driven by browser providers particularly Google and Apple, and their security concerns about the trustworthiness of the web’s public key infrastructure (PKI). Historically, certificates could remain valid for several years, creating a prolonged window of exposure if a private key was compromised, a certificate incorrect certificates being wrongly used (organization or domain information became outdated – fore example domain name changes, acquisition of services etc). Certificate revocation mechanisms, particularly CRLs and OCSP, have not proven to be reliable as a means of ensuring browsers quickly stop trusting compromised certificates. Shorter lifetimes reduce this exposure and encourage the automation of certificate management.

The driving of changes has shifted from browser providers unilaterally declaring they won’t trust certificates that are older than a certain age, to a more formalized process governed primarily by the CA/Browser Forum, (an independent body bringing together Certificate Authorities and browser vendors).

Useful resources

  1. A/Browser Forum — TLS Baseline Requirements, version 2.3.1 (4 October 2026), section 6.3.2. Authoritative current hard limits: 398 days before 15 March 2026; 200 days from that date; 100 days from 15 March 2027; 47 days from 15 March 2029. Section 1.1 and the revision history identify 1 July 2012 as the original effective date. https://cabforum.org/working-groups/server/baseline-requirements/requirements/
  2. CA/Browser Forum — FAQ for Baseline Requirements. Historical summary of the 60-month limit from July 2012 and the 39-month general limit from April 2015. This is a historical source, not the current rule set. https://cabforum.org/faq-about-the-baseline-requirements/
  3. CA/Browser Forum — Ballot 193: 825-day Certificate Lifetimes (17 March 2017). Establishes the March 2018 transition to 825 days; also records the preceding 39-month rule and the qualifying 60-month legacy exception through June 2016. The page contains amendment markup; read the replacement wording in context. https://cabforum.org/2017/03/17/ballot-193-825-day-certificate-lifetimes/
  4. Apple — About upcoming limits on trusted certificates. Announces the 398-day maximum for public TLS server certificates issued on or after 1 September 2020 and explains its scope. This change originated in browser/platform trust policy. https://support.apple.com/en-us/102028
  5. CA/Browser Forum — Ballot SC081v3: Introduce Schedule of Reducing Validity and Data Reuse Periods (11 April 2025). Records the successful vote for the 2026–2029 reductions; source 1 confirms the resulting current requirements. The April milestone is the vote result, not the end of the subsequent intellectual-property review period. https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/
  6. IETF Datatracker — draft-barnes-acme-00 (28 January 2015). First revision of the ACME Internet-Draft, before working-group standardization. https://datatracker.ietf.org/doc/draft-barnes-acme/00/
  7. Let’s Encrypt / ISRG — Entering Public Beta (3 December 2015). Announces public availability without invitations. https://letsencrypt.org/2015/12/03/entering-public-beta
  8. Electronic Frontier Foundation — Announcing Certbot: EFF’s Client for Let’s Encrypt (May 2016). Explains the rename, the EFF home, and ACME-based issuance and configuration automation. https://www.eff.org/deeplinks/2016/05/announcing-certbot-new-tls-robot
  9. Let’s Encrypt community — New name (Certbot) for Python Let’s Encrypt client (12 May 2016). Dated contemporary announcement linking to EFF’s announcement; supports the exact event date. https://community.letsencrypt.org/t/new-name-certbot-for-python-lets-encrypt-client/15688
  10. RFC Editor — RFC 8555: Automatic Certificate Management Environment (ACME), March 2019. Standards-track specification, published as a Proposed Standard. https://www.rfc-editor.org/info/rfc8555/
  11. ACME — RFC 8555: Automated Certificate Management Environment; supports automated certificate issuance, renewal and revocation.
  12. CMP / CMPv3 — RFC 9810: Certificate Management Protocol; supports PKI enrolment and certificate lifecycle operations.
  13. CMP over HTTP — RFC 9811: Defines HTTP transport for CMP messages.
  14. EST — RFC 7030: Enrollment over Secure Transport; supports certificate enrolment and re-enrolment over HTTPS.
  15. SCEP — RFC 8894: Simple Certificate Enrolment Protocol; commonly used for network devices.
  16. CMC — RFC 5272: Certificate Management over CMS; supports certificate requests and management messages. Updated to — RFC 10002: Updated specification superseding RFC 5272.
  17. KMIP 2.1 — OASIS KMIP 2.1 Specification: Key Management Interoperability Protocol; manages cryptographic keys, certificates and other cryptographic objects.
  18. PKCS#11 3.1 — OASIS PKCS#11 3.1 Specification: Cryptographic Token Interface; standard API for accessing cryptographic hardware and software tokens.
  19. SCIM 2.0 — RFC 7644: System for Cross-domain Identity Management; supports identity and account provisioning rather than certificate management.
  20. SPIFFE — SPIFFE Specifications: Secure Production Identity Framework for Everyone; standardizes workload identity across heterogeneous systems.
  21. X.509-SVID — X.509-SVID Specification: Defines X.509 certificates used for workload identity and authentication.
  22. X.509 PKIX — RFC 5280: Defines the Internet X.509 public key certificate and Certificate Revocation List (CRL) profile.
  23. CA/Browser Forum Baseline Requirements — Current Baseline Requirements: Industry requirements governing publicly trusted TLS certificate issuance, domain validation, security practices and maximum certificate validity periods.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Can patches have functional impacts?

27 Tuesday Oct 2026

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, cybersecurity, languages, OS, patches, patching, Security, Technology, vulnerability

With the recent upheaval from the newly discovered PHP vulnerability (often referred to as PHP2Shell), and updates to OpenTelemetry declarations, it struck me that it’s worth highlighting an assumption we all tend to make with patches.

We tend to think patches introduce fixes or defences that mean functional behavior is immutable. It is an understandable reading based on definitions such as Semantic Versioning (semver):

PATCH version when you make backward compatible bug fixes

However, languages and frameworks point towards a different model, where defined API behaviour will be respected.

The bottom line is if you’re exploiting undocumented behaviors, it makes it all the more necessary to test and ideally in an automated manner changes not only in the applications but also for the stack.

If you are taking advantage of undocumented behavior, then it is worth adding an abstraction layer to allow mitigation changes to be added.

The following identifies the statements at a language level relating to what impact a patch can have:

Language / runtimeBehavior changes in patchesBreaking API changes in patchesPolicy, limitations and supporting documentation
GoYesExceptionalExplicit security, unspecified-behavior and specification-correction exceptions. Go compatibility promise
PHPYes, for correctionsGenerally noSecurity and bug fixes permitted; public userland API must remain backward compatible. Internal ABI/API has narrowly defined exceptions. PHP release policy
Python (CPython)Yes, restrictedExceptionalReasonable bug fixes may affect behavior; stronger incompatibilities require exceptional justification. PEP 387
Java (OpenJDK / Oracle JDK)YesGenerally noSecurity corrections can alter runtime behavior; source, binary and behavioral compatibility are evaluated separately. OpenJDK compatibility
C# / .NET runtimeExceptionalExceptionalBreaking changes generally prohibited during servicing, except for security vulnerabilities or other critical issues. .NET release policy
F# / .NET runtimeExceptionalExceptionalShares .NET runtime servicing policies; compiler and language tooling have their own compatibility considerations. .NET compatibility rules
VB.NET / .NET runtimeExceptionalExceptionalSame underlying .NET servicing guarantees. .NET servicing
TypeScriptYes, restrictedNot normallyPatch releases may contain high-priority regression, performance-regression and safe language-service corrections. Ordinary breaking changes are targeted at larger releases. TypeScript release process
JavaScript (ECMAScript)Not applicable at language-specification levelNot applicableECMAScript is a specification, not a separately patched runtime. Behavior depends on implementations such as V8, SpiderMonkey, JavaScriptCore and Node.js. ECMAScript specification
Node.jsYes, for correctionsExceptionalPatch releases accept fixes; semver-major changes are excluded under normal procedures. LTS security exceptions can affect native API/ABI compatibility when unavoidable. Node.js release procedures

The following table, takes the same approach looking at how operating systems view changes for patches:

Operating systemBehavior changes in maintenance patchesBreaking API / ABI changesPolicy, limitations and supporting documentation
Red Hat Enterprise Linux (RHEL)YesGenerally avoidedBackports security and bug fixes into supported versions. Red Hat explicitly tests for unwanted side effects, so this is not general permission to break compatibility. RHEL security backporting
SUSE Linux Enterprise Server (SLES)YesGenerally avoidedMaintenance includes security corrections and defect fixes; individual components have different lifecycle rules. Service packs allow broader changes. SUSE support policy and SLES release notes
Oracle LinuxYesNo general permission establishedOracle distributes security, bug-fix and enhancement errata as package updates. Security hardening can alter functionality independently of major OS releases. Oracle Linux errata policy
Ubuntu LTSYes, cautiouslyGenerally avoidedBackported security patches prioritize compatibility. Exceptional changes and certain feature updates can be accepted under the Stable Release Updates process. Ubuntu security updates
Debian StableYes, restrictedGenerally avoidedImportant bug and security fixes are acceptable; updates are deliberately minimal and require review. Debian Stable update rules
Microsoft Windows 11 / Windows ServerYesNo general permission establishedCumulative security, quality and out-of-band updates may change functionality. Windows 11 may also deliver feature enhancements through monthly servicing. Windows update release policy
Apple macOSPossibleNo general permission establishedSecurity and maintenance releases can affect operating behavior, but there is no general published patch-version exception equivalent to Go’s. Apple security releases

Often when building solutions we don’t see low level frameworks and how they work, so we may not know about any undocumented feature exploitation.

The challenge it can bring

This points to increased testing, which ideally is automated. The reaction is initially is so what. But, when vulnerabilities like PHP come up which had openly triggered systems being disconnected from the net.

So extending testing (even if automated) is going to experience resistance from the commercial parts of an organization who are unlikely to understand the nuances, but will react to headlining risks.

How do we move forward?

There are some strategies we can adopt that can help us. The industry used to talk about security in depth and, more recently, system resilience. Regardless of terminology, we want to introduce control layers that reduce the chance of malicious intent. Often referred to as compensating controls.

For example, if the vulnerability is overloading the runtime by sending an oversized payload, a firewall or proxy layer can have rules that prevent payloads from being too big from hitting the backend.

If the networking allows it, then segmenting the network reduces the risk of a breach affecting other environments.

This doesn’t mean if we can patch, that we shouldn’t. What it does mean is we can be considered and not knee-jerk panic and unwittingly introduce new problems.

Conclusion

Adding additional security and proxy layers and segmenting system environments may seem like overkill, particularly when introducing new systems. But these steps create the means to be far more resilient.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OpAMP demo

20 Tuesday Oct 2026

Posted by mp3monster in Fluent Observability, Fluentbit, Fluentd, General, OpAMP, Technology

≈ Leave a comment

Tags

CLI, conference, configuration, demo, Fluent Bit, FluentBit, Fluentd, LLM, logging, monitoring, observability, OpAMP, Technology, video

Last week saw the Open Source Observability Day (OSOD) during which I presented on the subject of the Open Agent Management Protocol, covering the underlying protocol that forms the foundation of a wider capability we have been building along with a demo of what is available.

If you want to see the presentation, and hear the full story not only of the core protocol but how it can help with the ideas around ChatOps and AIOps head over to the OSOD website (and previous years can be seen on YouTube with playlists for each year e.g. OSOD 2025). Of course if you have been following my blog posts you’ll know this.

The demo is recorded against a version prior to the 0.6 release.

Running the demo yourself

If you want to try this out for yourself, you can do so. You’ll need to:

  • Deploy the OpAMP services,
  • Setup the connectivity to Slack, and AI if you want to explore the chat ops capabilities.
  • Deploy the relevant agents (we’ve go something in the pipeline to simplify this)
  • Download the demo configuration and adjust based on your deployment locations.

Get the configuration paths and security tokens for Slack and AI, and firing up the demo is very simple using the CLI.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Moving Open Source Observability closer to turn-key solutions

17 Saturday Oct 2026

Posted by mp3monster in Fluent Observability, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, automation, chatgpt, configuration, Fluent Bit, FluentBit, Fluentd, LLM, logging, monitoring, observability, OpAMP, Open Telemetry, OTel, Technology

There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:

  • Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
  • Deployed services and applications with default monitoring configuration
  • Prebuilt dashboards

While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.

This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.

Detecting and Generating Configuration

The process we need has the following steps:

  • find the deployed solution(s)
  • Understand how they are deployed
  • Generate a configuration based on the detection
  • Push the configuration out to agents

Detect and capture config

One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.

While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …

As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.

Templates make the result reusable

By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.

There are limits

So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.

Convenience as a weakness?

While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.

Backend Bootstrapping

Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.

Useful Links

  • OSQuery homepage and GitHub
  • Jinja2 Doc.
  • Primer on Jinja templating

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OpAMP V0.6 implementation released

12 Monday Oct 2026

Posted by mp3monster in Fluent Observability, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, chatgpt, development, LLM, observability, OpAMP, release, Technology

We’ve made the latest version of our implementation of the OpAMP specification available. It has taken longer to ready than the previous versions, but there is a lot been included, both in the core functionality, but also capabilities supporting the use and setup. Here are the headlines…

  • The project has its own organization now (https://github.com/orgs/opamp-observe/) as mentioned here, but also a domain of https://opamp.info/ which will bounce you to the GitHub pages.
  • Automated regression from clean to deployed and running a series of end-to-end tests as well as producing a guaranteed fresh set of artefacts has been implemented. This can be run against AWS (currently in use) or Azure (needs refining). These are triggered on changes to the main branch.
  • GitHub pages with links all the documentation are established now.
  • The Python wheel release artefacts are no incorporated into the release assets.
  • The configuration for the Open Source Observability Day conference demo is available in the https://github.com/opamp-observe/demo-repository repo.
  • We’ve added support for a couple more open-source observability tools:
    • Vector
    • Elastic Agent
    • Elastic Heart beat
  • The client/consumer side has been refactored so it is easy to build your own observability agent controls and independently deploy them (not supported by the config editor).
  • The CLI has been enhanced so it can help quickly spin up more advanced configurations – helpful for demos and real deployments.
  • The Dev CLI has been improved.
  • The server side now has a basic client side (Supervisor/Observer) configuration editor UI. Which can be deployed as a plugin.
  • Improvements around the use of Podman and Docker for the containerization.
OpaMP v0.6 now labelled

What next?

We’ve added to the documentation pages for the roadmap, initially we’re working to fill out the regression test sets at the moment. Other possibilities:

  • Expand the client types
  • Extend the client types to exploit more features.

Perhaps the most interesting idea, we’re thinking about is can we exploit tools like Netdata to discover what solutions are deployed and generate a requirements/configuration definition which can be used to generate a suitable agent configuration file.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Observability as a security sensor using Fluent Bit, OpenTelemetry and AIDE

09 Friday Oct 2026

Posted by mp3monster in Fluent Observability, Fluentbit, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, cyber-security, cybersecurity, Security, Technology

Observability is usually looked at from the perspective of applications and their needs and the cloud native platforms. Nothing wrong with that, but it is also a rather narrow view of the telemetry we are already collecting or are very capable of collecting. Our observability agents (Fluent Bit, Fluentd, Elastic Agent …) can collect signals (logs, metrics and traces) that can contribute to the security operations within an enterprise. More importantly, the same open-source tooling can help answer a deceptively simple question such as has someone changed the system, or changed the tooling that tells us what the system is doing?

When it comes to security, that second part matters. An attacker does not always need to exploit a new vulnerability. If they can alter a service definition, scheduled task, access-control file, logging configuration or telemetry destination, they can prevent us from sensing what damage they maybe doing, including opening up simpler or more invasive routes into your systems. MITRE ATT&CK describes this kind of action as a defence-evasion technique. Research from Palo Alto Networks Unit 42 has also described cloud logging services as an attack surface, including changes that stop, redirect or selectively reduce audit collection. Not only that getting at logs, means they can potentially determine when systems that are running, and find new attack vectors.

As you can see being able to detect configuration and file manipulation is an important factor for security. Many may react to this with concern about introducing cost for expensive security tools, but if there are simple means to listen to OS activities that relate to file manipulation then we can leverage our observability estate for this aspect of security.

The wider open-source stack

That neutrality for our observability agents is valuable for security. A host or cluster can emit operational and security telemetry through the same managed pipeline, while routing data to one or more destinations such as OpenSearch, Loki, ClickHouse, a SIEM, object storage or an OTLP-compatible service. Not to mention if the event is for an application configuration, we can evaluate it against application and service viewpoints such as managing configuration drift.

The roles are complementary:

  • AIDE is a open-source Linux utility that provides the file-integrity check. It builds a known-good database and reports additions, removals and changes to selected files and attributes. Aide achieves this through the use of the iNodes API, which if you preferred could also be tapped into even more directly.
  • Fluent Bit or other OTel compliant tools collect the result close to the workload, adds context, buffers during a network interruption and routes the event. Our backend services which are in all likelihood OTLP capable, means the events can be send nearly anywhere for analysis and alerting
  • The analytical backend correlates the integrity event with logins, process starts, deployments, traces and other activity, then drives an alert or investigation.

AIDE is deliberately focused. It does not need to become a SIEM and Fluent Bit does not need to become a file-integrity engine. Combining small tools through open interfaces lets each component do the job it is good at.

We can use Aide not only to monitor our configurations, but when the deployed binaries are disturbed. We can extend this setup further. By exploiting custom commands with OpAMP (particularly for a supervisor) can be used to not only manage the observability agent, but also manage the deployment of configuration to tools like Aide. Now if we think something is being abused, the option to change our file integrity monitoring is easily in our reach.

Why file integrity belongs in the observability conversation

The available incident statistics do not normally include a neat category called “configuration-file manipulation”. The behaviour is spread across techniques such as creating or modifying system processes, scheduled tasks, startup mechanisms, accounts, policies and logging controls. That makes a single adoption or attack percentage difficult to defend. It does not make the risk rare.

For example, Google Cloud’s M-Trends 2025 reporting included techniques such as Create or Modify System Process (20.6%), Account Manipulation (19.9%), Scheduled Task/Job (14.3%), Registry Run Keys/Startup Folder (11.5%) and Web Shell (7.2%) among observed investigations. These are broader ATT&CK behaviours, not a count of configuration-file changes, but many are enabled or persisted through changes to system configuration.

There is also a compliance driver. PCI DSS v4.0.1 Requirement 11.5.2 explicitly requires a change-detection mechanism, such as file-integrity monitoring, for critical files, with comparisons at least weekly and alerts for unauthorised modification. NIST SP 800-53 Revision 5 control SI-7 requires monitoring for unauthorised changes to software, firmware and information. Other standards may describe the required outcome as configuration control, integrity, audit protection or system monitoring rather than naming a particular product.

FIM helps provide the evidence behind those outcomes:

  • what changed;
  • which host or workload reported it;
  • when the change was detected;
  • which attributes or content hashes changed;
  • whether a change matched an approved deployment window;
  • whether the alert was acknowledged and investigated; and
  • whether the integrity baseline was updated through a controlled process.

The last point is easily overlooked. Automatically accepting every change into the baseline produces a very tidy report and a very weak control.

What should be monitored?

Starting with every file on every host usually creates cost and noise. I would start with the files that can change control flow, identity, collection or evidence retention:

  • Fluent Bit configuration, parser files and included fragments;
  • Vector configuration and transformation definitions;
  • OpenTelemetry Collector configuration;
  • systemd units and drop-ins for telemetry agents;
  • log rotation and retention configuration;
  • TLS trust stores, client certificates and references to credentials;
  • auditd, syslog and journald configuration;
  • scheduled tasks, startup scripts and authorised SSH keys;
  • alert rules, dashboards and detection content managed as files; and
  • the AIDE configuration and baseline database itself.

Runtime state, offset databases, buffers and normal log files should normally be excluded. They are expected to change and monitoring them obscures the files that should remain stable.

There is an architectural wrinkle: if AIDE reports through Fluent Bit, and an attacker changes both AIDE and Fluent Bit, the local signal can be suppressed. The practical response is defence in depth. Restrict write access, ship events off-host quickly, monitor the agent configuration and service unit, protect or sign the AIDE baseline, and alert when expected heartbeats or scheduled check results disappear. Silence is also a signal.

Example: AIDE reporting through Fluent Bit and OTLP

The following is a starting point rather than a universal production configuration. Package paths and AIDE defaults differ between Linux distributions, so it should be merged with the supplied configuration rather than replacing it blindly.

First, add an AIDE rule group for the observability configuration. For example, in /etc/aide/aide.conf.d/observability.conf:

# Permissions, inode, links, ownership, size, timestamps and SHA-256 content.
OBS_CONFIG = p+i+n+u+g+s+m+c+sha256
​
/etc/fluent-bit                         OBS_CONFIG
/etc/vector                             OBS_CONFIG
/etc/otelcol-contrib                   OBS_CONFIG
/etc/systemd/system/fluent-bit.service.d OBS_CONFIG
/etc/systemd/system/vector.service.d     OBS_CONFIG
/etc/systemd/system/otelcol.service.d   OBS_CONFIG
/etc/aide                               OBS_CONFIG
​
# These are runtime data and should not be treated as static configuration.
!/var/lib/fluent-bit
!/var/lib/vector
!/var/lib/otelcol
!/var/log

The main AIDE configuration can direct its report to a file that Fluent Bit is permitted to read:

database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout=true
​
report_url=file:/var/log/aide/aide.log
report_level=changed_attributes

Initialise and approve the baseline using the procedure provided by the operating-system package. The baseline database should be writable only by the account performing the controlled update; consider retaining a signed or read-only copy outside the host.

Run aide --check from a systemd timer or the organisation’s scheduler. An hourly check provides better detection latency than the weekly minimum stated by PCI DSS, although the right frequency depends on estate size, I/O cost and risk. The scheduler should also emit a success or failure event so that a missing check can be detected.

Fluent Bit can then tail the report, add security context and forward the records over OTLP. The Tail input uses a database to retain offsets, and the OpenTelemetry output supports OTLP/HTTP and OTLP/gRPC.

service:
flush: 5
log_level: info
storage.path: /var/lib/fluent-bit/storage
pipeline:
inputs:
- name: tail
tag: security.fim.aide
path: /var/log/aide/aide.log
key: body
path_key: log.file.path
db: /var/lib/fluent-bit/aide-tail.db
db.sync: normal
read_from_head: false
refresh_interval: 5
rotate_wait: 30
skip_empty_lines: true
skip_long_lines: true
storage.type: filesystem
filters:
- name: modify
match: security.fim.aide
Add:
- event.domain file
- event.category file
- event.type change
- security.control file_integrity
- service.name aide
outputs:
- name: opentelemetry
match: security.fim.aide
host: otel-collector.internal
port: 4318
logs_uri: /v1/logs
tls: on
tls.verify: on
compress: gzip

This example emits each AIDE report line as a log record. That is simple and robust, but the backend will need to group lines belonging to the same check. For a more mature deployment I would place a small wrapper around the AIDE invocation that converts each detected change into JSON Lines and adds a check identifier, hostname, baseline version and execution result. Fluent Bit can then use its JSON parser and every changed file becomes a structured event.

Vector can implement the same pattern with its file source and an OTLP-capable destination or intermediate Collector. The OpenTelemetry Collector can also collect files using the contributed filelog receiver. The choice is less important than applying consistent fields, reliable buffering, secure transport and an independently managed destination.

From event collection to a security control

Collecting the AIDE output is only the first half of the solution. To turn it into a control, the backend needs useful detection logic and operational ownership.

I would start with alerts for:

  • a change to an observability agent destination, filter or exclusion rule;
  • a change to a telemetry service unit or executable;
  • deletion or replacement of the AIDE database;
  • changes outside an approved deployment window;
  • a burst of changes across several hosts;
  • a configuration change followed by a fall in log volume; and
  • a missing AIDE run, agent heartbeat or expected integrity summary.

Correlating the FIM event with deployment records is important. A checksum change shortly after an approved GitOps rollout is very different from the same change following an interactive root login at 02:00. Observability provides the surrounding evidence that allows the integrity event to be assessed rather than simply counted.

There is also value in feeding an integrity event back into operational telemetry. A changed collector configuration can annotate dashboards, traces and incident timelines. The question changes from “why did the logs stop?” to “did the logs stop when this configuration changed?” That is a much better starting point for both an SRE and a security analyst.

Final thoughts

The observability and security domains have spent years building separate pipelines for data that frequently starts on the same machine. Open standards and focused open-source tools give us an opportunity to reduce that duplication.

AIDE supplies a mature integrity signal. Fluent Bit and Vector make that signal transportable and enrichable. OpenTelemetry makes the route to the wider platform portable. None of these components replaces access control, endpoint detection or a SIEM, but together they add a useful security capability to infrastructure many organisations already operate.

Perhaps the most important configuration to observe is the configuration that decides what we are allowed to observe.

Resources

  • AIDE project
  • Fluent Bit Tail input
  • Fluent Bit OpenTelemetry output
  • Vector file source
  • OpenTelemetry Collector configuration
  • MITRE ATT&CK: Impair Defences – Disable or Modify Cloud Logs
  • Unit 42: Abusing Cloud Logging Services for Defense Evasion and Visibility
  • NIST SP 800-53 Revision 5
  • PCI Security Standards document library
File Integrity Management using Aide and Fluent Bit

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    October 2026
    M T W T F S S
     1234
    567891011
    12131415161718
    19202122232425
    262728293031  
    « Sep    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d