• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Security

CVE publication trend and how are known exploits are changing?

02 Monday Nov 2026

Posted by mp3monster in General, Security, Technology

≈ Leave a comment

Tags

attack, CIRCL, CISA, compensating controls, controls, CRACI, CVE, CVSS], exploits, KEV, patching, Security, Technology, trends, VulnCheck, vulnerabilities

With the explosion in the number of CVEs (Common Vulnerabilities and Exposures) being published, it begs the question as to whether the number of Known Exploited Vulnerabilities (KEV) is trending in the same direction. The natural assumption would be that there be some level of correlation. It took some exploring of data before we started to see any data trends that made sense, and threw up some interesting observations.

Before we look at what we might be able to infer from the data and the patterns we see, let’s start with the looking at the data we’ve sourced and plotted.

Data sources

  • CVE Publications — CRACI / CVE Program: Monthly counts of published Common Vulnerabilities and Exposures (CVEs), categorized by CVSS severity: Critical, High, Medium, Low and Unknown. The data originates from the CVE Program’s official records and is presented as stacked areas to illustrate changes in vulnerability volumes and severity distribution over time. Sources: CRACI CVE Publication Trends | CVE Program GitHub.
  • CISA Known Exploited Vulnerabilities (KEV): Monthly counts of vulnerabilities added to the US Cybersecurity and Infrastructure Security Agency’s catalogue of vulnerabilities confirmed to have been exploited in the wild. Counts are based on the date vulnerabilities entered the catalogue rather than when exploitation first occurred. Presented as the red line. Sources: CISA KEV Catalogue | CISA KEV GitHub.
  • CIRCL Vulnerability-Lookup: Monthly deduplicated counts of vulnerabilities recorded across multiple exploitation intelligence catalogues, including CISA, Shadowserver, ENISA, Previdian and CIRCL. By aggregating these sources, it provides broader visibility of known exploitation than any individual catalogue, although counts reflect when vulnerabilities were listed rather than necessarily first exploited. Presented as the purple line. Sources: CIRCL Vulnerability-Lookup | GitHub Repository.
  • VulnCheck KEV: Monthly counts of vulnerabilities identified as exploited in the wild through intelligence collected from security researchers, vendors, incident investigations and other publicly available evidence. VulnCheck provides an additional perspective beyond CISA, although its reporting volumes may be influenced by changes in intelligence coverage and retrospective discoveries. Presented as the thinner teal line, with the monthly figures currently treated as provisional pending independent validation. Sources: VulnCheck | Exploit Intelligence Documentation.

Understanding the data

Spikes in KEV data and varying sources

We have seen (and it is visible in our graph that the different sources of KEV data tend to show early spikes, typically triggered when the catalogue is being populated with retrospective data. We initially started looking at individual reporting sources such as VulnCheck thinking that they would give a true figure and the aggregated reporting data such as CIRCL would provide amplified patterns. The opposite appeared to be the case. We’ve kept the VulnCheck and CISA KEV reporting help illustrate this. Where Vulncheck’s spiking tend to be dramatic compared where CIRCL might show a small uptick, but then in the last quarter of 2025 we start to see opposing trends. It appears that CIRCL’s aggregation of sources has smoothed out the fluctuations. CISA’s KEV numbers in comparison a notably different in volume to VulnCheck. The CISA numbers generally follow the same pattern as CIRCL until 2026 – with the worst divergence 50 exploitations identified rather than the triple digit variance of VulnCheck.

Critical and High CVE fluctuations

If you look carefully and the CVE trends in the different categories, while there is clearly an acceleration in the last 12 months (which we could attribute to initiatives such as GlassWing) if you look carefully, you’ll notice that the growth in Severe and High scoring CVEs is growing faster. It is only two dates, but this table supports the observation:

SeveritySeptember 2025September 2026Growth
Critical2951,555+427%
High1,2756,160+383%
Medium2,3315,285+127%
Low148564+281%
Unknown2631,366+419%
Total4,31214,930+246%

The telling difference is if you compare the total growth against the growth numbers and the individual severity growth numbers. The critical numbers have multiplied by nearly 1.75.

Exploits (KEV) appear to be tracking vulnerabilities (CVE)

Perhaps the most important, point in the data is that based on the aggregated CIRCL counts, exploits are following the upward trend with a bit of lag which fits what thought we’d see. Yes at this point we’re are mindful that this could be finding data that supports a hypothesis (particularly given other individual KEV numbers may not support the trend). But on balance I’m inclined to trust the CIRCL numbers because if other numbers are erratic we’d not see such a similar upward track.

How is patching performance responding?

Keeping up with patching is likely to prove challenging, the 2026 Edgescan report picks out a rather telling observation, which supports the problems of not keeping up:

The most striking insight is that vulnerabilities from 2025, 2023, and 2024 account for more than 50% of all occurrences, indicating that newly published or recently discovered issues are the primary drivers of risk this year.

The suggestion is that the really high profile vulnerabilities get addressed, but the progress to address those that don’t get the notoriety is weaker. The alternative possibility relates not on individual exposures but when failures grab the headlines that attention turns back to patching. Another indicator that patching can’t keep up is the Verizon 2026 DBIR reported 26% of CISA KEV vulnerabilities fully remediated in its analyzed population versus 38% in the prior report, with median full remediation time increasing from 32 to 43 days (assuming Verizon’s dataset is representative).

Conclusion

While this is not an exhaustive analysis, it would be fair to treat it as anecdotal evidence to indicate:

  • CVE acceleration is being followed by KEV acceleration – possibly the most concerning.
  • If historic data shows we can’t keep up with patching, then its only going to get worse – supporting the expectation of climbing KEVs.

We do therefore need to look to adopt differing strategies to protect ourselves, and new measurements to reflect whether the new approaches are paying dividends. So what can we do differently, and I’m not saying don’t patch, I’m saying if you have a finite patching capacity, use it wisely, and invest in actions that give you the best vulnerability reduction.

Steps

Patching is by its very nature after the horse has bolted, this is why we have zero day exploits. The bottom line is there will always be vulnerabilities, the question is who finds them first and how do we react. Perhaps an exceptional illustration is the Apollo 11 Guidance Computer 1201 and 1202 errors (more about this and other aspects of the Apollo program at Apollo 11 Journals, the BBC also have an excellent podcast – 13 Minutes to the Moon that broke this down). This essentially was a memory overload error (should sound familiar as it is an attack vector, only difference is the overload was innocent for Apollo 11). The lesson is, the ground crew had encountered various issues during training, so put operational processes in place to allow them to quickly determine cause and action. Just as with modern businesses, the lunar programme could not be stopped to create months to rework Apollo’s software to avoid the issue, so they ensured there are what we might call compensating controls – a manual ground crew process that could tell Armstrong and aldrin what to do.

Continuing compensating controls, is not to have reactions ready, but also preventing attacks from reaching vulnerabilities. For example Oracle’s WebLogic Server has some CVEs around its T3 protocol. If you’re not using this as a communication channel between severs you can switch it off in the configuration. Better still, put those servers within a subnet with network security rules that prevent T3 traffic from getting into the subnet. There are multiple CVEs for T3 – so rather than patching everyone one of them, you’ve prevented the exploit from being usable by anyone other than someone having been able to get into that subnet.

The next upside of such a control is that even if the T3 vulnerability was exploited, the ‘blast radius’ is constrained, as the network configuration will make it a lot harder to move to other servers as first they need to find away to see beyond the local network. This brings another philosophical change which doesn’t have to be difficult to achieve. We always look at security on the inbound traffic – but we should be just as observant about outbound paths, and have controls. One very contemporary aspect of this is to put gateways between applications or agents and LLMs. The focus is that credentials to external tools and services are only introduced beyond the sphere of the LLM(s). But we can also prevent the LLM from being hijacked into abusing tools (from the LLM view point it is an inbound security control). But we should treat all outbound data this way. Its the fastest way to detect and stop data egress.

It doesn’t need a chaos monkey but …

Many will have over the years heard about the development of the chaos monkey at Netflix by Greg Orzell which is a small app that can be unleashed and will try to tear everything up. While this maybe seen as rather extreme, the underlying message is once its secured, don’t assume it remains secure (controls can be accidentally removed). Put basic security probes into the environment – you can test non-destructively. Don’t put a procedure in place and let is rot on a shelf, if you have capacity – fail things over. The more these sorts of processes a run, the more it becomes ‘muscle memory’ and while early iterations will be painful, that pain and cost will be minor to when it happens for real.

It isn’t if, but when …

We all like to think these things won’t happen to us, and we’re prepared. But worldwide 1 in 5 will have cancer in their lifetime. Many cancers are treatable, and largely routine – because the medical profession have had to treat the problem over and over. We just have to see IT attacks in the same manner.

Data sources and reporting

  1. CISA Known Exploited Vulnerabilities catalogue.
  2. CVE monthly trends, including CISA additions — cve.tools.
  3. VulnCheck: State of Exploitation 2026 (analysing 2025).
  4. CIRCL Vulnerability-Lookup | GitHub Repository
  5. CRACI CVE Publication Trends | CVE Program GitHub
  6. VulnCheck: State of Exploitation, H1 2026.
  7. VulnCheck: Expanding KEV, July 2025.
  8. Verizon 2026 DBIR.
  9. Google Threat Intelligence: 2025 zero-day review.
  10. Edgescan report 2026

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Are we heading to a Certificate-apocalypse?

29 Thursday Oct 2026

Posted by mp3monster in development, General, Security, Technology

≈ Leave a comment

Tags

ACME, AI, certbot, certificates, CLM, Cloud, CMC, CMP, cybersecurity, SCEP, Security, Technology, TLS

Most engineers are probably aware of the shortening life span of TLS certificates. this year certificate maximum life drops from 398 to 200 days. But what may not have registered is that the roadmap sees the max certificate time will continue to half every year. This means that by 2029 certificate lifecycle will be down to 47 days (31 days (core duration) + core duration/2 days + 1 day). But before 2029, in less than six months we’ll see the current certificate durations half from 200 to 100 days. As with the 2026 change, the new limits come into effect on the 15th March. Which does create an odd situation where certificates issued on the 14th March will outlast certificates issued on the following day.

The addition of half the core duration period is to allow for the recovery from failures. as of 2029 the expectation is every month the certificates should be rotated.

Manual processes are unsustainable

This means that the rotation of certificates has got to be accelerated to be monthly, a velocity that is simply not practical to be done by hand. This means the deployment of certificates really needs to be automated. There are a number of frameworks and standards that support the distribution of certificates.

Perhaps the best known and leading solution has been ACME protocol working in conjunction with Let’s Encrypt CA. The least known is the use of OpAMP also supports certificate distribution. The following table shows the different standards that specifically address certificate management (CLM), some implementations are linked to a Certificate authority , others are not.

StandardSpecificationOpen-source implementationsCommercial implementations
ACMERFC 8555Certbot (client), acme.sh (client), lego (client), step-ca (server), Boulder (server)DigiCert, Sectigo, GlobalSign, EJBCA Enterprise
CMP / CMPv3RFC 9810EJBCA Community, Bouncy Castle (library)EJBCA Enterprise, DigiCert Trust Lifecycle Manager
ESTRFC 7030Cisco libEST (client/server library), EJBCA Community (edition-dependent)EJBCA Enterprise, DigiCert
SCEPRFC 8894MicroMDM SCEP, EJBCA CommunityMicrosoft Intune, EJBCA Enterprise, DigiCert
CMCRFC 5272Dogtag PKICommercial implementations are less prominent; verify explicit CMC support and supported operations with individual PKI vendors

To understand the differences between the different standards, it is worth looking at the standards from the perspective of the operations that are needed to manage certificates.

FunctionACMECMPESTSCEPVendor
CLM API
Certificate issuanceYesYesYesYesUsually
Certificate renewalYesYesYesYesUsually
Certificate revocationYesYesNo native operationLimitedUsually
CA certificate rolloverLimitedYesYesLimitedVaries
Central certificate inventoryNoNoNoNoUsually
Ownership / application mappingNoNoNoNoVaries
Policy managementLimitedLimitedLimitedLimitedUsually
Application deploymentNoNoNoNoVendor-specific
Application activationNoNoNoNoThrough integrations
Deployment verificationNoNoNoNoVaries

You’ll note that the deployment, activation and verification steps fall into the domain of vendor solutions, and are variable. However we can address such considerations in an open-source (or commercial) manner through tooling such as Puppet, Chef and certain payload types provided by OpAMP.

These CLM options aren’t the end of the story as you may wish to look more broadly at no just the request and distribution of certificates, but the mechanisms through which the certificate is stored, and there is an overlap with such management with keys as well as certificates. For example APIs may use keys of certificates to help verify authenticity of the request. This also extends to mTLS operations where the client needs certificate to verify the server requests. For example KMIP provides an interface that can support cryptographic object (keys, certificates (public, private, symmetric), secret data, PGP Keys).

StandardSpecificationOpen-source implementationsCommercial implementations
KMIPOASIS KMIPPyKMIP (Python client/server)HashiCorp Vault Enterprise, Thales CipherTrust
PKCS#11PKCS#11 v3.1SoftHSM, OpenSCHashiCorp Vault Enterprise, Thales Luna HSM
SCIMRFC 7644elimity SCIM (Go library)Microsoft Entra ID, Okta

Managing the impact

There are several impacts that need to be considered:

  • tracking certificate expiry
  • Last mile from client deployer to server folder and any refresh nudge required

Tracking certificate expiry

Traditionally certificate expiry management could be handled as a manual process – after all if you have a log of which systems had certificates then if you simply reviewed the list once a month, you could spot expiries with time to spare. Even if the certificate distribution process was automated across many nodes. This just isn’t going to work, and the contingency time incorporated into the certificate life is contracting. Rather than trusting to a tracker spreadsheet being managed we should really focus on actual certificate data. This points to having a means by which application observability has a reporting metric on when a certificate will expire (date or a number of days). The later will prove to be easier as a metrics measure, but the value needs to be in context of when the metric was collected.

Last mile challenge

While automation frameworks like Certbot and step-ca can drive the certificate issuance. These processes get the certificate to the relevant server. But we still need to move the certificate intro the location the application expects the certificate to be, and do we need to nudge the server to reload the certificate? Certbot for example expects to have a script to trigger for this action. This can get a little messy to deploy and maintain, particularly if there are subtle variances in deployments (as a result of manual application deployment in legacy environments), configuration creep etc. One possible way to simplify this is to leverage the capabilities of environment scanners to locate apps and their configs. Then use that to drive the auto creation of that last mile.

Why the change?

The history for the reduction has largely been driven by browser providers particularly Google and Apple, and their security concerns about the trustworthiness of the web’s public key infrastructure (PKI). Historically, certificates could remain valid for several years, creating a prolonged window of exposure if a private key was compromised, a certificate incorrect certificates being wrongly used (organization or domain information became outdated – fore example domain name changes, acquisition of services etc). Certificate revocation mechanisms, particularly CRLs and OCSP, have not proven to be reliable as a means of ensuring browsers quickly stop trusting compromised certificates. Shorter lifetimes reduce this exposure and encourage the automation of certificate management.

The driving of changes has shifted from browser providers unilaterally declaring they won’t trust certificates that are older than a certain age, to a more formalized process governed primarily by the CA/Browser Forum, (an independent body bringing together Certificate Authorities and browser vendors).

Useful resources

  1. A/Browser Forum — TLS Baseline Requirements, version 2.3.1 (4 October 2026), section 6.3.2. Authoritative current hard limits: 398 days before 15 March 2026; 200 days from that date; 100 days from 15 March 2027; 47 days from 15 March 2029. Section 1.1 and the revision history identify 1 July 2012 as the original effective date. https://cabforum.org/working-groups/server/baseline-requirements/requirements/
  2. CA/Browser Forum — FAQ for Baseline Requirements. Historical summary of the 60-month limit from July 2012 and the 39-month general limit from April 2015. This is a historical source, not the current rule set. https://cabforum.org/faq-about-the-baseline-requirements/
  3. CA/Browser Forum — Ballot 193: 825-day Certificate Lifetimes (17 March 2017). Establishes the March 2018 transition to 825 days; also records the preceding 39-month rule and the qualifying 60-month legacy exception through June 2016. The page contains amendment markup; read the replacement wording in context. https://cabforum.org/2017/03/17/ballot-193-825-day-certificate-lifetimes/
  4. Apple — About upcoming limits on trusted certificates. Announces the 398-day maximum for public TLS server certificates issued on or after 1 September 2020 and explains its scope. This change originated in browser/platform trust policy. https://support.apple.com/en-us/102028
  5. CA/Browser Forum — Ballot SC081v3: Introduce Schedule of Reducing Validity and Data Reuse Periods (11 April 2025). Records the successful vote for the 2026–2029 reductions; source 1 confirms the resulting current requirements. The April milestone is the vote result, not the end of the subsequent intellectual-property review period. https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/
  6. IETF Datatracker — draft-barnes-acme-00 (28 January 2015). First revision of the ACME Internet-Draft, before working-group standardization. https://datatracker.ietf.org/doc/draft-barnes-acme/00/
  7. Let’s Encrypt / ISRG — Entering Public Beta (3 December 2015). Announces public availability without invitations. https://letsencrypt.org/2015/12/03/entering-public-beta
  8. Electronic Frontier Foundation — Announcing Certbot: EFF’s Client for Let’s Encrypt (May 2016). Explains the rename, the EFF home, and ACME-based issuance and configuration automation. https://www.eff.org/deeplinks/2016/05/announcing-certbot-new-tls-robot
  9. Let’s Encrypt community — New name (Certbot) for Python Let’s Encrypt client (12 May 2016). Dated contemporary announcement linking to EFF’s announcement; supports the exact event date. https://community.letsencrypt.org/t/new-name-certbot-for-python-lets-encrypt-client/15688
  10. RFC Editor — RFC 8555: Automatic Certificate Management Environment (ACME), March 2019. Standards-track specification, published as a Proposed Standard. https://www.rfc-editor.org/info/rfc8555/
  11. ACME — RFC 8555: Automated Certificate Management Environment; supports automated certificate issuance, renewal and revocation.
  12. CMP / CMPv3 — RFC 9810: Certificate Management Protocol; supports PKI enrolment and certificate lifecycle operations.
  13. CMP over HTTP — RFC 9811: Defines HTTP transport for CMP messages.
  14. EST — RFC 7030: Enrollment over Secure Transport; supports certificate enrolment and re-enrolment over HTTPS.
  15. SCEP — RFC 8894: Simple Certificate Enrolment Protocol; commonly used for network devices.
  16. CMC — RFC 5272: Certificate Management over CMS; supports certificate requests and management messages. Updated to — RFC 10002: Updated specification superseding RFC 5272.
  17. KMIP 2.1 — OASIS KMIP 2.1 Specification: Key Management Interoperability Protocol; manages cryptographic keys, certificates and other cryptographic objects.
  18. PKCS#11 3.1 — OASIS PKCS#11 3.1 Specification: Cryptographic Token Interface; standard API for accessing cryptographic hardware and software tokens.
  19. SCIM 2.0 — RFC 7644: System for Cross-domain Identity Management; supports identity and account provisioning rather than certificate management.
  20. SPIFFE — SPIFFE Specifications: Secure Production Identity Framework for Everyone; standardizes workload identity across heterogeneous systems.
  21. X.509-SVID — X.509-SVID Specification: Defines X.509 certificates used for workload identity and authentication.
  22. X.509 PKIX — RFC 5280: Defines the Internet X.509 public key certificate and Certificate Revocation List (CRL) profile.
  23. CA/Browser Forum Baseline Requirements — Current Baseline Requirements: Industry requirements governing publicly trusted TLS certificate issuance, domain validation, security practices and maximum certificate validity periods.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Can patches have functional impacts?

27 Tuesday Oct 2026

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, cybersecurity, languages, OS, patches, patching, Security, Technology, vulnerability

With the recent upheaval from the newly discovered PHP vulnerability (often referred to as PHP2Shell), and updates to OpenTelemetry declarations, it struck me that it’s worth highlighting an assumption we all tend to make with patches.

We tend to think patches introduce fixes or defences that mean functional behavior is immutable. It is an understandable reading based on definitions such as Semantic Versioning (semver):

PATCH version when you make backward compatible bug fixes

However, languages and frameworks point towards a different model, where defined API behaviour will be respected.

The bottom line is if you’re exploiting undocumented behaviors, it makes it all the more necessary to test and ideally in an automated manner changes not only in the applications but also for the stack.

If you are taking advantage of undocumented behavior, then it is worth adding an abstraction layer to allow mitigation changes to be added.

The following identifies the statements at a language level relating to what impact a patch can have:

Language / runtimeBehavior changes in patchesBreaking API changes in patchesPolicy, limitations and supporting documentation
GoYesExceptionalExplicit security, unspecified-behavior and specification-correction exceptions. Go compatibility promise
PHPYes, for correctionsGenerally noSecurity and bug fixes permitted; public userland API must remain backward compatible. Internal ABI/API has narrowly defined exceptions. PHP release policy
Python (CPython)Yes, restrictedExceptionalReasonable bug fixes may affect behavior; stronger incompatibilities require exceptional justification. PEP 387
Java (OpenJDK / Oracle JDK)YesGenerally noSecurity corrections can alter runtime behavior; source, binary and behavioral compatibility are evaluated separately. OpenJDK compatibility
C# / .NET runtimeExceptionalExceptionalBreaking changes generally prohibited during servicing, except for security vulnerabilities or other critical issues. .NET release policy
F# / .NET runtimeExceptionalExceptionalShares .NET runtime servicing policies; compiler and language tooling have their own compatibility considerations. .NET compatibility rules
VB.NET / .NET runtimeExceptionalExceptionalSame underlying .NET servicing guarantees. .NET servicing
TypeScriptYes, restrictedNot normallyPatch releases may contain high-priority regression, performance-regression and safe language-service corrections. Ordinary breaking changes are targeted at larger releases. TypeScript release process
JavaScript (ECMAScript)Not applicable at language-specification levelNot applicableECMAScript is a specification, not a separately patched runtime. Behavior depends on implementations such as V8, SpiderMonkey, JavaScriptCore and Node.js. ECMAScript specification
Node.jsYes, for correctionsExceptionalPatch releases accept fixes; semver-major changes are excluded under normal procedures. LTS security exceptions can affect native API/ABI compatibility when unavoidable. Node.js release procedures

The following table, takes the same approach looking at how operating systems view changes for patches:

Operating systemBehavior changes in maintenance patchesBreaking API / ABI changesPolicy, limitations and supporting documentation
Red Hat Enterprise Linux (RHEL)YesGenerally avoidedBackports security and bug fixes into supported versions. Red Hat explicitly tests for unwanted side effects, so this is not general permission to break compatibility. RHEL security backporting
SUSE Linux Enterprise Server (SLES)YesGenerally avoidedMaintenance includes security corrections and defect fixes; individual components have different lifecycle rules. Service packs allow broader changes. SUSE support policy and SLES release notes
Oracle LinuxYesNo general permission establishedOracle distributes security, bug-fix and enhancement errata as package updates. Security hardening can alter functionality independently of major OS releases. Oracle Linux errata policy
Ubuntu LTSYes, cautiouslyGenerally avoidedBackported security patches prioritize compatibility. Exceptional changes and certain feature updates can be accepted under the Stable Release Updates process. Ubuntu security updates
Debian StableYes, restrictedGenerally avoidedImportant bug and security fixes are acceptable; updates are deliberately minimal and require review. Debian Stable update rules
Microsoft Windows 11 / Windows ServerYesNo general permission establishedCumulative security, quality and out-of-band updates may change functionality. Windows 11 may also deliver feature enhancements through monthly servicing. Windows update release policy
Apple macOSPossibleNo general permission establishedSecurity and maintenance releases can affect operating behavior, but there is no general published patch-version exception equivalent to Go’s. Apple security releases

Often when building solutions we don’t see low level frameworks and how they work, so we may not know about any undocumented feature exploitation.

The challenge it can bring

This points to increased testing, which ideally is automated. The reaction is initially is so what. But, when vulnerabilities like PHP come up which had openly triggered systems being disconnected from the net.

So extending testing (even if automated) is going to experience resistance from the commercial parts of an organization who are unlikely to understand the nuances, but will react to headlining risks.

How do we move forward?

There are some strategies we can adopt that can help us. The industry used to talk about security in depth and, more recently, system resilience. Regardless of terminology, we want to introduce control layers that reduce the chance of malicious intent. Often referred to as compensating controls.

For example, if the vulnerability is overloading the runtime by sending an oversized payload, a firewall or proxy layer can have rules that prevent payloads from being too big from hitting the backend.

If the networking allows it, then segmenting the network reduces the risk of a breach affecting other environments.

This doesn’t mean if we can patch, that we shouldn’t. What it does mean is we can be considered and not knee-jerk panic and unwittingly introduce new problems.

Conclusion

Adding additional security and proxy layers and segmenting system environments may seem like overkill, particularly when introducing new systems. But these steps create the means to be far more resilient.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Observability as a security sensor using Fluent Bit, OpenTelemetry and AIDE

09 Friday Oct 2026

Posted by mp3monster in Fluent Observability, Fluentbit, General, OpAMP, Technology

≈ Leave a comment

Tags

AI, cyber-security, cybersecurity, Security, Technology

Observability is usually looked at from the perspective of applications and their needs and the cloud native platforms. Nothing wrong with that, but it is also a rather narrow view of the telemetry we are already collecting or are very capable of collecting. Our observability agents (Fluent Bit, Fluentd, Elastic Agent …) can collect signals (logs, metrics and traces) that can contribute to the security operations within an enterprise. More importantly, the same open-source tooling can help answer a deceptively simple question such as has someone changed the system, or changed the tooling that tells us what the system is doing?

When it comes to security, that second part matters. An attacker does not always need to exploit a new vulnerability. If they can alter a service definition, scheduled task, access-control file, logging configuration or telemetry destination, they can prevent us from sensing what damage they maybe doing, including opening up simpler or more invasive routes into your systems. MITRE ATT&CK describes this kind of action as a defence-evasion technique. Research from Palo Alto Networks Unit 42 has also described cloud logging services as an attack surface, including changes that stop, redirect or selectively reduce audit collection. Not only that getting at logs, means they can potentially determine when systems that are running, and find new attack vectors.

As you can see being able to detect configuration and file manipulation is an important factor for security. Many may react to this with concern about introducing cost for expensive security tools, but if there are simple means to listen to OS activities that relate to file manipulation then we can leverage our observability estate for this aspect of security.

The wider open-source stack

That neutrality for our observability agents is valuable for security. A host or cluster can emit operational and security telemetry through the same managed pipeline, while routing data to one or more destinations such as OpenSearch, Loki, ClickHouse, a SIEM, object storage or an OTLP-compatible service. Not to mention if the event is for an application configuration, we can evaluate it against application and service viewpoints such as managing configuration drift.

The roles are complementary:

  • AIDE is a open-source Linux utility that provides the file-integrity check. It builds a known-good database and reports additions, removals and changes to selected files and attributes. Aide achieves this through the use of the iNodes API, which if you preferred could also be tapped into even more directly.
  • Fluent Bit or other OTel compliant tools collect the result close to the workload, adds context, buffers during a network interruption and routes the event. Our backend services which are in all likelihood OTLP capable, means the events can be send nearly anywhere for analysis and alerting
  • The analytical backend correlates the integrity event with logins, process starts, deployments, traces and other activity, then drives an alert or investigation.

AIDE is deliberately focused. It does not need to become a SIEM and Fluent Bit does not need to become a file-integrity engine. Combining small tools through open interfaces lets each component do the job it is good at.

We can use Aide not only to monitor our configurations, but when the deployed binaries are disturbed. We can extend this setup further. By exploiting custom commands with OpAMP (particularly for a supervisor) can be used to not only manage the observability agent, but also manage the deployment of configuration to tools like Aide. Now if we think something is being abused, the option to change our file integrity monitoring is easily in our reach.

Why file integrity belongs in the observability conversation

The available incident statistics do not normally include a neat category called “configuration-file manipulation”. The behaviour is spread across techniques such as creating or modifying system processes, scheduled tasks, startup mechanisms, accounts, policies and logging controls. That makes a single adoption or attack percentage difficult to defend. It does not make the risk rare.

For example, Google Cloud’s M-Trends 2025 reporting included techniques such as Create or Modify System Process (20.6%), Account Manipulation (19.9%), Scheduled Task/Job (14.3%), Registry Run Keys/Startup Folder (11.5%) and Web Shell (7.2%) among observed investigations. These are broader ATT&CK behaviours, not a count of configuration-file changes, but many are enabled or persisted through changes to system configuration.

There is also a compliance driver. PCI DSS v4.0.1 Requirement 11.5.2 explicitly requires a change-detection mechanism, such as file-integrity monitoring, for critical files, with comparisons at least weekly and alerts for unauthorised modification. NIST SP 800-53 Revision 5 control SI-7 requires monitoring for unauthorised changes to software, firmware and information. Other standards may describe the required outcome as configuration control, integrity, audit protection or system monitoring rather than naming a particular product.

FIM helps provide the evidence behind those outcomes:

  • what changed;
  • which host or workload reported it;
  • when the change was detected;
  • which attributes or content hashes changed;
  • whether a change matched an approved deployment window;
  • whether the alert was acknowledged and investigated; and
  • whether the integrity baseline was updated through a controlled process.

The last point is easily overlooked. Automatically accepting every change into the baseline produces a very tidy report and a very weak control.

What should be monitored?

Starting with every file on every host usually creates cost and noise. I would start with the files that can change control flow, identity, collection or evidence retention:

  • Fluent Bit configuration, parser files and included fragments;
  • Vector configuration and transformation definitions;
  • OpenTelemetry Collector configuration;
  • systemd units and drop-ins for telemetry agents;
  • log rotation and retention configuration;
  • TLS trust stores, client certificates and references to credentials;
  • auditd, syslog and journald configuration;
  • scheduled tasks, startup scripts and authorised SSH keys;
  • alert rules, dashboards and detection content managed as files; and
  • the AIDE configuration and baseline database itself.

Runtime state, offset databases, buffers and normal log files should normally be excluded. They are expected to change and monitoring them obscures the files that should remain stable.

There is an architectural wrinkle: if AIDE reports through Fluent Bit, and an attacker changes both AIDE and Fluent Bit, the local signal can be suppressed. The practical response is defence in depth. Restrict write access, ship events off-host quickly, monitor the agent configuration and service unit, protect or sign the AIDE baseline, and alert when expected heartbeats or scheduled check results disappear. Silence is also a signal.

Example: AIDE reporting through Fluent Bit and OTLP

The following is a starting point rather than a universal production configuration. Package paths and AIDE defaults differ between Linux distributions, so it should be merged with the supplied configuration rather than replacing it blindly.

First, add an AIDE rule group for the observability configuration. For example, in /etc/aide/aide.conf.d/observability.conf:

# Permissions, inode, links, ownership, size, timestamps and SHA-256 content.
OBS_CONFIG = p+i+n+u+g+s+m+c+sha256
​
/etc/fluent-bit                         OBS_CONFIG
/etc/vector                             OBS_CONFIG
/etc/otelcol-contrib                   OBS_CONFIG
/etc/systemd/system/fluent-bit.service.d OBS_CONFIG
/etc/systemd/system/vector.service.d     OBS_CONFIG
/etc/systemd/system/otelcol.service.d   OBS_CONFIG
/etc/aide                               OBS_CONFIG
​
# These are runtime data and should not be treated as static configuration.
!/var/lib/fluent-bit
!/var/lib/vector
!/var/lib/otelcol
!/var/log

The main AIDE configuration can direct its report to a file that Fluent Bit is permitted to read:

database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout=true
​
report_url=file:/var/log/aide/aide.log
report_level=changed_attributes

Initialise and approve the baseline using the procedure provided by the operating-system package. The baseline database should be writable only by the account performing the controlled update; consider retaining a signed or read-only copy outside the host.

Run aide --check from a systemd timer or the organisation’s scheduler. An hourly check provides better detection latency than the weekly minimum stated by PCI DSS, although the right frequency depends on estate size, I/O cost and risk. The scheduler should also emit a success or failure event so that a missing check can be detected.

Fluent Bit can then tail the report, add security context and forward the records over OTLP. The Tail input uses a database to retain offsets, and the OpenTelemetry output supports OTLP/HTTP and OTLP/gRPC.

service:
flush: 5
log_level: info
storage.path: /var/lib/fluent-bit/storage
pipeline:
inputs:
- name: tail
tag: security.fim.aide
path: /var/log/aide/aide.log
key: body
path_key: log.file.path
db: /var/lib/fluent-bit/aide-tail.db
db.sync: normal
read_from_head: false
refresh_interval: 5
rotate_wait: 30
skip_empty_lines: true
skip_long_lines: true
storage.type: filesystem
filters:
- name: modify
match: security.fim.aide
Add:
- event.domain file
- event.category file
- event.type change
- security.control file_integrity
- service.name aide
outputs:
- name: opentelemetry
match: security.fim.aide
host: otel-collector.internal
port: 4318
logs_uri: /v1/logs
tls: on
tls.verify: on
compress: gzip

This example emits each AIDE report line as a log record. That is simple and robust, but the backend will need to group lines belonging to the same check. For a more mature deployment I would place a small wrapper around the AIDE invocation that converts each detected change into JSON Lines and adds a check identifier, hostname, baseline version and execution result. Fluent Bit can then use its JSON parser and every changed file becomes a structured event.

Vector can implement the same pattern with its file source and an OTLP-capable destination or intermediate Collector. The OpenTelemetry Collector can also collect files using the contributed filelog receiver. The choice is less important than applying consistent fields, reliable buffering, secure transport and an independently managed destination.

From event collection to a security control

Collecting the AIDE output is only the first half of the solution. To turn it into a control, the backend needs useful detection logic and operational ownership.

I would start with alerts for:

  • a change to an observability agent destination, filter or exclusion rule;
  • a change to a telemetry service unit or executable;
  • deletion or replacement of the AIDE database;
  • changes outside an approved deployment window;
  • a burst of changes across several hosts;
  • a configuration change followed by a fall in log volume; and
  • a missing AIDE run, agent heartbeat or expected integrity summary.

Correlating the FIM event with deployment records is important. A checksum change shortly after an approved GitOps rollout is very different from the same change following an interactive root login at 02:00. Observability provides the surrounding evidence that allows the integrity event to be assessed rather than simply counted.

There is also value in feeding an integrity event back into operational telemetry. A changed collector configuration can annotate dashboards, traces and incident timelines. The question changes from “why did the logs stop?” to “did the logs stop when this configuration changed?” That is a much better starting point for both an SRE and a security analyst.

Final thoughts

The observability and security domains have spent years building separate pipelines for data that frequently starts on the same machine. Open standards and focused open-source tools give us an opportunity to reduce that duplication.

AIDE supplies a mature integrity signal. Fluent Bit and Vector make that signal transportable and enrichable. OpenTelemetry makes the route to the wider platform portable. None of these components replaces access control, endpoint detection or a SIEM, but together they add a useful security capability to infrastructure many organisations already operate.

Perhaps the most important configuration to observe is the configuration that decides what we are allowed to observe.

Resources

  • AIDE project
  • Fluent Bit Tail input
  • Fluent Bit OpenTelemetry output
  • Vector file source
  • OpenTelemetry Collector configuration
  • MITRE ATT&CK: Impair Defences – Disable or Modify Cloud Logs
  • Unit 42: Abusing Cloud Logging Services for Defense Evasion and Visibility
  • NIST SP 800-53 Revision 5
  • PCI Security Standards document library
File Integrity Management using Aide and Fluent Bit

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Relationship between CVEs, CVSS, CWE and lots of other TLAs starting with C

07 Friday Aug 2026

Posted by mp3monster in development, General, Security, Technology

≈ Leave a comment

Tags

CVE, CVSS, CWE, cyber, MITRE, NIST, Owasp, Security, standards

A lot of developers are focused on CVEs, as we need to know does the library, language or other building block I use to create a solution have an identified vulnerability that needs mitigating? We probably also pay attention to OWASP (Open Worldwide Application Security Project). But CVEs and OWASP aren’t the beginning and end of the story. This a larger ecosystem of standards involved.

CWE (Common Weakness Enumeration) is of particular interest if you get a chance to step back from fire fighting CVEs. This is interesting because, rather than addressing individually identified vulnerabilities, like the OWASP Top 10s, it looks at a classifies the vulnerabilities. When you get down the Base and Variant categories we’re into specific details. This is helpful, as classifying our CVEs against CWE can tell us the technical domains where the maximum effort can return the greatest level of mitigation/remediation of CVEs (be that creating a patch or defining a mitigation strategy).

I was going to sketch out the relationships between the key standards, but realized an LLM can do a better job, so here is a visual summary:

With so many TLAs in the diagram, here is a quick reference pulled together (including from my own library of handy links).

Acronym / referenceExpanded formOfficial HTTP linkOne-sentence summaryAuthority / allocation relevance
CVECommon Vulnerabilities and Exposurescve.org/about/overviewCVE provides globally recognised identifiers and records for publicly disclosed cybersecurity vulnerabilities. (cve.org)CVE IDs are assigned by authorised CNAs or by MITRE/CVE Program structures.
CNACVE Numbering Authoritycve.org/partnerinformation/listofpartnersCNAs are authorised organisations that assign CVE IDs and publish CVE Records within an agreed scope. (cve.org)Primary delegated authorities for allocating CVE IDs.
CNA-LRCNA of Last Resortcve.org/ProgramOrganization/StructureA CNA-LR supports CVE assignment when no more specific CNA is available for a vulnerability. (cve.org)Fallback route for CVE ID allocation.
MITREThe MITRE Corporationmitre.org/focus-areas/cybersecurity/capabilities-resourcesMITRE is closely associated with CVE, CWE, CAPEC and ATT&CK as a maintainer, operator or steward of major cybersecurity knowledge bases. (MITRE)Current CVE Program Secretariat and maintainer of CWE/CAPEC resources.
CWECommon Weakness Enumerationcwe.mitre.orgCWE is a community-developed catalogue of software and hardware weakness types that can lead to vulnerabilities. (Common Weakness Enumeration)CWE IDs are maintained by MITRE; they classify weakness types, not individual vulnerabilities.
CWSSCommon Weakness Scoring Systemcwe.mitre.org/cwssCWSS is a CWE-related scoring method for prioritising software weaknesses, but the older version referenced by MITRE is marked obsolete. (Common Weakness Enumeration)Scores weaknesses, not CVE vulnerabilities; largely secondary compared with CVSS in current vulnerability workflows.
CVSS v4.0Common Vulnerability Scoring System version 4.0first.org/cvss/v4.0CVSS v4.0 is the current major CVSS standard family used to express vulnerability severity with base, threat, environmental and supplemental metrics. (FIRST Forum) (Forum of Incident Response and Security Teams)Maintained by FIRST; it scores vulnerabilities but does not allocate vulnerability IDs.
Defines the scoring structure/ranking scale, not ID allocation.
NISTNational Institute of Standards and Technologynist.govNIST hosts and maintains important security automation resources including NVD and CPE-related specifications. (NVD)Maintains NVD and official CPE dictionary infrastructure.
NVDNational Vulnerability Databasenvd.nist.govNVD enriches CVE records with structured metadata such as affected platforms, weakness classifications, references and severity scores.Enrichment database, not the allocator of CVE IDs.
CPECommon Platform Enumerationnvd.nist.gov/Products/CPECPE provides standardised names for identifying affected products, platforms and software configurations. (NVD)Supports product/platform matching in NVD and vulnerability scanners.
CAPECCommon Attack Pattern Enumeration and Classificationcapec.mitre.orgCAPEC is a MITRE-maintained catalogue of attack patterns used to understand how weaknesses may be exploited. (CAPEC)CAPEC entries relate to CWEs and help connect weaknesses to attack behaviour.
CSAFCommon Security Advisory Frameworkoasis-open.org/committees/csafCSAF is an OASIS standard for creating and exchanging structured machine-readable security advisories. (OASIS)Advisory exchange format that can carry CVE, product status, remediation and scoring data.
VEXVulnerability Exploitability eXchangecisa.gov SBOM resourcesVEX communicates whether a known vulnerability is exploitable in a specific product or deployment context. (CISA)Helps qualify CVE findings so consumers can distinguish exploitable from non-exploitable exposure.
CISACybersecurity and Infrastructure Security Agencycisa.govCISA publishes operational vulnerability guidance and maintains the Known Exploited Vulnerabilities catalogue. (CISA)Maintains KEV and participates in the wider CVE/vulnerability ecosystem.
KEVKnown Exploited Vulnerabilities cataloguecisa.gov/known-exploited-vulnerabilities-catalogKEV is CISA’s authoritative catalogue of vulnerabilities known to have been exploited in the wild. (CISA)Prioritisation signal based on observed exploitation, not severity alone.
EPSSExploit Prediction Scoring Systemfirst.org/epssEPSS estimates the probability that a published CVE will be exploited in the wild within the next 30 days. (FIRST Forum)Probability-based prioritisation signal that complements CVSS and KEV.
CERT/CCCERT Coordination Centerkb.cert.org/vulsCERT/CC coordinates vulnerability disclosure and publishes Vulnerability Notes. (CERT Coordination Center)Can act as a coordinator in disclosure workflows and may interact with CNA/CVE processes.
CVDCoordinated Vulnerability DisclosureCERT Guide to CVDCVD is the process of coordinating information among finders, vendors and other stakeholders before public disclosure and mitigation communication. (CERT Coordination Center)Process framework around disclosure; not itself an ID system.
SBOMSoftware Bill of Materialsntia.gov/software-bill-materialsAn SBOM describes the software components that make up a product so downstream users can reason about exposure and supply-chain risk. (NTIA)Often paired with VEX and vulnerability feeds to assess affected components.
CycloneDXCycloneDX Bill of Materials standardcyclonedx.orgCycloneDX is an OWASP full-stack Bill of Materials standard for supply-chain and cyber-risk use cases. (CycloneDX)SBOM/VEX-capable format used in vulnerability management pipelines.
SPDXSoftware Package Data Exchangespdx.devSPDX is an open standard, ISO/IEC 5962:2021, for representing SBOMs and other software supply-chain metadata. (SPDX)SBOM format often used alongside vulnerability data sources.
SWIDSoftware Identification tagNIST SWID guidanceSWID tags are ISO/IEC 19770-2 software identification files that can support software asset, patch and vulnerability management. (NIST Computer Security Resource Center)Product identification standard related to CPE-style inventory matching.
SSVCStakeholder-Specific Vulnerability CategorizationCMU SEI SSVC v2.0SSVC uses decision trees to prioritise vulnerability response actions for specific stakeholder contexts. (SEI)Prioritisation model that complements CVSS, KEV and EPSS.
MITRE ATT&CKAdversarial Tactics, Techniques and Common Knowledgeattack.mitre.orgMITRE ATT&CK is a knowledge base of adversary tactics, techniques and procedures based on real-world observations. (MITRE ATT&CK)Related attack-behaviour framework that can be connected to CAPEC/CWE/CVE analysis.
OWASP Top 10Open Worldwide Application Security Project Top 10owasp.org/Top10The OWASP Top 10 is a standard awareness document for developers and web application security risks. (OWASP Foundation)Useful application-security reference that often maps conceptually to CWEs.
OSVOpen Source Vulnerabilities / OSV schemaosv.devOSV provides a machine-readable vulnerability format and database that maps open-source vulnerabilities precisely to package versions or commit hashes. (OSV)Adjacent vulnerability-data ecosystem, especially useful for open-source dependency scanning.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Software Patching doesn’t mean you’re secure

03 Monday Aug 2026

Posted by mp3monster in General, Technology

≈ 1 Comment

Tags

CEMLI, Mythos, patching, prioritization, RBVM, risk, Security, Technology, vulnerability

Ok, as a heading, that is a bit clickbait. But the underlying message is true. Let’s ask ourselves some simple questions. What is a patch? The Cambridge Dictionary says within the context a computers:

small computer program that can be added to an existing program in order to make the existing program work as it should

The key here is change to make something existing to be correct. That means there is something wrong. this can include a means to circumvent security. But to need a fix, we first need to find the fault. As a result that must mean we there is ALWAYS the chance of a fault. Simply put, there is always a vulnerability before a fix. Therefore regardless of how patched we are there is always a possibility however small of their being a vulnerability and that a malicious actor (or hopefully a bug bonus schema or something like Mythos) finds it first.

Cultural challenge

As an industry, our first question when there is a problem is, have you applied the patches (or bug fixes), or have you upgraded to the latest version yet? This is almost as pervasive as the old joke when there is a computer problem: Have you turned it off and on again?

Don’t get me wrong, if you have patches, I’d err towards applying them. But, I’d also advocate trying make a risk assessment as to what patching could go wrong; if you can safely eliminate a problem, it is better to do so. But, blindly patching can be an issue. Patches can conflict other parts of the system, be applied to systems resulting devices being ‘bricked‘ or hitting the ‘blue screen of death‘.

Let me illustrate, using desktop PC. When I receive a windows update, it takes a couple of minutes to see what is there. If the update looks like a malware signatures update – no hesitation, the risk of not taking the change is bigger than bot. If the update is for a driver, then we’re more cautious, which driver, can I install the older one. Time to make sure there is an OS recovery point. If it’s something like a H1 or H2 cumulative and features update – time to run a backup and a restore point.

In an enterprise software environment, be that monolithic on-premises ERPs, through to vast microservice deployments there are the potential for problems. Particularly when you consider the amount of possible customisation that could be involved (in the Oracle domain you’ll hear about CEMLI). This means the possible permutations makes it impossible for the vendor to provide assurances. Not to mention if you’ve applied Modification level changes a patch may well conflict with your modification.

Security in Depth

But security, and the principle of security in depth, is far more than just patching. But I will always advocate more. The ideal world is that we have layers of security, so if one can’t be patched, then other layers are mitigations, or to use another term often used in enterprise security, a ‘compensating controls’ (the idea that if you can’t address a vulnerability in one place, you have security controls elsewhere that can compensate for a weakness).

But we forget – breaches precede patches

The thing we tend to overlook when we get fixated on the idea that staying patched means we’re secure is that a patch always follows a breach. Remember, somewhere along the line, a ‘hacker’ (preferably a security researcher, penetration tester, or a white-hat hacker looking to earn money through bug bounty schemes) will find a vulnerability and breach a system as a result. It is only now that the exploit (vulnerability) is known that the work on creating a patch to address it.

But sometimes a patch isn’t practical (it has massive performance implications and demands a rewrite of a key element of the solution). Therefore, we can address the issue with mitigating controls – this could be at a code level, e.g. adding upfront checks for a specific scenario, through to preventing a piece of software from being used in a particular way.

A vulnerability doesn’t mean you’re vulnerable

This is the difficult perspective, and makes interpreting vulnerability scanner data hard to understand. Often, vulnerability scanners will look across a server and determine what products and components are deployed, and what versions of that functionality are deployed. Then look up those components for the list of attributed vulnerabilities. This is helpful, you know, you need to give this some due consideration. But we should not forget that software (and particularly enterprise software) has many configuration points that can cause it to follow different paths, and the vulnerability may lie only in one path. The latest version of WebLogic server has 650-700 mBeans (one of the techniques to configure the WLS behavior).

Look at it another way, you’re a single person with a car (which usually has 4 seats). You need the car to commute to and from work, and to do common chores like going shopping. But one of the back seats ended up getting ripped. The garage wants thousands to replace it. This does not stop you from using the car to commute to work and to do other necessary tasks. Yo have probably unconsciously applied a ‘compensating control’ – never take more than 2 other people in the car with you, and prevent a back seat passenger from sitting on the ripped seat. Frustrating – maybe, but not inoperable. Of course, applying that compensating control to use the car as a taxi would not be commercially viable. You, as the car owner, need to do some due diligence, such as ensuring the rip isn’t a symptom of a more fundamental risk? Yes. Do you need to consider the likelihood of needing more than 2 passengers? Yes.

Patch velocity

One of the challenges that are developing is the velocity of patches coming as AI is able to be used to locate potential errors that then get addressed. This creates an issue of the organization’s being able to rollout patches. This may not sound too challenging, until consider

  • not all organizations have a pure Kubernetes ecosystem with the means to orchestrate node replacement quickly and easily.
  • Capacity demands (people, and or compute resources) needed for performing verification steps for any change can out strip resources, or impact other change effort – such as legal compliance, or changes to keep you in business (although a breach could be just as catastrophic).
  • Demand impacts, when organizations are also trying to create bandwidth and/or budget to transition off vulnerable systems can end up being blocked.
  • Patch volumes for an environment that is already considered ‘fragile’ are going to generate a lot of work, as not only does the patch need to be applied, but a lot of effort will be needed to demonstrate the the wider business the possible impact has been validated.

To combat velocity, is to work on ccompensating controls in other areas of your system. If the application has a networking weakness, consider not patching the software, and ensuring networking configurations block the network port(s) that are associated with the vulnerability. Add monitoring, and checks to ensure the change to the network aren’t accidentally undone. If you do later apply the patch to address the weakness, don’t unpick the network mitigations, as this gives you ‘security in depth’.

Scanning tools

We can also use detection or scanning tooling to help. For solutions that you have development control over, and preproduction environments should be using these tools. But we have to be careful how we interpret the outcomes of these tools. Blindly take their reports can send us chasing issues, that aren’t really issues. While measuring detecting and measuring risk management is a positive thing, it needs to be done in an informed way.

For example, you run the scanning tool with admin level privileges so it can inspect everything. But, then it detects the finger print of an old version of Java with vulnerabilities in the Swing UI library. But the only reason Java is present is to run some CLI admin utilities. It is never used by the core application, there are no deployed Swing solutions in the environment. If we don’t give consideration to the context of the problem, we end up patching an issue that is in all probability going to be a problem.

At the same time, we can use this understanding to prioritise patching. Yes, applying a patch if available is good, but if applying a patch that breaks your tool is not. Spending time patching a vulnerability, when you don’t use the vulnerable code is eating into capacity to actually implement business change which may well eliminate the problem anyway (for example standardising scripting tools on Python).

What does this all mean?

Bottom line is, if you can patch, you have the ability and capacity to do so then it’s better to do so, as it keeps another layer of defence in place. But you don’t have access to patches (out of support etc), capacity to solve everything immediately then intelligent assessment of the vulnerability, risk driven prioritisation, look to mitigation strategies.

Defence in depth is not a security seller’s motto, but a genuine way to ensure that if one point fails, the next should protect you. But if you can’t patch, understand what the vulnerability is, and ensure you have mitigations in place. Ensure the issue and its implications in your context are documented along with the mitigations – so you have auditable content for any audit. You have been told where the minefield is, so you’ve fenced the area off, so people don’t wander into the issue unwittingly.

Useful Resources

  • NIST compensating security control
  • Patch velocity
  • NCSC Vulnerability management
  • Risk Based Vulnerability Management

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Anthropic Mythos – an LLM with potent security sting

13 Monday Apr 2026

Posted by mp3monster in AI, General, Technology

≈ 1 Comment

Tags

AI, Anthropic, Mythos, Security

There has been a rapidly growing series of articles being written about the limited launch of Mythos, a new LLM. The evolution of models has helped quickly advance AI-assisted software development. But the capabilities of Mythos and Project Glasswing that really grabbed attention and concern.

Glasswing is an initiative that allows major partner software and service vendors to access the Mythos model. This is because Mythos has made significant advances in identifying software vulnerabilities and generating exploits for them. This has been illustrated by Anthropic’s Red team – which found bugs in OpenBSD (OS) that have evaded detection for as much as 27 years. While the BSD family of operating systems isn’t as pervasive as Linux, they both share a similar open ethos and a sufficient community to keep them active and maintained. The underlying message here is that we can find and exploit such vulnerabilities, and there are certainly opportunities to do so elsewhere, in software that can affect a great many more users, such as Firefox.

Having key software vendors, such as OS and browser vendors, get access is certainly a positive step, but it doesn’t address a key consideration. Applying code fixes and releasing updates does not, by itself, equate to being safer. The true challenge is for end users and organisations to recognise the need to roll out updates quickly. This is where the source of true concern should be. The concerns …

  • Organisations don’t always release patches as soon as they’re available. There is an element of testing to ensure no adverse impact on each organisation’s setup. Even with simple browser changes, something affects the app’s behaviour.
  • Change represents risk, and organisations that experience issues during rollouts become increasingly risk-averse. Ironically, this is counterintuitive, but a very human reaction.
  • Vendors’ patching tends to prioritise the latest versions of products, which can create dependency challenges. Bringing software up to date can result in a growing infrastructure footprint (more storage, memory and CPU needed – vendors add capabilities and features to compete and meet customer feature needs, driving continuous growth). That can really add costs, particularly in highly distributed use cases, such as user desktops and IoT devices. Addressing the accumulation of patches means devices no longer have the capability to properly service the new footprint. Consider this: why do people replace smartphones? Sometimes it’s hardware features like a better camera, but often it’s simply not enough storage or not being able to run all the apps, photos, etc.
  • .Digging into some of the details from the Red Team shows that the LLM usage costs to uncover the vulnerabilities run from $50 – $20,000. This could have ramifications for smaller, more specialised software solutions where the cost of regularly rerunning the analysis outstrips potential revenue. As a result, we could suddenly see software product prices climb, or companies simply stop producing products we depend on. This may also see bad actors wanting to more quickly recoup the cost by accelerating the use of new exploits, in other words, more attacks, coming more quickly. Such considerations will create more pressure on the speed of patch cycles.
  • This level of capability suggests that we really do need to ensure people shift from boundary-style security to security at every layer of our solutions. That’s not just simply authentication, but code being defensive, validating data values it gets given and os on.

All of this means we have to change mindsets from just enough, or simply putting a front-line security layer in place, to embedding. As end users, we must start to adopt several behaviours:

  • Security conscious with our own devices – keeping software up to date and patched. I would consider my family to be above average when it comes to tech savvy, but even I am having to go in and run Windows updates on laptops, for example.
  • Start voting with our feet – many of the services we use are largely or entirely software-powered (banks, energy providers), if those providers show signs of not taking security seriously enough, time to go elsewhere before we become victims.

Keeping up

One observation that the Mythos and Project Glasswing reporting is that the advancements are significant step changes, not incremental advancements (for example, Antghropic’s Sonnet 4.6 was only released a couple of months ago, and didn’t score highly for creating exploits – although better at detection). This suggests a couple of things …

  • IT law has always played a game of catch-up, but if the advancements are going to be this large and this frequent, we have to start legislating against hypotheticals and allowing legal precedents to produce fine detail interpretations.
  • We may have to consider big-brother observation of AI use, mitigated by strong transparency rules governing the handling of findings.
  • Is the idea that we need to start looking at incorporating something like Asimov’s 3 Laws of Robotics into LLMs now looking far-fetched?
  • Do we need to start thinking about mitigating the risk of deep exploits by bringing back the possibility that systems must be air-gapped?

Hyperbole?

It would be easy to put this down to hyperbole, or wanting to be a click-baity, but this is gaining a lot of high-profile attention, just consider these examples:

  • What Is Claude Mythos—And Why Anthropic Won’t Let Anyone Use It (Forbes)
  • Anthropic’s new AI tool has implications for us all – whether we can use it or not (The Guardian UK)
  • Mythos autonomously exploited vulnerabilities that survived 27 years of human review. Security teams need a new detection playbook (Venture Beat)
  • Anthropic’s Mythos Will Force a Cybersecurity Reckoning—Just Not the One You Think (Wired)

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

MCP Security

30 Thursday Oct 2025

Posted by mp3monster in AI, development, General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, attack, attacks, cybersecurity, MCP, model context protocol, Paper, Security, Technology, vectors

MCP (Model Context Protocol) has really taken off as a way to amplify the power of AI, providing tools for utilising data to supplement what a foundation model has already been trained on, and so on.

With the rapid uptake of a standard and technology that has been development/implementation led aspects of governance and security can take time to catch up. While the use of credentials with tools and how they propagate is well covered, there are other attack vectors to consider. On the surface, it may seem superficial until you start looking more closely. A recent paper Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions highlights this well, and I thought (even if for my own benefit) to explain some of the vectors.

I’ve also created a visual representation based on the paper of the vectors described.

The inner ring represents each threat, with its color denoting the likely origin of the threat. The outer ring groups threats into four categories, reflecting where in the lifecycle of an MCP solution the threat could originate.

I won’t go through all the vectors in detail, though I’ve summarized them below (the paper provides much more detail on each vector). But let’s take a look at one or two to highlight the unusual nature of some of the issues, where the threat in some respects is a hybrid of potential attack vectors we’ve seen elsewhere. It will be easy to view some of the vectors as fairly superficial until you start walking through the consequences of the attack, at which point things look a lot more insidious.

Several of the vectors can be characterised as forms of spoofing, such as namespace typosquatting, where a malicious tool is registered on a portal of MCP services, appearing to be a genuine service — for example, banking.com and bankin.com. Part of the problem here is that there are a number of MCP registries/markets, but the governance they have and use to mitigate abuse varies, and as this report points out, those with stronger governance tend to have smaller numbers of services registered. This isn’t a new problem; we have seen it before with other types of repositories (PyPI, npm, etc.). The difference here is that the attacker could install malicious logic, but also implement identity theft, where a spoofed service mimics the real service’s need for credentials. As the UI is likely to be primarily textual, it is far easier to deceive (compared to, say, a website, where the layout is adrift or we inspect URIs for graphics that might give clues to something being wrong). A similar vector is Tool Name Conflict, where the tool metadata provided makes it difficult for the LLM to distinguish the correct tool from a spoofed one, leading the LLM to trust the spoof rather than the user.

Another vector, which looks a little like search engine gaming (additional text is hidden in web pages to help websites improve their search rankings), is Preference Manipulation Attacks, where the tool description can include additional details to prompt the LLM to select one solution over another.

The last aspect of MCP attacks I wanted to touch upon is that, as an MCP tool can provide prompts or LLM workflows, it is possible for the tool to co-opt other utilities or tools to action the malicious operations. For example, an MCP-provided prompt or tool could ask the LLM to use an approved FTP tool to transfer a file, such as a secure token, to a legitimate service, such as Microsoft OneDrive, but rather than an approved account, it is using a different one for that task. While the MCP spec says that such external connectivity actions should have the tool request approval, if we see a request coming from something we trust, it is very typical for people to just say okay without looking too closely.

Even with these few illustrations, tooling interaction with an LLM comes with deceptive risks, partially because we are asking the LLM to work on our behalf, but we have not yet trained LLMs to reason about whether an action’s intent is in the user’s best interests. Furthermore, we need to educate users on the risks and telltale signs of malicious use.

Attack Vector Summary

The following list provides a brief summary of the attack vectors. The original paper examines each in greater depth, illustrating many of the vectors and describing possible mitigation strategies. While many technical things can be done. One of the most valuable things is to help potential users understand the risks, use that to guide which MCP solutions are used, and watch for signs that things aren’t as they should be.

Continue reading →

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit v4 the big news

17 Thursday Apr 2025

Posted by mp3monster in Books, Fluentbit, General, Technology

≈ Leave a comment

Tags

book, CNCF, development, eBPF, Fluent Bit, logs, OpenTelemetry, processors, sampling, Security, Trace, zig

With the announcement of Fluent Bit v4 at Kubecon Europe, we thought it worthwhile to take a look at what it means, aside from celebrating 10 years of Fluent Bit.

Firstly, normally using Semantic Versioning would suggest likely breaking (or incompatible changes to use SemVer wording) changes. The good news is that, like all the previous version changes for Fluent Bit the numbering change only reflects the arrival of major new features.

This is good news for me as the author of Logs and Telemetry with Fluent Bit, as it means the book remains entirely relevant. The book obviously won’t address the latest features, but we’ll try to cover those here as supplemental content.

Let’s reflect upon the new features, their benefits, and their implications.

  • New features in Processors allowing:
    • Conditionality to be included.
    • Trace sampling.
  • More flexible support for TLS (v1.3, choosing ciphers to enable)
  • New language for custom plugins in the form of Zig

Security Improvements

While security for many is not something that will get most developers excited about, there are things here that will make a CSO (Chief Security Officer) smile. Any developer who knows implementing security behaviors because it is a good thing, rather than because you have been told to do it, makes a CSO happy, puts them in a good place, to be given some more lianency when there is a need to do something that would get the CSO hot under the collar. Given this, we can now win those points with CSOs by using new Fluent Bit configurations that control TLS versions (1.1 – 1.3) and ciphers to support in use.

But even more fundamental than that are the improvements around basic credentials management. Historically, credentials and tokens had to be explicit in a configuration file or referenced back to an environment variable. Now, such values can come from a file, and as a result, there is no explicitness in the configuration. File security can manage access and visibility of such details. This will also make credentials rotation a lot easier to implement.

Processor Improvements

The processor improvements are probably the most exciting changes. Processors allow us to introduce additional activities within the pipeline as part of a process such as an input, rather than requiring additional buffer fetch and return which we see in standard plugin operations.

Of course, the downside is that if the processor introduces a lot of effort, we can create unexpected problems, such as back pressure, for example, as a result of a processor working hard on an input.

The other factor that extending processors bring is that they are not supported in classic format, meaning that to exploit such formats, you do need to define your configuration using YAML. The only thing I’m not a fan of, is that the configuration for these features does make me think I’m having to read algorithms expressed with Backus Naur form (BNF).

Trace Sampling

Firstly, the processors supporting OpenTelemetry Tracing can now sample. This is probably Fluent Bit’s only weakness in the Open Telemetry domain until now. Sampling is essential here as traces can become significant as you track application executions through many spans. When combined with each new transaction creating a new trace, traces can become voluminous. To control this explosion on telemetry data, we want to sample traces, collecting a percentage of typical traces (performance, latency, no errors, etc) and the outliers, where tracing will show us where a process is suffering, e.g., an end-to-end process is slowing because of a bottleneck. We can dictate how the sampling is applied based on values of existing attributes, the trace status, status codes, latencies, the number of spans, etc.

Conditionality in Processors

Conditionality makes it easier to respond to aspects of logs. For example, only when the logging payload has several attributes with specific values do we want to filter the event out for more attention. For example, an application reporting that it is starting up, and logs are classified as representing an error – then we may want to add a tag to the event so it can be easily filtered and routed to the escalation process.

Plugins with Zig

The enablement of Zig for plugin development (input, output and filters) is strictly an experimental feature. The contributors are confident they have covered all the typical use cases. But the innate flexibility supporting a language always represents potential edge cases never considered and may require some additional work to address.

Let’s be honest: Zig isn’t a well-known language. So, let’s start by looking briefly at it and why the community has adopted it for custom plugin development as an alternative to the existing options with Lua and WASM.

So Zig has a number of characteristics that align with the Fluent Bit ethos better than Lua and WASM, specifically:

  • It is a compiled rather than interpreted language, meaning that we reduce the runtime overheads of an interpreter or JIT compiler such as Lua and the proxy layer of WASM. This aligns to be very fast/minimal compute overhead to do its job, – ideal for IoT and minimising the cost of side-care container deployments.
  • The footprint for the Zig executable is very, very small—smaller than even a C-generated binary! As with the previous point, this lends itself to common Fluent Bit deployments.
  • The language definition is formally defined, compact, and freely available. This means you should be able to take a tool chain from anyone, and it is easy for specialist chip vendors to provide compilers.
  • Based on those who have tried, cross-compiling is far easier to deal with than working with GCC, MSVC, etc. Making it a lot easier to develop with the benefits we want from Go. Unlike Go – to connect to the C binary of Fluent Bit doesn’t require the use of a translation layer.

One of Zig’s characteristics that differs from C is its stronger typing and its approach of, rather than prescribing how edge cases are handled, e.g., null pointers, working to prevent you from entering those conditions.

Zig has been around for a few years (the first pre-release was in 2017, and the first non-pre-release was in August 2023). This is long enough for the supporting tooling to be pretty well fleshed out with package management, important building blocks such as the HTTP server, etc.

While asking a large enterprise with more conservative approaches to development (particularly when IT is seen as an overhead, and source of risk rather than a differentiator/revenue generator) to consider adopting Zig could be challenging compared to adopting, say Go. The different potential values here, make for some interesting potential.

Not Only, but Also

While we have made some significant advancements, each Fluent Bit release brings a variety of improvements in its plugins. For example, working with it with eBPF, HTTP output supports more compression techniques, such as Snappy and ZSTD, and Exit having a configurable delay.

The Plus version of library dependencies is being updated to exploit new capabilities or ensure Fluent Bit isn’t using libraries with vulnerabilities.

Additional resources

    • Chronosphere announcement

    • Announcement YouTube video

    • book
    • My links to technical resources – we’ve extended to include Zig related resources

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Fluent Bit and AI: Unlocking Machine Learning Potential

30 Monday Dec 2024

Posted by mp3monster in Fluentbit, General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, Cloud, Data Drift, development, Fluent Bit, GenAI, Machine Learning, ML, observability, Security, Technology, Tensor Lite, TensorFlow

These days, everywhere you look, there are references to Generative AI, to the point that what have Fluent Bit and GenAI got to do with each other? GenAI has the potential to help with observability, but it also needs observation to measure its performance, whether it is being abused, etc. You may recall a few years back that Microsoft was trailing new AI features for Bing, and after only having it in use for a couple of days, it had been recorded generating abusive comments and so on (Microsoft’s Tay is such an example).

But this isn’t the aspect of GenAI (or the foundations of AI with Machine Learning (ML)) I was thinking about. Fluent Bit can be linked to GenAI through its TensorFlow plugin. Is this genuinely of value or just a bit of ‘me too’?

There are plenty of backend use cases once the telemetry has been incorporated into an analytics platform, for example:

  • Making it easy to query and mine the observability data, such as natural language searching – to simplify expressing what is being looked for.
  • Outlier / Anomaly detection – when signals, particularly metrics, diverge from the normal patterns of behavior, we have the first signs of a problem. This is more Machine Learning than generative AI.
  • Using AI agents to tune monitoring thresholds and alerting scenarios

But these are all backend, big data style use cases and do not center on Fluent Bit’s core value of getting data sources to appropriate destination systems for such analysis or visualization.

To incorporate AI into Fluent Bit pipelines, we need to overcome a key issue – AI tends to be computationally heavy – making it potentially too slow for streams of signals being generated by our applications and too expensive given that most logs reflecting ‘business as usual’ are, in effect, low value.

There are some genuine use cases where lightweight AI can deliver value. First, we should be a little more precise. The TensorFlow plugin is the TensorFlow Lite version, also known as LiteRT. The name comes from the fact that it is a lite-weight solution intended to be deployable using small devices (by AI standards). This fits the Fluent Bit model of having a small footprint.

So, where can we put such a use case:

  • Translating stack traces into actionable information can be challenging. A trained ML or AI model can help classify and characterize the cause of a stack trace. As a result, we can move from the log to triggering appropriate actions.
  • Targeted use cases where we’ve filtered out most signal data to help analyze specific events – for example, we want to prevent the propagation of PII data downstream. Some PII data can be easily isolated through patterns using REGEX. For example, credit card IDs are a pattern of 4 digits in 4 groups. Phone numbers and email addresses can also be easily identified. However, postal addresses aren’t easy, particularly when handling multinational addresses, where the postal code/zip code can’t be used as an indicative pattern. Using AI to help with such checks means we must filter out signals to only examine messages that could accidentally carry such information.

When adopting AI into such scenarios, we have to be aware of the problems that can impact the use of ML and AI. These use cases are less high profile than the issues of hallucinations but just as important. As we’re observing software, which will change over time. As a result, payloads or data shifts (technically referred to as data drift) and the detection rate can drop. So, we need to measure the efficacy of the model. However, issues such as data drift need to be taken into account, as the scenario being detected may change in volume, reflecting changes in software usage and/or changes in how the solution works.

There are ways to help address such considerations, such as tracking false positive outcomes, and if the model can provide confidence scoring, is there a trend in the score?

Conclusion

There are good use cases for using Machine Learning (and, to an extent, Artificial Intelligence) within an observability pipeline – but we have to be selective in its application as:

  • The cost of the computation can outweigh the benefits
  • The execution time for such computation can be notably slower than our pipeline, leading to risks of back pressure if applied to every event in the pipeline.
  • The effectiveness and how much data drift might occur (we might initially see very good results, but then things can fall off).

Possibly, the most useful application is when the AI/ML engine has been trained to recognize patterns of events that preceded a serious operational issue (strictly, this is the use of ML).

Forward-looking

The true potential for Gen AI is when we move beyond isolating potential faults based on pattern recognition to using AI to help recommend or even trigger remediation processes.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
← Older posts
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d