Are we heading to a Certificate-apocalypse?

Tags

, , , , , , , , , , , ,

Most engineers are probably aware of the shortening life span of TLS certificates. this year certificate maximum life drops from 398 to 200 days. But what may not have registered is that the roadmap sees the max certificate time will continue to half every year. This means that by 2029 certificate lifecycle will be down to 47 days (31 days (core duration) + core duration/2 days + 1 day). But before 2029, in less than six months we’ll see the current certificate durations half from 200 to 100 days. As with the 2026 change, the new limits come into effect on the 15th March. Which does create an odd situation where certificates issued on the 14th March will outlast certificates issued on the following day.

The addition of half the core duration period is to allow for the recovery from failures. as of 2029 the expectation is every month the certificates should be rotated.

Manual processes are unsustainable

This means that the rotation of certificates has got to be accelerated to be monthly, a velocity that is simply not practical to be done by hand. This means the deployment of certificates really needs to be automated. There are a number of frameworks and standards that support the distribution of certificates.

Perhaps the best known and leading solution has been ACME protocol working in conjunction with Let’s Encrypt CA. The least known is the use of OpAMP also supports certificate distribution. The following table shows the different standards that specifically address certificate management (CLM), some implementations are linked to a Certificate authority , others are not.

StandardSpecificationOpen-source implementationsCommercial implementations
ACMERFC 8555Certbot (client), acme.sh (client), lego (client), step-ca (server), Boulder (server)DigiCert, Sectigo, GlobalSign, EJBCA Enterprise
CMP / CMPv3RFC 9810EJBCA Community, Bouncy Castle (library)EJBCA Enterprise, DigiCert Trust Lifecycle Manager
ESTRFC 7030Cisco libEST (client/server library), EJBCA Community (edition-dependent)EJBCA Enterprise, DigiCert
SCEPRFC 8894MicroMDM SCEP, EJBCA CommunityMicrosoft Intune, EJBCA Enterprise, DigiCert
CMCRFC 5272Dogtag PKICommercial implementations are less prominent; verify explicit CMC support and supported operations with individual PKI vendors

To understand the differences between the different standards, it is worth looking at the standards from the perspective of the operations that are needed to manage certificates.

FunctionACMECMPESTSCEPVendor
CLM API
Certificate issuanceYesYesYesYesUsually
Certificate renewalYesYesYesYesUsually
Certificate revocationYesYesNo native operationLimitedUsually
CA certificate rolloverLimitedYesYesLimitedVaries
Central certificate inventoryNoNoNoNoUsually
Ownership / application mappingNoNoNoNoVaries
Policy managementLimitedLimitedLimitedLimitedUsually
Application deploymentNoNoNoNoVendor-specific
Application activationNoNoNoNoThrough integrations
Deployment verificationNoNoNoNoVaries

You’ll note that the deployment, activation and verification steps fall into the domain of vendor solutions, and are variable. However we can address such considerations in an open-source (or commercial) manner through tooling such as Puppet, Chef and certain payload types provided by OpAMP.

These CLM options aren’t the end of the story as you may wish to look more broadly at no just the request and distribution of certificates, but the mechanisms through which the certificate is stored, and there is an overlap with such management with keys as well as certificates. For example APIs may use keys of certificates to help verify authenticity of the request. This also extends to mTLS operations where the client needs certificate to verify the server requests. For example KMIP provides an interface that can support cryptographic object (keys, certificates (public, private, symmetric), secret data, PGP Keys).

StandardSpecificationOpen-source implementationsCommercial implementations
KMIPOASIS KMIPPyKMIP (Python client/server)HashiCorp Vault Enterprise, Thales CipherTrust
PKCS#11PKCS#11 v3.1SoftHSM, OpenSCHashiCorp Vault Enterprise, Thales Luna HSM
SCIMRFC 7644elimity SCIM (Go library)Microsoft Entra ID, Okta

Managing the impact

There are several impacts that need to be considered:

  • tracking certificate expiry
  • Last mile from client deployer to server folder and any refresh nudge required

Tracking certificate expiry

Traditionally certificate expiry management could be handled as a manual process – after all if you have a log of which systems had certificates then if you simply reviewed the list once a month, you could spot expiries with time to spare. Even if the certificate distribution process was automated across many nodes. This just isn’t going to work, and the contingency time incorporated into the certificate life is contracting. Rather than trusting to a tracker spreadsheet being managed we should really focus on actual certificate data. This points to having a means by which application observability has a reporting metric on when a certificate will expire (date or a number of days). The later will prove to be easier as a metrics measure, but the value needs to be in context of when the metric was collected.

Last mile challenge

While automation frameworks like Certbot and step-ca can drive the certificate issuance. These processes get the certificate to the relevant server. But we still need to move the certificate intro the location the application expects the certificate to be, and do we need to nudge the server to reload the certificate? Certbot for example expects to have a script to trigger for this action. This can get a little messy to deploy and maintain, particularly if there are subtle variances in deployments (as a result of manual application deployment in legacy environments), configuration creep etc. One possible way to simplify this is to leverage the capabilities of environment scanners to locate apps and their configs. Then use that to drive the auto creation of that last mile.

Why the change?

The history for the reduction has largely been driven by browser providers particularly Google and Apple, and their security concerns about the trustworthiness of the web’s public key infrastructure (PKI). Historically, certificates could remain valid for several years, creating a prolonged window of exposure if a private key was compromised, a certificate incorrect certificates being wrongly used (organization or domain information became outdated – fore example domain name changes, acquisition of services etc). Certificate revocation mechanisms, particularly CRLs and OCSP, have not proven to be reliable as a means of ensuring browsers quickly stop trusting compromised certificates. Shorter lifetimes reduce this exposure and encourage the automation of certificate management.

The driving of changes has shifted from browser providers unilaterally declaring they won’t trust certificates that are older than a certain age, to a more formalized process governed primarily by the CA/Browser Forum, (an independent body bringing together Certificate Authorities and browser vendors).

Useful resources

  1. A/Browser Forum — TLS Baseline Requirements, version 2.3.1 (4 October 2026), section 6.3.2. Authoritative current hard limits: 398 days before 15 March 2026; 200 days from that date; 100 days from 15 March 2027; 47 days from 15 March 2029. Section 1.1 and the revision history identify 1 July 2012 as the original effective date. https://cabforum.org/working-groups/server/baseline-requirements/requirements/
  2. CA/Browser Forum — FAQ for Baseline Requirements. Historical summary of the 60-month limit from July 2012 and the 39-month general limit from April 2015. This is a historical source, not the current rule set. https://cabforum.org/faq-about-the-baseline-requirements/
  3. CA/Browser Forum — Ballot 193: 825-day Certificate Lifetimes (17 March 2017). Establishes the March 2018 transition to 825 days; also records the preceding 39-month rule and the qualifying 60-month legacy exception through June 2016. The page contains amendment markup; read the replacement wording in context. https://cabforum.org/2017/03/17/ballot-193-825-day-certificate-lifetimes/
  4. Apple — About upcoming limits on trusted certificates. Announces the 398-day maximum for public TLS server certificates issued on or after 1 September 2020 and explains its scope. This change originated in browser/platform trust policy. https://support.apple.com/en-us/102028
  5. CA/Browser Forum — Ballot SC081v3: Introduce Schedule of Reducing Validity and Data Reuse Periods (11 April 2025). Records the successful vote for the 2026–2029 reductions; source 1 confirms the resulting current requirements. The April milestone is the vote result, not the end of the subsequent intellectual-property review period. https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/
  6. IETF Datatracker — draft-barnes-acme-00 (28 January 2015). First revision of the ACME Internet-Draft, before working-group standardization. https://datatracker.ietf.org/doc/draft-barnes-acme/00/
  7. Let’s Encrypt / ISRG — Entering Public Beta (3 December 2015). Announces public availability without invitations. https://letsencrypt.org/2015/12/03/entering-public-beta
  8. Electronic Frontier Foundation — Announcing Certbot: EFF’s Client for Let’s Encrypt (May 2016). Explains the rename, the EFF home, and ACME-based issuance and configuration automation. https://www.eff.org/deeplinks/2016/05/announcing-certbot-new-tls-robot
  9. Let’s Encrypt community — New name (Certbot) for Python Let’s Encrypt client (12 May 2016). Dated contemporary announcement linking to EFF’s announcement; supports the exact event date. https://community.letsencrypt.org/t/new-name-certbot-for-python-lets-encrypt-client/15688
  10. RFC Editor — RFC 8555: Automatic Certificate Management Environment (ACME), March 2019. Standards-track specification, published as a Proposed Standard. https://www.rfc-editor.org/info/rfc8555/
  11. ACME — RFC 8555: Automated Certificate Management Environment; supports automated certificate issuance, renewal and revocation.
  12. CMP / CMPv3 — RFC 9810: Certificate Management Protocol; supports PKI enrolment and certificate lifecycle operations.
  13. CMP over HTTP — RFC 9811: Defines HTTP transport for CMP messages.
  14. EST — RFC 7030: Enrollment over Secure Transport; supports certificate enrolment and re-enrolment over HTTPS.
  15. SCEP — RFC 8894: Simple Certificate Enrolment Protocol; commonly used for network devices.
  16. CMC — RFC 5272: Certificate Management over CMS; supports certificate requests and management messages. Updated to — RFC 10002: Updated specification superseding RFC 5272.
  17. KMIP 2.1 — OASIS KMIP 2.1 Specification: Key Management Interoperability Protocol; manages cryptographic keys, certificates and other cryptographic objects.
  18. PKCS#11 3.1 — OASIS PKCS#11 3.1 Specification: Cryptographic Token Interface; standard API for accessing cryptographic hardware and software tokens.
  19. SCIM 2.0 — RFC 7644: System for Cross-domain Identity Management; supports identity and account provisioning rather than certificate management.
  20. SPIFFE — SPIFFE Specifications: Secure Production Identity Framework for Everyone; standardizes workload identity across heterogeneous systems.
  21. X.509-SVID — X.509-SVID Specification: Defines X.509 certificates used for workload identity and authentication.
  22. X.509 PKIX — RFC 5280: Defines the Internet X.509 public key certificate and Certificate Revocation List (CRL) profile.
  23. CA/Browser Forum Baseline Requirements — Current Baseline Requirements: Industry requirements governing publicly trusted TLS certificate issuance, domain validation, security practices and maximum certificate validity periods.

Can patches have functional impacts?

Tags

, , , , , , , , ,

With the recent upheaval from the newly discovered PHP vulnerability (often referred to as PHP2Shell), and updates to OpenTelemetry declarations, it struck me that it’s worth highlighting an assumption we all tend to make with patches.

We tend to think patches introduce fixes or defences that mean functional behavior is immutable. It is an understandable reading based on definitions such as Semantic Versioning (semver):

PATCH version when you make backward compatible bug fixes

However, languages and frameworks point towards a different model, where defined API behaviour will be respected.

The bottom line is if you’re exploiting undocumented behaviors, it makes it all the more necessary to test and ideally in an automated manner changes not only in the applications but also for the stack.

If you are taking advantage of undocumented behavior, then it is worth adding an abstraction layer to allow mitigation changes to be added.

The following identifies the statements at a language level relating to what impact a patch can have:

Language / runtimeBehavior changes in patchesBreaking API changes in patchesPolicy, limitations and supporting documentation
GoYesExceptionalExplicit security, unspecified-behavior and specification-correction exceptions. Go compatibility promise
PHPYes, for correctionsGenerally noSecurity and bug fixes permitted; public userland API must remain backward compatible. Internal ABI/API has narrowly defined exceptions. PHP release policy
Python (CPython)Yes, restrictedExceptionalReasonable bug fixes may affect behavior; stronger incompatibilities require exceptional justification. PEP 387
Java (OpenJDK / Oracle JDK)YesGenerally noSecurity corrections can alter runtime behavior; source, binary and behavioral compatibility are evaluated separately. OpenJDK compatibility
C# / .NET runtimeExceptionalExceptionalBreaking changes generally prohibited during servicing, except for security vulnerabilities or other critical issues. .NET release policy
F# / .NET runtimeExceptionalExceptionalShares .NET runtime servicing policies; compiler and language tooling have their own compatibility considerations. .NET compatibility rules
VB.NET / .NET runtimeExceptionalExceptionalSame underlying .NET servicing guarantees. .NET servicing
TypeScriptYes, restrictedNot normallyPatch releases may contain high-priority regression, performance-regression and safe language-service corrections. Ordinary breaking changes are targeted at larger releases. TypeScript release process
JavaScript (ECMAScript)Not applicable at language-specification levelNot applicableECMAScript is a specification, not a separately patched runtime. Behavior depends on implementations such as V8, SpiderMonkey, JavaScriptCore and Node.js. ECMAScript specification
Node.jsYes, for correctionsExceptionalPatch releases accept fixes; semver-major changes are excluded under normal procedures. LTS security exceptions can affect native API/ABI compatibility when unavoidable. Node.js release procedures

The following table, takes the same approach looking at how operating systems view changes for patches:

Operating systemBehavior changes in maintenance patchesBreaking API / ABI changesPolicy, limitations and supporting documentation
Red Hat Enterprise Linux (RHEL)YesGenerally avoidedBackports security and bug fixes into supported versions. Red Hat explicitly tests for unwanted side effects, so this is not general permission to break compatibility. RHEL security backporting
SUSE Linux Enterprise Server (SLES)YesGenerally avoidedMaintenance includes security corrections and defect fixes; individual components have different lifecycle rules. Service packs allow broader changes. SUSE support policy and SLES release notes
Oracle LinuxYesNo general permission establishedOracle distributes security, bug-fix and enhancement errata as package updates. Security hardening can alter functionality independently of major OS releases. Oracle Linux errata policy
Ubuntu LTSYes, cautiouslyGenerally avoidedBackported security patches prioritize compatibility. Exceptional changes and certain feature updates can be accepted under the Stable Release Updates process. Ubuntu security updates
Debian StableYes, restrictedGenerally avoidedImportant bug and security fixes are acceptable; updates are deliberately minimal and require review. Debian Stable update rules
Microsoft Windows 11 / Windows ServerYesNo general permission establishedCumulative security, quality and out-of-band updates may change functionality. Windows 11 may also deliver feature enhancements through monthly servicing. Windows update release policy
Apple macOSPossibleNo general permission establishedSecurity and maintenance releases can affect operating behavior, but there is no general published patch-version exception equivalent to Go’s. Apple security releases

Often when building solutions we don’t see low level frameworks and how they work, so we may not know about any undocumented feature exploitation.

The challenge it can bring

This points to increased testing, which ideally is automated. The reaction is initially is so what. But, when vulnerabilities like PHP come up which had openly triggered systems being disconnected from the net.

So extending testing (even if automated) is going to experience resistance from the commercial parts of an organization who are unlikely to understand the nuances, but will react to headlining risks.

How do we move forward?

There are some strategies we can adopt that can help us. The industry used to talk about security in depth and, more recently, system resilience. Regardless of terminology, we want to introduce control layers that reduce the chance of malicious intent. Often referred to as compensating controls.

For example, if the vulnerability is overloading the runtime by sending an oversized payload, a firewall or proxy layer can have rules that prevent payloads from being too big from hitting the backend.

If the networking allows it, then segmenting the network reduces the risk of a breach affecting other environments.

This doesn’t mean if we can patch, that we shouldn’t. What it does mean is we can be considered and not knee-jerk panic and unwittingly introduce new problems.

Conclusion

Adding additional security and proxy layers and segmenting system environments may seem like overkill, particularly when introducing new systems. But these steps create the means to be far more resilient.

OpAMP demo

Tags

, , , , , , , , , , , , ,

Last week saw the Open Source Observability Day (OSOD) during which I presented on the subject of the Open Agent Management Protocol, covering the underlying protocol that forms the foundation of a wider capability we have been building along with a demo of what is available.

If you want to see the presentation, and hear the full story not only of the core protocol but how it can help with the ideas around ChatOps and AIOps head over to the OSOD website (and previous years can be seen on YouTube with playlists for each year e.g. OSOD 2025). Of course if you have been following my blog posts you’ll know this.

The demo is recorded against a version prior to the 0.6 release.

Running the demo yourself

If you want to try this out for yourself, you can do so. You’ll need to:

  • Deploy the OpAMP services,
  • Setup the connectivity to Slack, and AI if you want to explore the chat ops capabilities.
  • Deploy the relevant agents (we’ve go something in the pipeline to simplify this)
  • Download the demo configuration and adjust based on your deployment locations.

Get the configuration paths and security tokens for Slack and AI, and firing up the demo is very simple using the CLI.

Moving Open Source Observability closer to turn-key solutions

Tags

, , , , , , , , , , , , , , ,

There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:

  • Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
  • Deployed services and applications with default monitoring configuration
  • Prebuilt dashboards

While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.

This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.

Detecting and Generating Configuration

The process we need has the following steps:

  • find the deployed solution(s)
  • Understand how they are deployed
  • Generate a configuration based on the detection
  • Push the configuration out to agents

Detect and capture config

One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.

While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …

As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.

Templates make the result reusable

By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.

There are limits

So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.

Convenience as a weakness?

While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.

Backend Bootstrapping

Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.

Useful Links

OpAMP V0.6 implementation released

Tags

, , , , , , , ,

We’ve made the latest version of our implementation of the OpAMP specification available. It has taken longer to ready than the previous versions, but there is a lot been included, both in the core functionality, but also capabilities supporting the use and setup. Here are the headlines…

  • The project has its own organization now (https://github.com/orgs/opamp-observe/) as mentioned here, but also a domain of https://opamp.info/ which will bounce you to the GitHub pages.
  • Automated regression from clean to deployed and running a series of end-to-end tests as well as producing a guaranteed fresh set of artefacts has been implemented. This can be run against AWS (currently in use) or Azure (needs refining). These are triggered on changes to the main branch.
  • GitHub pages with links all the documentation are established now.
  • The Python wheel release artefacts are no incorporated into the release assets.
  • The configuration for the Open Source Observability Day conference demo is available in the https://github.com/opamp-observe/demo-repository repo.
  • We’ve added support for a couple more open-source observability tools:
  • The client/consumer side has been refactored so it is easy to build your own observability agent controls and independently deploy them (not supported by the config editor).
  • The CLI has been enhanced so it can help quickly spin up more advanced configurations – helpful for demos and real deployments.
  • The Dev CLI has been improved.
  • The server side now has a basic client side (Supervisor/Observer) configuration editor UI. Which can be deployed as a plugin.
  • Improvements around the use of Podman and Docker for the containerization.
OpaMP v0.6 now labelled

What next?

We’ve added to the documentation pages for the roadmap, initially we’re working to fill out the regression test sets at the moment. Other possibilities:

  • Expand the client types
  • Extend the client types to exploit more features.

Perhaps the most interesting idea, we’re thinking about is can we exploit tools like Netdata to discover what solutions are deployed and generate a requirements/configuration definition which can be used to generate a suitable agent configuration file.

Observability as a security sensor using Fluent Bit, OpenTelemetry and AIDE

Tags

, , , ,

Observability is usually looked at from the perspective of applications and their needs and the cloud native platforms. Nothing wrong with that, but it is also a rather narrow view of the telemetry we are already collecting or are very capable of collecting. Our observability agents (Fluent Bit, Fluentd, Elastic Agent …) can collect signals (logs, metrics and traces) that can contribute to the security operations within an enterprise. More importantly, the same open-source tooling can help answer a deceptively simple question such as has someone changed the system, or changed the tooling that tells us what the system is doing?

When it comes to security, that second part matters. An attacker does not always need to exploit a new vulnerability. If they can alter a service definition, scheduled task, access-control file, logging configuration or telemetry destination, they can prevent us from sensing what damage they maybe doing, including opening up simpler or more invasive routes into your systems. MITRE ATT&CK describes this kind of action as a defence-evasion technique. Research from Palo Alto Networks Unit 42 has also described cloud logging services as an attack surface, including changes that stop, redirect or selectively reduce audit collection. Not only that getting at logs, means they can potentially determine when systems that are running, and find new attack vectors.

As you can see being able to detect configuration and file manipulation is an important factor for security. Many may react to this with concern about introducing cost for expensive security tools, but if there are simple means to listen to OS activities that relate to file manipulation then we can leverage our observability estate for this aspect of security.

The wider open-source stack

That neutrality for our observability agents is valuable for security. A host or cluster can emit operational and security telemetry through the same managed pipeline, while routing data to one or more destinations such as OpenSearch, Loki, ClickHouse, a SIEM, object storage or an OTLP-compatible service. Not to mention if the event is for an application configuration, we can evaluate it against application and service viewpoints such as managing configuration drift.

The roles are complementary:

  • AIDE is a open-source Linux utility that provides the file-integrity check. It builds a known-good database and reports additions, removals and changes to selected files and attributes. Aide achieves this through the use of the iNodes API, which if you preferred could also be tapped into even more directly.
  • Fluent Bit or other OTel compliant tools collect the result close to the workload, adds context, buffers during a network interruption and routes the event. Our backend services which are in all likelihood OTLP capable, means the events can be send nearly anywhere for analysis and alerting
  • The analytical backend correlates the integrity event with logins, process starts, deployments, traces and other activity, then drives an alert or investigation.

AIDE is deliberately focused. It does not need to become a SIEM and Fluent Bit does not need to become a file-integrity engine. Combining small tools through open interfaces lets each component do the job it is good at.

We can use Aide not only to monitor our configurations, but when the deployed binaries are disturbed. We can extend this setup further. By exploiting custom commands with OpAMP (particularly for a supervisor) can be used to not only manage the observability agent, but also manage the deployment of configuration to tools like Aide. Now if we think something is being abused, the option to change our file integrity monitoring is easily in our reach.

Why file integrity belongs in the observability conversation

The available incident statistics do not normally include a neat category called “configuration-file manipulation”. The behaviour is spread across techniques such as creating or modifying system processes, scheduled tasks, startup mechanisms, accounts, policies and logging controls. That makes a single adoption or attack percentage difficult to defend. It does not make the risk rare.

For example, Google Cloud’s M-Trends 2025 reporting included techniques such as Create or Modify System Process (20.6%), Account Manipulation (19.9%), Scheduled Task/Job (14.3%), Registry Run Keys/Startup Folder (11.5%) and Web Shell (7.2%) among observed investigations. These are broader ATT&CK behaviours, not a count of configuration-file changes, but many are enabled or persisted through changes to system configuration.

There is also a compliance driver. PCI DSS v4.0.1 Requirement 11.5.2 explicitly requires a change-detection mechanism, such as file-integrity monitoring, for critical files, with comparisons at least weekly and alerts for unauthorised modification. NIST SP 800-53 Revision 5 control SI-7 requires monitoring for unauthorised changes to software, firmware and information. Other standards may describe the required outcome as configuration control, integrity, audit protection or system monitoring rather than naming a particular product.

FIM helps provide the evidence behind those outcomes:

  • what changed;
  • which host or workload reported it;
  • when the change was detected;
  • which attributes or content hashes changed;
  • whether a change matched an approved deployment window;
  • whether the alert was acknowledged and investigated; and
  • whether the integrity baseline was updated through a controlled process.

The last point is easily overlooked. Automatically accepting every change into the baseline produces a very tidy report and a very weak control.

What should be monitored?

Starting with every file on every host usually creates cost and noise. I would start with the files that can change control flow, identity, collection or evidence retention:

  • Fluent Bit configuration, parser files and included fragments;
  • Vector configuration and transformation definitions;
  • OpenTelemetry Collector configuration;
  • systemd units and drop-ins for telemetry agents;
  • log rotation and retention configuration;
  • TLS trust stores, client certificates and references to credentials;
  • auditd, syslog and journald configuration;
  • scheduled tasks, startup scripts and authorised SSH keys;
  • alert rules, dashboards and detection content managed as files; and
  • the AIDE configuration and baseline database itself.

Runtime state, offset databases, buffers and normal log files should normally be excluded. They are expected to change and monitoring them obscures the files that should remain stable.

There is an architectural wrinkle: if AIDE reports through Fluent Bit, and an attacker changes both AIDE and Fluent Bit, the local signal can be suppressed. The practical response is defence in depth. Restrict write access, ship events off-host quickly, monitor the agent configuration and service unit, protect or sign the AIDE baseline, and alert when expected heartbeats or scheduled check results disappear. Silence is also a signal.

Example: AIDE reporting through Fluent Bit and OTLP

The following is a starting point rather than a universal production configuration. Package paths and AIDE defaults differ between Linux distributions, so it should be merged with the supplied configuration rather than replacing it blindly.

First, add an AIDE rule group for the observability configuration. For example, in /etc/aide/aide.conf.d/observability.conf:

# Permissions, inode, links, ownership, size, timestamps and SHA-256 content.
OBS_CONFIG = p+i+n+u+g+s+m+c+sha256
​
/etc/fluent-bit                         OBS_CONFIG
/etc/vector                             OBS_CONFIG
/etc/otelcol-contrib                   OBS_CONFIG
/etc/systemd/system/fluent-bit.service.d OBS_CONFIG
/etc/systemd/system/vector.service.d     OBS_CONFIG
/etc/systemd/system/otelcol.service.d   OBS_CONFIG
/etc/aide                               OBS_CONFIG
​
# These are runtime data and should not be treated as static configuration.
!/var/lib/fluent-bit
!/var/lib/vector
!/var/lib/otelcol
!/var/log

The main AIDE configuration can direct its report to a file that Fluent Bit is permitted to read:

database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout=true
​
report_url=file:/var/log/aide/aide.log
report_level=changed_attributes

Initialise and approve the baseline using the procedure provided by the operating-system package. The baseline database should be writable only by the account performing the controlled update; consider retaining a signed or read-only copy outside the host.

Run aide --check from a systemd timer or the organisation’s scheduler. An hourly check provides better detection latency than the weekly minimum stated by PCI DSS, although the right frequency depends on estate size, I/O cost and risk. The scheduler should also emit a success or failure event so that a missing check can be detected.

Fluent Bit can then tail the report, add security context and forward the records over OTLP. The Tail input uses a database to retain offsets, and the OpenTelemetry output supports OTLP/HTTP and OTLP/gRPC.

service:
flush: 5
log_level: info
storage.path: /var/lib/fluent-bit/storage
pipeline:
inputs:
- name: tail
tag: security.fim.aide
path: /var/log/aide/aide.log
key: body
path_key: log.file.path
db: /var/lib/fluent-bit/aide-tail.db
db.sync: normal
read_from_head: false
refresh_interval: 5
rotate_wait: 30
skip_empty_lines: true
skip_long_lines: true
storage.type: filesystem
filters:
- name: modify
match: security.fim.aide
Add:
- event.domain file
- event.category file
- event.type change
- security.control file_integrity
- service.name aide
outputs:
- name: opentelemetry
match: security.fim.aide
host: otel-collector.internal
port: 4318
logs_uri: /v1/logs
tls: on
tls.verify: on
compress: gzip

This example emits each AIDE report line as a log record. That is simple and robust, but the backend will need to group lines belonging to the same check. For a more mature deployment I would place a small wrapper around the AIDE invocation that converts each detected change into JSON Lines and adds a check identifier, hostname, baseline version and execution result. Fluent Bit can then use its JSON parser and every changed file becomes a structured event.

Vector can implement the same pattern with its file source and an OTLP-capable destination or intermediate Collector. The OpenTelemetry Collector can also collect files using the contributed filelog receiver. The choice is less important than applying consistent fields, reliable buffering, secure transport and an independently managed destination.

From event collection to a security control

Collecting the AIDE output is only the first half of the solution. To turn it into a control, the backend needs useful detection logic and operational ownership.

I would start with alerts for:

  • a change to an observability agent destination, filter or exclusion rule;
  • a change to a telemetry service unit or executable;
  • deletion or replacement of the AIDE database;
  • changes outside an approved deployment window;
  • a burst of changes across several hosts;
  • a configuration change followed by a fall in log volume; and
  • a missing AIDE run, agent heartbeat or expected integrity summary.

Correlating the FIM event with deployment records is important. A checksum change shortly after an approved GitOps rollout is very different from the same change following an interactive root login at 02:00. Observability provides the surrounding evidence that allows the integrity event to be assessed rather than simply counted.

There is also value in feeding an integrity event back into operational telemetry. A changed collector configuration can annotate dashboards, traces and incident timelines. The question changes from “why did the logs stop?” to “did the logs stop when this configuration changed?” That is a much better starting point for both an SRE and a security analyst.

Final thoughts

The observability and security domains have spent years building separate pipelines for data that frequently starts on the same machine. Open standards and focused open-source tools give us an opportunity to reduce that duplication.

AIDE supplies a mature integrity signal. Fluent Bit and Vector make that signal transportable and enrichable. OpenTelemetry makes the route to the wider platform portable. None of these components replaces access control, endpoint detection or a SIEM, but together they add a useful security capability to infrastructure many organisations already operate.

Perhaps the most important configuration to observe is the configuration that decides what we are allowed to observe.

Resources

File Integrity Management using Aide and Fluent Bit

Open Source Observability Day Speaker

Tags

, , , , , , , ,

I’m excited to share that I will be speaking as part of the Open Source Observability Day virtual conference. I’ll be talking about the Open Telemetry specification OPen Agent Management Protocol (or OpAMP). To register for the event go here..

Not only will I be going through the what the spec enables, and the value it aims to support we’ll be touching on how OpAMP can be used to support ChatOps and AIOps. While the idea of unleashing AI Agents onto a production may look pretty scared given all the recent news coming from OpenAI, Anthropic and others on how their latest models are escaping and hacking other web sites. I’ll show how among other things how mixing AIOps nd OpAMP can offer the AI smarts while training deterministic controls.

The session isn’t just slideware, we’ll demo our Open Source OpAMP implementation which gives the foundations for using AIOps. If you want to try the demo yourself we’re in the process of making available in GitHub.

The way we can use OpAMP with AIOps to get the effectiveness of knowledge, and deterministic behaviour for operational change

1st co-authored book (on Oracle Integration) still being promoted

Tags

, , , , , ,

Ten years ago I was starting the co-authoring our first book, which turned out to be the first published on Oracle PaaS. Specifically it was about Integration Cloud Service, which was became Oracle Integration Cloud (OIC). The book was published in the Spring of 2017. It was a rewarding experience, and the first of a series of books written over the last 10 years.

Most technology books, particularly those that are focussed on a specific technology tend to have a relatively short shelf life. – part of the reason publishers have early release programmes.

But our first book has been promoted to us as part of Amazon’s daily Kindle Reads email as you can see here…

While I am certainly not complaining, I am curious as to how Amazon’s algorithm choice the book.

If you’ve purchased the book. Then Thankyou.

OpAMP Project – own org in GitHub

Tags

, ,

As the OpAMP project has grown from a small idea to something pretty substantive, we’ve decided to put it in its own Org and repository with GitHub. This should mean it no longer looks a pet project. The repo name has also been changed as we have address more than just Fluentd and Fluent Bit needs with the Elastic observability pipeline tools having been incorporated with the basic features as well asl Vector.

Git Repo for all OpAMP related content (OpAMP Observe)

The Tool we’ve migrated formerly Fluent-OpAMP now OpAMP-Core

We’ve been checking and correcting links as a result of the changes.

We haven’t migrated the tooling such as the LogGenerator, FluentBit configuration converter to the new Org, but would welcome thoughts on this.

Fluentd & Fluent Bit OpAMP implementation
Fluentd & Fluent Bit OpAMP implementation logo

Could the AI hacks have unintended consequences for AIOps

Tags

, , , , , ,

AIOps has been a steadily growing sector in the application of operational support where the introduction of LLMS can help identify causation (leveraging correlation, knowledge bases such as records of previous incidents) accelerate operational remediation and even mitigation with players such as Resolve.ai, ObserverIQ, and Keep (Open Source).

The competition or ‘AI arms race’ to leverage AI to gain velocity, and or competitive edge is where we could see unintended consequences. A little bit of back ground first.

OpenAI’s attack on HuggingFace

The OpenAI unintended attack on HuggingFace is well documeented (along with similar cases for Meta and Anthropic). Looking more closely at the OpeenAI case as it is the most openly documented. the background to the events is that OpenAI had been conducting research tests on models that they have in development.

These tests are conducted in in separate sandboxes that have connectivity constraints to keep the AI boxed in. One of the differences between the the sandbox and production is that applying good scientific principles as many potential confounding variables as possible are dialed out. Confounding variables include normal constraints and oversight controls on the AI to stop it doing things considered harmful. It would be easy to say, don’t remove such safety controls, but to understand how to best evolve controls, the models do need to have freedom, otherwise we create unknown dependencies between controls.

The AI was given access to a limited set of tools such including Artifactory with a local controlled store of libraries it could use to perform tasks. Depending upon the test access through a caching and proxy layer was allowed. Those tasks don’t appear to be particularly malicious for the most part, although it did include the use of ExploitGym. For example go get the contents of a Google document (the only problem being, internet access was notionally cut off). Some of what happened is a little eye opening, the headlines, I picked up on, and believe could impact people’s views are:

  • Prior to the cyber testing simpler test scenarios such as getting the model to complete several spreadsheet tasks using Google’s Office tools. Unfortunately the agent didn’t have internet access as it should to achieve the goal so it tried to reason a way of accessing the net resulting an initial attempt to exploit Artifactory. While it failed, it did store content into Artifactory.
  • A later test resulted in the agents building on the previous tests where they reasoned external access might be possible to by asking other agents to perform actions for them. This lead to Artifactory being used as a collaboration channel.
  • Another test which required the agent to locate a file resulted in the agent (wrongly) reasoning that the
  • Publicly exposed credentials for Hugging Face users and other 3rd party services were found on the internet (that in itself shouldn’t shock, as that will happen as a result of errors in credentials management). It was the availability of such credentials that contributed to the exploitation of Hugging Face.
  • The AI reasoned that to access the internet it needed to a way out. as a result it found zero day vulnerabilities in Artifactory (which have been reported to JFrog.

The following timeline – tells the story over time.

How does this impact AIOps?

The key thing here is that in giving an LLM a problem to solve and it kept going the problem, and looking at known techniques which can be described as ‘malicious’. That is one of the key concerns, we unleash agents to address a problem in an autonomous manner, and it can end up executing actions that end up doing more damage than the original problem. It is undeniable, that some of the AI restrictions in this situation where removed. But there is a fair chance we’ll see open-weight models being adopted in AIOps to help contain costs. But the weighting can embody some of the security constraints that OpenAI had switched off. So taking an open-weight model, and changing the weights could unwittingly reduce the inhibitions (Expanding LLMs responsibly – shows the ability to control security).

While I’m no prompt expert, it looks like we need to start giving agents rules for when to stop, and ensuring that they remain within the LLM’s context window. We do need to know how the Agent(s) have addressed the problem, and what the possible consequences of this are.

It is human nature to trust things if the out come looks correct. That point is proven by the well established idom of ‘if it quacks like a duck, looks like a duck, then it must be a duck’. This means we’re at risk of trusting the AI has got the solution correct. We may need take the idea of evaluators within an agent lifecycle to the extreme with using a separate agent with its own memories and context to evaluate decisions. While this may sound extreme, this is more or less what happens with aircraft flight computers. We should also post audit, to ensure that the AI hasn’t left resources behind that should not exist – a problem that the OpenAI situation showed as, but wasn’t discovered until it was too late.

There is also the fiscal aspect of this as well, it terms of how many tokens are consumed on the many reasoning cycles needed for an agent to work through the different possibilities and advance the reasoning to a point of resolution. Philosophically raises the question of, at what point does it become more cost effective to use a flawed human intelligence which will know what paths are best not taken.

Ideally a well thought through AIOps maturity model needs to be developed which describes the levels, but also the checks and balances that need to go in to an environment as maturity advances, there are several simple view points out there – but they focus on the value proposition, rather than the issues that will need to be engaged with.

References

AI Ops