Last week saw the Open Source Observability Day (OSOD) during which I presented on the subject of the Open Agent Management Protocol, covering the underlying protocol that forms the foundation of a wider capability we have been building along with a demo of what is available.
If you want to see the presentation, and hear the full story not only of the core protocol but how it can help with the ideas around ChatOps and AIOps head over to the OSOD website (and previous years can be seen on YouTube with playlists for each year e.g. OSOD 2025). Of course if you have been following my blog posts you’ll know this.
The demo is recorded against a version prior to the 0.6 release.
Running the demo yourself
If you want to try this out for yourself, you can do so. You’ll need to:
There are several things that separate the turn-key/commercial observability solutions from the pure open source options. The most visible of these are:
Fleet management of you agents, particularly when working outside of a Kubernetes ecosystem
Deployed services and applications with default monitoring configuration
Prebuilt dashboards
While the issue of fleet management is a subject I’ve covered at length with OpAMP. We’ve not looked so closely at the discovery and automation. While the OpAMP server (and standalone option) provides schema driven editor, the key separation is to drive the configuration based on finding what solutions are deployed.
This process can look very sophisticated, but there are a number of open source tools that can help address it pretty easily. By bringing therse tools together we can achieve the same effect of detect and configure agents. At this stage we’re not looking at the backend of creating out of the box dashboards, but the potential to evolve the approach is there.
Detecting and Generating Configuration
The process we need has the following steps:
find the deployed solution(s)
Understand how they are deployed
Generate a configuration based on the detection
Push the configuration out to agents
Detect and capture config
One of the more active solutions that provides a means to detect deployments, and find configurations in this space is OSQuery, which is used by some serious players (Airbnb, Dropbox, Netflix, Palantir, Etsy, and Uber are just a few). OSQuery is particularly appealing solution because of its ease of use. All the information OSQuery can make available is accessible through the use of SQL including all the power SQL provides to mix data together.
While SQL is powerful, we do want to return the results in an easily consumable manner, so the CLI tool allows us to ask for the result to be returned in JSON. This makes the result easy to work with and we can use it with technologies such as JSONPath, particularly if we can impose a defined schema on the output, something that can be done as we know the queries being run, and can use constructs such as select … as …
As we can query the JSON output to retrieve the relevant parts of the output, for example the result of the queries to look for an Oracle Database or a RedHat/IBM WildFly application server. We can then combine that data with a template, such as the templating capabilities of Jinja2. The template defines the configuration that we would want to use, for example we want to tail log files, we want to establish a Prometheus metrics pull. Then the values are pulled from the JSON to create an instance of the file.
Templates make the result reusable
By using templates, known queries and their result structures we can generate reusable configurations on a product by product basis. The last step is stitching it together. To be reusable each template is discrete for an application or standard stack. So we up with multiple configuration files that need to be combined. This could be addressed by simply merging all the files together, but a lot of the observability tools such as Fluent Bit and Fluentd provide the ability to include files. Obviously you still need to work out what to include to combine, but this pretty straight forward.
There are limits
So this brings together all the sources, for destination side for the turn key solutions is known as the backend is the core product. In the open source solution you’re probably going to run multiple tools, so you need to have a more advanced routing. You can create a standard backend routing and include it.
Convenience as a weakness?
While having turn-key solutions, makes setup so much easier, could it also be a weakness? As we can see with our automation, we’ve not got tailoring to filter data in the pipeline, or park it out of the way. Which ratchets up the costs, because of the retention costs, and greater computational effort to trawl the data. Nor can we derive new signals through correlation inflight. But then our approach with templating creates the same challenge, BUT this approach doesn’t preclude you from incorporating into the solution such smarts.
Backend Bootstrapping
Rapid configuration of the backend is going to be a bit harder because the chances are you’re wanting to mix and match the backend tools (one of the strengths of not going down a turnkey adoption). We’ll explore the possibilities of making this step easier in another post.
We’ve made the latest version of our implementation of the OpAMP specification available. It has taken longer to ready than the previous versions, but there is a lot been included, both in the core functionality, but also capabilities supporting the use and setup. Here are the headlines…
Automated regression from clean to deployed and running a series of end-to-end tests as well as producing a guaranteed fresh set of artefacts has been implemented. This can be run against AWS (currently in use) or Azure (needs refining). These are triggered on changes to the main branch.
GitHub pages with links all the documentation are established now.
The Python wheel release artefacts are no incorporated into the release assets.
The client/consumer side has been refactored so it is easy to build your own observability agent controls and independently deploy them (not supported by the config editor).
The CLI has been enhanced so it can help quickly spin up more advanced configurations – helpful for demos and real deployments.
The Dev CLI has been improved.
The server side now has a basic client side (Supervisor/Observer) configuration editor UI. Which can be deployed as a plugin.
Improvements around the use of Podman and Docker for the containerization.
What next?
We’ve added to the documentation pages for the roadmap, initially we’re working to fill out the regression test sets at the moment. Other possibilities:
Expand the client types
Extend the client types to exploit more features.
Perhaps the most interesting idea, we’re thinking about is can we exploit tools like Netdata to discover what solutions are deployed and generate a requirements/configuration definition which can be used to generate a suitable agent configuration file.
Observability is usually looked at from the perspective of applications and their needs and the cloud native platforms. Nothing wrong with that, but it is also a rather narrow view of the telemetry we are already collecting or are very capable of collecting. Our observability agents (Fluent Bit, Fluentd, Elastic Agent …) can collect signals (logs, metrics and traces) that can contribute to the security operations within an enterprise. More importantly, the same open-source tooling can help answer a deceptively simple question such as has someone changed the system, or changed the tooling that tells us what the system is doing?
When it comes to security, that second part matters. An attacker does not always need to exploit a new vulnerability. If they can alter a service definition, scheduled task, access-control file, logging configuration or telemetry destination, they can prevent us from sensing what damage they maybe doing, including opening up simpler or more invasive routes into your systems. MITRE ATT&CK describes this kind of action as a defence-evasion technique. Research from Palo Alto Networks Unit 42 has also described cloud logging services as an attack surface, including changes that stop, redirect or selectively reduce audit collection. Not only that getting at logs, means they can potentially determine when systems that are running, and find new attack vectors.
As you can see being able to detect configuration and file manipulation is an important factor for security. Many may react to this with concern about introducing cost for expensive security tools, but if there are simple means to listen to OS activities that relate to file manipulation then we can leverage our observability estate for this aspect of security.
The wider open-source stack
That neutrality for our observability agents is valuable for security. A host or cluster can emit operational and security telemetry through the same managed pipeline, while routing data to one or more destinations such as OpenSearch, Loki, ClickHouse, a SIEM, object storage or an OTLP-compatible service. Not to mention if the event is for an application configuration, we can evaluate it against application and service viewpoints such as managing configuration drift.
The roles are complementary:
AIDE is a open-source Linux utility that provides the file-integrity check. It builds a known-good database and reports additions, removals and changes to selected files and attributes. Aide achieves this through the use of the iNodes API, which if you preferred could also be tapped into even more directly.
Fluent Bit or other OTel compliant tools collect the result close to the workload, adds context, buffers during a network interruption and routes the event. Our backend services which are in all likelihood OTLP capable, means the events can be send nearly anywhere for analysis and alerting
The analytical backend correlates the integrity event with logins, process starts, deployments, traces and other activity, then drives an alert or investigation.
AIDE is deliberately focused. It does not need to become a SIEM and Fluent Bit does not need to become a file-integrity engine. Combining small tools through open interfaces lets each component do the job it is good at.
We can use Aide not only to monitor our configurations, but when the deployed binaries are disturbed. We can extend this setup further. By exploiting custom commands with OpAMP (particularly for a supervisor) can be used to not only manage the observability agent, but also manage the deployment of configuration to tools like Aide. Now if we think something is being abused, the option to change our file integrity monitoring is easily in our reach.
Why file integrity belongs in the observability conversation
The available incident statistics do not normally include a neat category called “configuration-file manipulation”. The behaviour is spread across techniques such as creating or modifying system processes, scheduled tasks, startup mechanisms, accounts, policies and logging controls. That makes a single adoption or attack percentage difficult to defend. It does not make the risk rare.
For example, Google Cloud’s M-Trends 2025 reporting included techniques such as Create or Modify System Process (20.6%), Account Manipulation (19.9%), Scheduled Task/Job (14.3%), Registry Run Keys/Startup Folder (11.5%) and Web Shell (7.2%) among observed investigations. These are broader ATT&CK behaviours, not a count of configuration-file changes, but many are enabled or persisted through changes to system configuration.
There is also a compliance driver. PCI DSS v4.0.1 Requirement 11.5.2 explicitly requires a change-detection mechanism, such as file-integrity monitoring, for critical files, with comparisons at least weekly and alerts for unauthorised modification. NIST SP 800-53 Revision 5 control SI-7 requires monitoring for unauthorised changes to software, firmware and information. Other standards may describe the required outcome as configuration control, integrity, audit protection or system monitoring rather than naming a particular product.
FIM helps provide the evidence behind those outcomes:
what changed;
which host or workload reported it;
when the change was detected;
which attributes or content hashes changed;
whether a change matched an approved deployment window;
whether the alert was acknowledged and investigated; and
whether the integrity baseline was updated through a controlled process.
The last point is easily overlooked. Automatically accepting every change into the baseline produces a very tidy report and a very weak control.
What should be monitored?
Starting with every file on every host usually creates cost and noise. I would start with the files that can change control flow, identity, collection or evidence retention:
Fluent Bit configuration, parser files and included fragments;
Vector configuration and transformation definitions;
OpenTelemetry Collector configuration;
systemd units and drop-ins for telemetry agents;
log rotation and retention configuration;
TLS trust stores, client certificates and references to credentials;
auditd, syslog and journald configuration;
scheduled tasks, startup scripts and authorised SSH keys;
alert rules, dashboards and detection content managed as files; and
the AIDE configuration and baseline database itself.
Runtime state, offset databases, buffers and normal log files should normally be excluded. They are expected to change and monitoring them obscures the files that should remain stable.
There is an architectural wrinkle: if AIDE reports through Fluent Bit, and an attacker changes both AIDE and Fluent Bit, the local signal can be suppressed. The practical response is defence in depth. Restrict write access, ship events off-host quickly, monitor the agent configuration and service unit, protect or sign the AIDE baseline, and alert when expected heartbeats or scheduled check results disappear. Silence is also a signal.
Example: AIDE reporting through Fluent Bit and OTLP
The following is a starting point rather than a universal production configuration. Package paths and AIDE defaults differ between Linux distributions, so it should be merged with the supplied configuration rather than replacing it blindly.
First, add an AIDE rule group for the observability configuration. For example, in /etc/aide/aide.conf.d/observability.conf:
# Permissions, inode, links, ownership, size, timestamps and SHA-256 content.
# These are runtime data and should not be treated as static configuration.
!/var/lib/fluent-bit
!/var/lib/vector
!/var/lib/otelcol
!/var/log
The main AIDE configuration can direct its report to a file that Fluent Bit is permitted to read:
database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout=true
report_url=file:/var/log/aide/aide.log
report_level=changed_attributes
Initialise and approve the baseline using the procedure provided by the operating-system package. The baseline database should be writable only by the account performing the controlled update; consider retaining a signed or read-only copy outside the host.
Run aide --check from a systemd timer or the organisation’s scheduler. An hourly check provides better detection latency than the weekly minimum stated by PCI DSS, although the right frequency depends on estate size, I/O cost and risk. The scheduler should also emit a success or failure event so that a missing check can be detected.
Fluent Bit can then tail the report, add security context and forward the records over OTLP. The Tail input uses a database to retain offsets, and the OpenTelemetry output supports OTLP/HTTP and OTLP/gRPC.
service:
flush: 5
log_level: info
storage.path: /var/lib/fluent-bit/storage
pipeline:
inputs:
-name: tail
tag: security.fim.aide
path: /var/log/aide/aide.log
key: body
path_key: log.file.path
db: /var/lib/fluent-bit/aide-tail.db
db.sync: normal
read_from_head: false
refresh_interval: 5
rotate_wait: 30
skip_empty_lines: true
skip_long_lines: true
storage.type: filesystem
filters:
-name: modify
match: security.fim.aide
Add:
- event.domain file
- event.category file
- event.type change
- security.control file_integrity
- service.name aide
outputs:
-name: opentelemetry
match: security.fim.aide
host: otel-collector.internal
port: 4318
logs_uri: /v1/logs
tls: on
tls.verify: on
compress: gzip
This example emits each AIDE report line as a log record. That is simple and robust, but the backend will need to group lines belonging to the same check. For a more mature deployment I would place a small wrapper around the AIDE invocation that converts each detected change into JSON Lines and adds a check identifier, hostname, baseline version and execution result. Fluent Bit can then use its JSON parser and every changed file becomes a structured event.
Vector can implement the same pattern with its file source and an OTLP-capable destination or intermediate Collector. The OpenTelemetry Collector can also collect files using the contributed filelog receiver. The choice is less important than applying consistent fields, reliable buffering, secure transport and an independently managed destination.
From event collection to a security control
Collecting the AIDE output is only the first half of the solution. To turn it into a control, the backend needs useful detection logic and operational ownership.
I would start with alerts for:
a change to an observability agent destination, filter or exclusion rule;
a change to a telemetry service unit or executable;
deletion or replacement of the AIDE database;
changes outside an approved deployment window;
a burst of changes across several hosts;
a configuration change followed by a fall in log volume; and
a missing AIDE run, agent heartbeat or expected integrity summary.
Correlating the FIM event with deployment records is important. A checksum change shortly after an approved GitOps rollout is very different from the same change following an interactive root login at 02:00. Observability provides the surrounding evidence that allows the integrity event to be assessed rather than simply counted.
There is also value in feeding an integrity event back into operational telemetry. A changed collector configuration can annotate dashboards, traces and incident timelines. The question changes from “why did the logs stop?” to “did the logs stop when this configuration changed?” That is a much better starting point for both an SRE and a security analyst.
Final thoughts
The observability and security domains have spent years building separate pipelines for data that frequently starts on the same machine. Open standards and focused open-source tools give us an opportunity to reduce that duplication.
AIDE supplies a mature integrity signal. Fluent Bit and Vector make that signal transportable and enrichable. OpenTelemetry makes the route to the wider platform portable. None of these components replaces access control, endpoint detection or a SIEM, but together they add a useful security capability to infrastructure many organisations already operate.
Perhaps the most important configuration to observe is the configuration that decides what we are allowed to observe.
Ten years ago I was starting the co-authoring our first book, which turned out to be the first published on Oracle PaaS. Specifically it was about Integration Cloud Service, which was became Oracle Integration Cloud (OIC). The book was published in the Spring of 2017. It was a rewarding experience, and the first of a series of books written over the last 10 years.
Most technology books, particularly those that are focussed on a specific technology tend to have a relatively short shelf life. – part of the reason publishers have early release programmes.
But our first book has been promoted to us as part of Amazon’s daily Kindle Reads email as you can see here…
While I am certainly not complaining, I am curious as to how Amazon’s algorithm choice the book.
As the OpAMP project has grown from a small idea to something pretty substantive, we’ve decided to put it in its own Org and repository with GitHub. This should mean it no longer looks a pet project. The repo name has also been changed as we have address more than just Fluentd and Fluent Bit needs with the Elastic observability pipeline tools having been incorporated with the basic features as well asl Vector.
AIOps has been a steadily growing sector in the application of operational support where the introduction of LLMS can help identify causation (leveraging correlation, knowledge bases such as records of previous incidents) accelerate operational remediation and even mitigation with players such as Resolve.ai, ObserverIQ, and Keep (Open Source).
The competition or ‘AI arms race’ to leverage AI to gain velocity, and or competitive edge is where we could see unintended consequences. A little bit of back ground first.
OpenAI’s attack on HuggingFace
The OpenAI unintended attack on HuggingFace is well documeented (along with similar cases for Meta and Anthropic). Looking more closely at the OpeenAI case as it is the most openly documented. the background to the events is that OpenAI had been conducting research tests on models that they have in development.
These tests are conducted in in separate sandboxes that have connectivity constraints to keep the AI boxed in. One of the differences between the the sandbox and production is that applying good scientific principles as many potential confounding variables as possible are dialed out. Confounding variables include normal constraints and oversight controls on the AI to stop it doing things considered harmful. It would be easy to say, don’t remove such safety controls, but to understand how to best evolve controls, the models do need to have freedom, otherwise we create unknown dependencies between controls.
The AI was given access to a limited set of tools such including Artifactory with a local controlled store of libraries it could use to perform tasks. Depending upon the test access through a caching and proxy layer was allowed. Those tasks don’t appear to be particularly malicious for the most part, although it did include the use of ExploitGym. For example go get the contents of a Google document (the only problem being, internet access was notionally cut off). Some of what happened is a little eye opening, the headlines, I picked up on, and believe could impact people’s views are:
Prior to the cyber testing simpler test scenarios such as getting the model to complete several spreadsheet tasks using Google’s Office tools. Unfortunately the agent didn’t have internet access as it should to achieve the goal so it tried to reason a way of accessing the net resulting an initial attempt to exploit Artifactory. While it failed, it did store content into Artifactory.
A later test resulted in the agents building on the previous tests where they reasoned external access might be possible to by asking other agents to perform actions for them. This lead to Artifactory being used as a collaboration channel.
Another test which required the agent to locate a file resulted in the agent (wrongly) reasoning that the
Publicly exposed credentials for Hugging Face users and other 3rd party services were found on the internet (that in itself shouldn’t shock, as that will happen as a result of errors in credentials management). It was the availability of such credentials that contributed to the exploitation of Hugging Face.
The AI reasoned that to access the internet it needed to a way out. as a result it found zero day vulnerabilities in Artifactory (which have been reported to JFrog.
The following timeline – tells the story over time.
How does this impact AIOps?
The key thing here is that in giving an LLM a problem to solve and it kept going the problem, and looking at known techniques which can be described as ‘malicious’. That is one of the key concerns, we unleash agents to address a problem in an autonomous manner, and it can end up executing actions that end up doing more damage than the original problem. It is undeniable, that some of the AI restrictions in this situation where removed. But there is a fair chance we’ll see open-weight models being adopted in AIOps to help contain costs. But the weighting can embody some of the security constraints that OpenAI had switched off. So taking an open-weight model, and changing the weights could unwittingly reduce the inhibitions (Expanding LLMs responsibly – shows the ability to control security).
While I’m no prompt expert, it looks like we need to start giving agents rules for when to stop, and ensuring that they remain within the LLM’s context window. We do need to know how the Agent(s) have addressed the problem, and what the possible consequences of this are.
It is human nature to trust things if the out come looks correct. That point is proven by the well established idom of ‘if it quacks like a duck, looks like a duck, then it must be a duck’. This means we’re at risk of trusting the AI has got the solution correct. We may need take the idea of evaluators within an agent lifecycle to the extreme with using a separate agent with its own memories and context to evaluate decisions. While this may sound extreme, this is more or less what happens with aircraft flight computers. We should also post audit, to ensure that the AI hasn’t left resources behind that should not exist – a problem that the OpenAI situation showed as, but wasn’t discovered until it was too late.
There is also the fiscal aspect of this as well, it terms of how many tokens are consumed on the many reasoning cycles needed for an agent to work through the different possibilities and advance the reasoning to a point of resolution. Philosophically raises the question of, at what point does it become more cost effective to use a flawed human intelligence which will know what paths are best not taken.
Ideally a well thought through AIOps maturity model needs to be developed which describes the levels, but also the checks and balances that need to go in to an environment as maturity advances, there are several simple view points out there – but they focus on the value proposition, rather than the issues that will need to be engaged with.
It is funny how when websites reference S3 (Simple Storage Service) storage, they typically reference a couple of the Hyperscalers (usually from Azure, GCP and AWS). While I get the reference to AWS, as they developed the service that has become the foundation for the API specification. But the association with just those hyperscalers is deeply misleading (and, to be honest, it bugs me). Any infrastructure as a service (IaaS) provider provides an S3 solution, and would probably not be taken seriously if they didn’t.
But S3 is not just a cloud-only capability today; enterprise-class SANs have S3 as part of their offering. There are open-source offerings that will allow you to overlay S3 onto commodity hardware Ceph, MinIO, to name two). Ceph even has an associated foundation that is a child of the Linux Foundation.
Why is this important?
Aside from the distributed storage capabilities. Perhaps a key factor is that the portability opens up several solution options that can be deployed both to the cloud and on-premises (yes, a POSIX filesystem is portable, but doesn’t offer versioning, and distribution isn’t transparent). One of these is that development around the adoption of HDFS has slowed.
Datalakes and cold-stored data
But S3 isn’t just helping data lakes; it is gaining adoption with OLAP structured storage because it is easy to use with Parquet files and indexes to create partitioning tables. Parquet’s columnar characteristics mean it’s easy to retrieve just the attributes you want (with focus on the rows through the use of indexing and segmentation). This helps power retained data that is cold stored (e.g. historic monthly finance figures kept in a database for analytical and compliance needs).
The open source ColdFront extension for Postgres (led by pgedge) leverages Parquet and S3 to store cold data. Still, the clever thing is that the feature transparently combines the relational data with the cold-stored data in Parquet files stored in S3.
S3 Tables
This confluence of Parquet files to support columnar table data storage has led to the development of a specialization of S3 in the form of S3 Tables. Today, S3 Tables are only an AWS feature, and it will be interesting to see if there is a wider adoption of the specification.
The Oracle database while not explicitly supporting the S3Tables API, can effect the same type of behavior by being able to treat S3 storage as an external table, and handle the metadata that will define a Parquet file within the S3 store as a while or segment of a table (for more read go here).
Ok, as a heading, that is a bit clickbait. But the underlying message is true. Let’s ask ourselves some simple questions. What is a patch? The Cambridge Dictionary says within the context a computers:
The key here is change to make something existing to be correct. That means there is something wrong. this can include a means to circumvent security. But to need a fix, we first need to find the fault. As a result that must mean we there is ALWAYS the chance of a fault. Simply put, there is always a vulnerability before a fix. Therefore regardless of how patched we are there is always a possibility however small of their being a vulnerability and that a malicious actor (or hopefully a bug bonus schema or something like Mythos) finds it first.
Cultural challenge
As an industry, our first question when there is a problem is, have you applied the patches (or bug fixes), or have you upgraded to the latest version yet? This is almost as pervasive as the old joke when there is a computer problem: Have you turned it off and on again?
Don’t get me wrong, if you have patches, I’d err towards applying them. But, I’d also advocate trying make a risk assessment as to what patching could go wrong; if you can safely eliminate a problem, it is better to do so. But, blindly patching can be an issue. Patches can conflict other parts of the system, be applied to systems resulting devices being ‘bricked‘ or hitting the ‘blue screen of death‘.
Let me illustrate, using desktop PC. When I receive a windows update, it takes a couple of minutes to see what is there. If the update looks like a malware signatures update – no hesitation, the risk of not taking the change is bigger than bot. If the update is for a driver, then we’re more cautious, which driver, can I install the older one. Time to make sure there is an OS recovery point. If it’s something like a H1 or H2 cumulative and features update – time to run a backup and a restore point.
In an enterprise software environment, be that monolithic on-premises ERPs, through to vast microservice deployments there are the potential for problems. Particularly when you consider the amount of possible customisation that could be involved (in the Oracle domain you’ll hear about CEMLI). This means the possible permutations makes it impossible for the vendor to provide assurances. Not to mention if you’ve applied Modification level changes a patch may well conflict with your modification.
Security in Depth
But security, and the principle of security in depth, is far more than just patching. But I will always advocate more. The ideal world is that we have layers of security, so if one can’t be patched, then other layers are mitigations, or to use another term often used in enterprise security, a ‘compensating controls’ (the idea that if you can’t address a vulnerability in one place, you have security controls elsewhere that can compensate for a weakness).
But we forget – breaches precede patches
The thing we tend to overlook when we get fixated on the idea that staying patched means we’re secure is that a patch always follows a breach. Remember, somewhere along the line, a ‘hacker’ (preferably a security researcher, penetration tester, or a white-hat hacker looking to earn money through bug bounty schemes) will find a vulnerability and breach a system as a result. It is only now that the exploit (vulnerability) is known that the work on creating a patch to address it.
But sometimes a patch isn’t practical (it has massive performance implications and demands a rewrite of a key element of the solution). Therefore, we can address the issue with mitigating controls – this could be at a code level, e.g. adding upfront checks for a specific scenario, through to preventing a piece of software from being used in a particular way.
A vulnerability doesn’t mean you’re vulnerable
This is the difficult perspective, and makes interpreting vulnerability scanner data hard to understand. Often, vulnerability scanners will look across a server and determine what products and components are deployed, and what versions of that functionality are deployed. Then look up those components for the list of attributed vulnerabilities. This is helpful, you know, you need to give this some due consideration. But we should not forget that software (and particularly enterprise software) has many configuration points that can cause it to follow different paths, and the vulnerability may lie only in one path. The latest version of WebLogic server has 650-700 mBeans (one of the techniques to configure the WLS behavior).
Look at it another way, you’re a single person with a car (which usually has 4 seats). You need the car to commute to and from work, and to do common chores like going shopping. But one of the back seats ended up getting ripped. The garage wants thousands to replace it. This does not stop you from using the car to commute to work and to do other necessary tasks. Yo have probably unconsciously applied a ‘compensating control’ – never take more than 2 other people in the car with you, and prevent a back seat passenger from sitting on the ripped seat. Frustrating – maybe, but not inoperable. Of course, applying that compensating control to use the car as a taxi would not be commercially viable. You, as the car owner, need to do some due diligence, such as ensuring the rip isn’t a symptom of a more fundamental risk? Yes. Do you need to consider the likelihood of needing more than 2 passengers? Yes.
Patch velocity
One of the challenges that are developing is the velocity of patches coming as AI is able to be used to locate potential errors that then get addressed. This creates an issue of the organization’s being able to rollout patches. This may not sound too challenging, until consider
not all organizations have a pure Kubernetes ecosystem with the means to orchestrate node replacement quickly and easily.
Capacity demands (people, and or compute resources) needed for performing verification steps for any change can out strip resources, or impact other change effort – such as legal compliance, or changes to keep you in business (although a breach could be just as catastrophic).
Demand impacts, when organizations are also trying to create bandwidth and/or budget to transition off vulnerable systems can end up being blocked.
Patch volumes for an environment that is already considered ‘fragile’ are going to generate a lot of work, as not only does the patch need to be applied, but a lot of effort will be needed to demonstrate the the wider business the possible impact has been validated.
To combat velocity, is to work on ccompensating controls in other areas of your system. If the application has a networking weakness, consider not patching the software, and ensuring networking configurations block the network port(s) that are associated with the vulnerability. Add monitoring, and checks to ensure the change to the network aren’t accidentally undone. If you do later apply the patch to address the weakness, don’t unpick the network mitigations, as this gives you ‘security in depth’.
Scanning tools
We can also use detection or scanning tooling to help. For solutions that you have development control over, and preproduction environments should be using these tools. But we have to be careful how we interpret the outcomes of these tools. Blindly take their reports can send us chasing issues, that aren’t really issues. While measuring detecting and measuring risk management is a positive thing, it needs to be done in an informed way.
For example, you run the scanning tool with admin level privileges so it can inspect everything. But, then it detects the finger print of an old version of Java with vulnerabilities in the Swing UI library. But the only reason Java is present is to run some CLI admin utilities. It is never used by the core application, there are no deployed Swing solutions in the environment. If we don’t give consideration to the context of the problem, we end up patching an issue that is in all probability going to be a problem.
At the same time, we can use this understanding to prioritise patching. Yes, applying a patch if available is good, but if applying a patch that breaks your tool is not. Spending time patching a vulnerability, when you don’t use the vulnerable code is eating into capacity to actually implement business change which may well eliminate the problem anyway (for example standardising scripting tools on Python).
What does this all mean?
Bottom line is, if you can patch, you have the ability and capacity to do so then it’s better to do so, as it keeps another layer of defence in place. But you don’t have access to patches (out of support etc), capacity to solve everything immediately then intelligent assessment of the vulnerability, risk driven prioritisation, look to mitigation strategies.
Defence in depth is not a security seller’s motto, but a genuine way to ensure that if one point fails, the next should protect you. But if you can’t patch, understand what the vulnerability is, and ensure you have mitigations in place. Ensure the issue and its implications in your context are documented along with the mitigations – so you have auditable content for any audit. You have been told where the minefield is, so you’ve fenced the area off, so people don’t wander into the issue unwittingly.
You must be logged in to post a comment.