The hyper scaler cloud vendors all offer Logging and monitoring capabilities. But they tend to focus on supporting their native services. If you’re aware of Oracle’s Cloud (OCI) messaging, then you’ll know that there is a strong recognition of the importance of multi-cloud. This extends not only to connecting apps across clouds but also to be able to observe and manage cloud-spanning solutions. Ultimately, most organizations want to headline observability-related views of their solutions.
Late last year, I presented these ideas, illustrating the ideas with the use of Fluent Bit and OCI’s Observability and Management products to visualize and analyze what is happening. I finally found the time to write how the very basic demo was built from a clean sheet over on the Oracle Devs blog on Medium.
This also highlights the fact that the Fluent Bit book, while I believe, once completed, will be through, can’t cover everything – and certainly not build end-to-end use cases like the Oracle Observability & Management example. To help address this, the book includes an appendix of helpful additional information, some of which I have included here, along with other content that we encounter – all of which can be found at Fluentd & Fluent Bit Additional stuff.
With the Christmas holidays happening, things slowed down enough to sit and catch up on some reading – which included reading Cloud Observability in Action by Michael Hausenblas from Manning. You could ask – why would I read a book about a domain you’ve written about (Logging In Action with Fluentd) and have an active book in development (Fluent Bit with Kubernetes)? The truth is, it’s good to see what others are saying on the subject, not to mention it is worth confirming I’m not overlapping/duplicating content. So what did I find?
Cloud Observability in Action by Michael Hausenblas
Cloud Observability In Action has been an easygoing and enjoyable read. Tech books can sometimes get a bit heavy going or dry, not the case here. Firstly, Michael went back to first principles, making the difference between Observability and monitoring – something that often gets muddied (and I’ve been guilty of this, as the latter is a subset of the former). Observability doesn’t roll off the tongue as smoothly as monitoring (although I rather like the trend of using O11y). This distinction, while helpful, particularly if you’re still finding your feet in this space, is good. What is more important is stepping back and asking what should we be observing and why we need to observe it. Plus, one of my pet points when presenting on the subject – we all have different observability needs – as a developer, an ops person, security, or auditors.
Next is Michael’s interesting take on how much O11y code is enough. Historically, I’ve taken the perspective – that enough is a factor of code complexity. More complex code – warrants more O11y or logging as this is where bugs are most likely to manifest themselves; secondly, I’ve looked at transaction and service boundaries. The problem is this approach can sometimes generate chatty code. I’ve certainly had to deal with chatty apps, and had to filter out the wheat from the chaff. So Michael’s approach of cost/benefit and measuring this using his B2I ratio (how much code is addressing the business problems over how much is instrumentation) was a really fresh perspective and presented in a very practical manner, with warnings about using such a measure too rigidly. It’s a really good perspective as well if you’re working on hyperscaling solutions where a couple of percentage point improvements can save tens of thousands of dollars. Pretty good going, and we’re only a couple of chapters into the book.
The book gets into the underlying ideas and concepts that inform OpenTelemetry, such as traces and spans, metrics, and how these relate to Observability. Some of the classic mistakes are called out, such as dimensioning metrics with high cardinality and why this will present real headaches for you.
As the data is understood, particularly metrics you can start to think about how to identify what normal is, what is abnormal, or an outlier. That then leads to developing Service Level Objectives (SLOs), such as an acceptable level of latency in the solution or how many errors can be tolerated.
The book isn’t all theory. The ideas are illustrated with small Go applications, which are instrumented, and the generated metrics, traces, and logs. Rather than using a technology such as Fluentd or Fluent Bit, Michael starts by keeping things simple and directly connecting the gathering of the metrics into tools such as Prometheus, Zipkin, Jaeger, and so on. In later chapters, the complexity of agents, aggregators, and collectors is addressed. Then, the choices and considerations for different backend solutions from cloud vendor-provided services such as OpenSearch, ElasticSearch, Splunk, Instana and so on. Then, the front-end visualization of the data is explored with tools such as Grafana, Kibana, cloud-provided tools, and so on.
As the book progresses, the chapters drill down into more detail, such as the differences and approaches for measuring containerized solutions vs. serverless implementations such as Lambda and the kinds of measures you may want. The book isn’t tied to technologies typically associated with modern Cloud Native solutions, but more traditional things like relational databases are taken into account.
The closing chapters address questions such as how to address alerting, incident management, and implementing SLOs. How to use these techniques and tools can help inform the development processes, not just production.
So I would recommend the book, if you’re trying to understand Observability (regardless of a cloud solution or not). If you’re trying to advance from the more traditional logging to a fuller capability, then this book is a great guide, showing what, why, and how to evaluate the value of doing so.
To come back to my opening question. The books have small points of overlap, but this is no bad thing, as it helps show how the different viewpoints intersect. I would actually say that the Observability in Action shows how the wider landscape fits together, the underlying value propositions that can help make the case for implementing a full observability solution. Then, Logging in Action and the new book, Fluent Bit with Kubernetes, give you some of the common context, and we drill into the details of how and what can be done with Fluent Bit and Fluentd. All Manning needs now is content to deep dive into Prometheus, Grafana, Jaeger, and OpenSearch to provide an end-to-end coverage of first principles to the art of the possible in Observability.
I also have to thank Michael for pointing his readers and sections of Logging in Action that directly relate and provide further depth into an area.
It’s been a busy time on the book front. With the book going into the Manning Early Access Program (MEAP). Since then, we have got three more chapters ready, and we’re waiting for the Manning Technical Editor to come back on the content. Once that feedback is addressed, we go into another peer review round, and the chapters will be added to the MEAP edition.
The peer review process is insightful and helps drive improvements to the content, as the reviewers are deeply informed – after all, some of them coded the functionality I’m writing about. So, the comments and questions that come back deserve careful consideration.
It isn’t just more chapters; the appendices have been moving forward, and we should see the first two appendices in the MEAP content soon.
If that isn’t enough for you – as we developed Chapter 8 (probably ready for Manning by the New Year) and worked on the appendices, we identified content that doesn’t neatly fit the book, so we’ve incorporated that content on this blog and in other places (we’re expecting a blog to be published early in the New Year here). We’ve created a page on this website to help collect together content here, and you can see all the blog content here.
But there is plenty of work still to be done, as Fluent Bit supports both YAML and classic configuration formats. We have focussed on the classic format in the book as more people use that at the moment. But, eventually, the YAML format will become more dominant, so we’re providing the YAML equivalent configurations for all the examples. These need to be finalized and checked.
We think Chapter 9 will come quickly, but the final chapter will take time, as we’re expecting the peer review feedback to land as we start Chapter 10. But there is also a lot more effort involved in the final chapter. Then, it’s another peer review round.
We’ve recently had several pieces published on other websites, so I thought we should link them together.
Here is a short piece on Ubuntu security on OCI here.
Another here (DZone) on the use of Solace for multi-cloud messaging.
We’re expecting another article to appear here in the New Year as well. Plus, the book is moving along at a very nice pace – we’ve got a separate post for that.
On the book front – watch out for the Manning and Packt festive promotions.
I’ve written about how railroad syntax diagrams (see here) are great for helping write code (or configuration files). Following the track through the diagram will give you the correct statement syntax, and generally, the diagrams don’t require you to jump around like BNF or eBNF representations.
As you may have seen, I’m currently writing a book on Fluent Bit, and guess what? We’re on the Stream Processor chapter. Looking broaching the use of syntax, it just felt right to have a railroad diagram. As the diagram is fairly large, it won’t print well, so here are the diagrams.
The master content is in GitHub here. If you want a quick reference to how the diagrams work, check here.
Stream Processor Configuration
Fluent Bit Stream Processor Configuration Syntax
While Fluent Bit’s core syntax is pretty straightforward, the syntax for the stream processing is a bit more complex, with a strong resemblance to SQL. As SQL is declarative in nature and can contain iterative and nested elements, RailRoad diagrams can really help.
The original BNF definition of the Stream SQL syntax is here.
Fluent Bit Configuration RailRoad Diagrams
When it comes to the core configuration files, RailRoad diagrams aren’t as effective because the configuration is more declarative in nature. But we’ve tried to capture the core essence here. The only issue is that representing things like the use of @include, which can show up in most parts of the file – arent so easy, and a list of attributes for each possible standard plugin would make the diagram enormously large and unwieldy.
Fluent Bit Classic Format Configuration
We know there are gaps in the current diagrams, which will be addressed. Including:
YAML format
@include
We should show that environment variables can be references as attributes
A better way to show the required indentation and line separation
All the signs are that we’ll be seeing another avalanche of material from Ryan Adams in the new year.
Adams is incredibly prolific. But unlike the steady flow of releases from, say Van Morrison (1 or 2 every year for the last 10+ years), or Tangerine Dream (120 releases if you include all their soundtrack)., Adams’ release pattern has shifted towards releasing albums as a known series of albums to multiple albums at once. Last year, we saw Chris, FM, Devolver, and Romeo and Juliet come out at once. Then, later two cover albums very close together (Nebraska, Blood on the Tracks). This year, we’ve had the Return to Carnegie Hall box set and Morning Glory.
This shift has really kicked in since Adams resurfaced after being investigated for possible criminal sexual behavior, messy divorce, and other such accusations. Which may well be linked to changes in his record label relationships. I’m not going to suggest Adams is an angel; it’s clear he is a flawed individual. But being flawed doesn’t mean he isn’t talented. Just look at Prince or Michael Jackson to see that
Despite the number of albums being released, the quality is remarkable. But I can’t help wondering if he was working within the constraints of a more conventional record company, whether we would see fewer official albums, but at a standard that equals or surpasses that of Gold or Cold Roses – considered as possibly his best work. Plus, a lot of bootleg releases with the rest of the material.
The new book focuses on Fluent Bit, given its significant advances, reflected by the fact it is now at Version 2 and is deserving of its own title. The new book is a free-standing book but is complimentary to the Logging In Action book. Logging in Action focuses on Fluentd but compliments by addressing more deeply deployment strategies for Fluent Bit and Fluentd. The new book engages a lot more with OpenTelemetry now it has matured, along with technologies such as Prometheus.
It is the fact that we’ve seen increasing focus in the cloud native space on startup speeds and efficiency in footprints that have helped drive Fluent Bit, as it operates with native binaries rather than using a Just-In-Time compilation like Ruby (used for Fluentd). The other significant development is the support for OpenTelemetry.
The book has entered the MEAP (Manning Early Access Program). The first three chapters have been peer-reviewed, and changes have been applied; another three are with the development editor. If you’ve not read a MEAP title before, you’ll find the critical content is in the chapters, but from my experience, as we work through the process, the chapters improve as feedback is received. In addition, as an author, when we have had time away from a chapter and then revisit it – it is easier to spot things that aren’t as clear as they could be. So, it is always worth returning as a reader and looking at chapters. Then, as we move to the production phases, any linguistic or readability issues that still exist are addressed as a copy editor goes through the manuscript.
I’d like to thank those involved with the peer review. Their suggestions and insights have been really helpful. Plus, the team at Calyptia is sponsoring the book (and happens to be employing a number of the Fluent Bit contributors).
We also have a discount code on the book, valid until 20th November – mlwilkins2
It’s been about twenty years since we’ve had any new original songs from Peter Gabriel. Now, for the last year, he has been teasing us by releasing a new track every month with two mixes called The Bright Side and The Dark Side. which sort of makes sense, given you could see I/O as a rather abstract representation of Ying and Yang.
With a track, each month has created an interesting experience, as it has given us time to absorb each track, rather than a big audio feast of an album, where the singles leap out at you, and then you start to appreciate the other tracks. If there is a downside, it is probably the fact it is no longer easy to say – these tracks are the singles. But to be honest I don’t think it matters to Peter Gabriel. There may be fan favorites, but that’s it certainly as far as it goes since Us.
However, even knowing which tracks are becoming fan favorites has been tough as Peter toured the album, and depending upon where you are in the world, you’ll have only heard some of the new songs, even though the core of the live show has been I/O.
The musical core of the band remains largely unchanged, with David Rhodes and Tony Levin with Manu Katche back on drums for most of the tracks. John Metcalfe is back, having also contributed so wonderfully with New Blood and the tours over the last ten years where Gabriel has used orchestral arrangement.
With this team, we have a real mix of style and sounds. From the very reflective Playing For Time, which opens with the muted horn reminiscent of tracks like Father Son on Ovo. Then there are tracks that are rhythm-heavy, like The Court, that would have fit in on the Up album.
As with all the two-letter-titled albums, there is a loose theme to the album. For I/O that is input and output, whether that is input from observation as suggested by Panopticom to the title track about how to absorb and contribute to the environment.
What the album shows and the tour demonstrated is that unlike some of his peers, Peter’s voice has changed, but the songs fit what sounds like a more weathered voice. The older songs, which may have been pitched higher, still have the energy and dynamics but perhaps pitched a little differently. So none of the challenges faced like Jim Kerr, who leans more of backing vocalists live, or Sting and Bono, who you can hear have to really work to hit some of the notes.
Peter has continued the idea that each song gets its own artwork associated with it, which came to prominence on the Us album (you can see more with Art From US). Some videos of this work can be seen here.
I/OPanopticom (artist Davif Spriggs)The CourtPlaying For TimeOlive Tree (artist Barthélémy Toguo)Love Can Heal (Artist Antony Micallef)This Is Home (artist David Moreno)And Still (Artist Megan Rooney)Road To Joy (Artist Ai Weiwei)Four Kinds of Horses (artist Cornelia Parker)
Along with the artwork, there have been some amazing videos. This is not big news, and the use of technology – particularly the application of some Generative AI. Check out these:
Development trends have shown a shift towards precompiled languages like Go and Rust away from interpreted or Just-In-Time (JIT) compiled languages like Java and Ruby as it removes the startup time of the language virtual machine and the JIT compiler as well as a smaller memory footprint. All desirable features when you’re scaling containerized solutions and percentage point savings can really add up.
Oracle has been leading the way with its work on GraalVM for some years now, and as a result, not only can GraalVM be used to produce native binary images from Java code, GraalVM also supports TuffleRuby and GraalPy, among others. As TruffleRuby is an open-source project, Oracle isn’t the only vendor contributing to it, work effort has also come from Shopify.
Helping Ruby move forward isn’t new for the Shopify engineering team, and part of that investment is that they have just announced the open-sourcing of a toolchain called Ruvy. Ruvy takes Ruby and creates a WebAssembly (WASM) from it the code. This builds on the existing project ruby.wasm. In doing so they’ve addressed the Ruby startup overhead of the language VM we mentioned. They have also simplified the process of deployment, eliminating the need for Web Assembly System Interface (WASI) arguments, and overcome constraints of class loading by reading files by having the code bundled within the assembly and then accessing the content using WASI-VFS, a simple virtual file system.
The published benchmarks show a massive performance boost in the process of executing where the Ruby code needs to be executed by the packaged JIT. For me, this is interesting as one of the related cloud-native trends is the shift from Fluentd to Fluent Bit. Fluentd was built with Ruby and has a huge portfolio of third-party extensions. But Fluent Bit is built using C to get those performance gains previously described. But it does support plugins through WASM. This raises an interesting question can we take existing Ruby plugins and wrap them so the required interfacing works – which should be minimal and more likely to be impacted by the fact Fluent Bit v2 has refined the internal data structure that was common to both Fluentd and Fluent Bit to allow Fluent Bit to more easily engaged with OpenTelemetry.
If the extra bit of wrapping code isn’t complex, then applying Ruvy should mean the core plugin can then work with Fluent Bit. If this can be templated, then Fluent Bit is going to make a big leap forward with the number of available plugins.
Infoworld published a rather clickbait incendiary new item the other week ‘few open source projects actively maintained’. Personally, I find these statements a little frustrating, as it would be easy for the less informed to assume that adopting open-source software is dangerous. There are several missed points here:
How well and frequently are close source solutions being maintained, and does that stop businesses from using end-of-life products? There is big business to be had in offering support to end-of-life solutions. Just look at companies like Rimini Street. Such organizations aren’t going to change software unless there is a major issue.
Not all open-source software is intended to be undergoing continuous maintenance? Shocking until you consider that open-source projects will remain open and available even when they have been declared end-of-life. Why? One of the things about open-source is you don’t know who is using the code, and suddenly pulling the code because the originator has decided they can no longer maintain their investment could put others in a difficult position. So, the right thing is to leave the source available and allow people to fork it so they can continue maintaining their own version of it or until they’ve migrated away. That way, the originator is not impacted by changes.
Next up, not all open-source projects need continued maintenance; many repositories exist to provide demo and sample solutions – so that developers can see how to use a product or service. These repositories shouldn’t need to change often. Frequent change could easily be a sign of an unstable product or service. These solutions may not be the most secure, as you don’t want to complicate the illustration with all the checks and balances that should be considered. Look at it this way: when we start learning a new language or tool, we start with the classic Hello World – which today means pointing your browser at a URL and seeing the words appear on the page. Do we insist that the initial implementation be secure? No, because it distracts from the basic message. For example, in my GitHub repository, I have multiple public repositories with Apache2 licenses attached to them – i.e., open-source. A number of them support the books I’ve written – they aren’t going to change – in fact, change would be a bad thing unless the associated book is corrected (this repo, for example).
When it comes to security vulnerabilities. This needs to be viewed with some intelligence. For several reasons:
As mentioned, our demo examples are unlikely to be patched with the latest versions of dependencies all the time. The point is to see how the code works. Unless the demo relates directly to something that has to be patched and that changes the demo itself. I don’t think it is unreasonable to expect developers to apply some intelligence to ensure dependencies (and therefore the risk of known vulnerabilities) are checked rather than blindly cutting and pasting. The majority of the time, such content will be published with a minimum version number, not a maximum.
Sometimes, a security vulnerability isn’t an issue. For example, I rarely run vulnerability checks on my LogSimulator. Not because I have a cavalier attitude to security but because I don’t expect it to ever be near a production environment, and the data flowing through the tool will be known and controlled by the user in advance of any activity. Secondly, it shouldn’t be using sensitive data, and thirdly, if there was any malicious intent intended, then I’d be more concerned about how secure its data source and configuration is. The tool is a command-line solution. That said, I still apply development practices that minimize potential exploitation.
Don’t get me wrong, there are risks with all software – closed and open-source, whether it is maintained or has security vulnerabilities. A software development team has a responsibility to make informed, risk-aware selections of software (open or closed source). If you have the means to check for risks, then they are best used. It is worth not only scanning our own code but also considering whether the dependencies we use have been scanned if appropriate (e.g. used in production). Utilizing innovations like SBOM, and exercising routine checks and reviews can also help.
While I can’t prove it, I suspect there are more risks being carried by organizations adopting a library that was considered sufficiently secure when downloaded, but subsequent vulnerabilities have been found, or selected mitigations to risks have been eroded over time.
Phil (aka MP3Monster)'s Blog 
You must be logged in to post a comment.