My day job as a technical architect means I spend a lot of time working on and around technical non-functional needs, from observability to APIs. And APIs are everywhere (sometimes we don’t talk about things like the OpenTelemetry Protocol (OTLP) as APIs, but this is what it is). and I’ve written and blogged on the subject many times in the past.
One of the things I tend to do is read books on the subject – always on the lookout for new strategies, ideas, and techniques for handling an API’s number one challenge – security. With a new book on Secure APIs from José Haro Peralto being published by Manning (as a Manning author, I have the perks of looking at books published and in the Early Access Program).
The Early Access Program means that after the first couple of chapters have been written and go through initial review processes, they’re made available. However, the book is still in development and has not gone through a full copy edit process. However, the core ideas and messages are there in the book.
The book so far looks really good. It comes across as very practical and illustrative of the points it needs from the outset, with some nicely presented insights about why API Security is such an important consideration—54% of web traffic is API-driven, organizations see as many as 10 million attacks per day, and a breach typically costs $6.1 million. If you’re trying to make a case for investing in API security – there are some great references here.
The book doesn’t just look at implementing the code that powers the API contract but also the tools from firewalls to gateways. It engages in the process of figuring out what risks an API needs to mitigate and the consequences of failing to do so. While the first couple of chapters look at the broader landscape and ideas. We can expect a closer look at things like the OWASP Top 10 (a resource that should be mandatory learning for anyone going to implement APIs or web app development more generally) as the book progresses.
The first couple of chapters read well and are easy to absorb, and we’re looking forward to reading the coming chapters, which will discuss the nuts and bolts of securing APIs.
The only observation to be aware of at this point is that, while not explicitly stated, the illustrations suggest a strong bias to RESTful web services with the appearance of just the Open API Initiative logo. While REST is the most common API approach, gRPC, and GraphQL are continuing to make big inroads and are supported by the Asynchronous API Spec. I suspect this will be addressed given José’ background and expertise. I#m looking forward to the coming chapters.
12th April Update – The last chapter, a use case Appendix, and a couple of chapter updates are heading to the MEAP release.
We’ve not been blogging too much as we’ve been very focused on the book. For the keen readers who have signed up for the MEAP (Manning Early Access Programme) of the book, another 2 chapters are in the process of being made available.
The last chapter has been submitted to our editor along with the appendix, which includes an enterprise use case that outlines a business scenario and illustrates how Fluent Bit can be applied.
We’ve received the feedback from the second peer review and have started to address it. I’m sure that every Manning author will testify as to how helpful the process is. While I recommended some of the reviewers to my editor, I didn’t know others. All the feedback comes back anomalously. So publicly, thank you to the reviewers. Constructive feedback is key to how we ensure that we are getting our points across, but also how details we may have overlooked or thought obvious get put right.
Unfortunately, authors can’t always address every comment. Sometimes, that is down to the fact that the layout has to work within the publisher’s guidelines. Sometimes, we simply can’t fit in suggested content, as we’re ultimately working to an agreed timeline, and people can be put off by 800-page books. For me, and I suspect other authors, those extras aren’t ignored; they’re fuel for blog ideas and content.
We’ve one more peer review cycle where the reviewers get pretty much the entire book, and once any edits for that are needed, we move into the copy editing, which is done by Manning, and I just need to confirm edits don’t accidentally change the meaning and emphasis. This will be a time when we can start blogging and sharing more.
A quick update on the book – very early this morning or late last night (depending on your perspective), we sent our development editor the final chapter of the Fluent Bit with Kubernetes book. There is still a way to go before we’re completed (with multiple reviews to happen, appropriate edits to be made, copy editing, etc. Still, it is an important milestone from an author’s perspective.
For the keen readers who have signed up for the MEAP (Manning Early Access Programme) of the book, I can confirm that the editorial team (preparation for eBook and website formatting, checking the edits to address the Technical Editor and Development Editor haven’t introduced any obvious issues) are working on the preparation of Chapter 7 – so that should be available soon. When this chapter is available, the content covering all the foundational aspects of Fluent Bit will be available. The remaining chapters reflect the advanced features.
anyone tracking the Fluent Bit with Kubernetes book progress will be pleased to know that several more chapters are being made available via MEAP (Early Access Program). This includes additional appendices.
We’re hoping to have the first draft of the final two chapters completed in the next couple of weeks so they can start the editorial and go into the peer review process. This includes chapters on extending Fluent Bit through WebAssembly and the Go language with an example of a multi-purpose DB input and output capability.
The exciting news is that Manning have released several more chapters of our Fluent Bit with Kubernetes book into the MEAP (Manning Early Access Program) – which means about two-thirds of the book is now available in MEAP form.
We’ve also been beefing up the supporting and related information on this website – as we can’t get everything into the book – for the static pages, the most relevant are here and here, and the blog post content can be seen here.
The sample configurations are in our GitHub repo here, and additional demos can be found here. We’ve got a pretty cool demo being built, which takes Fluent Bit into the world of ChatOps (and it isn’t just sending notifications) – it will eventually become visible in the repo – but to see it sooner, keep an eye out for our conference presentations.
The hyper scaler cloud vendors all offer Logging and monitoring capabilities. But they tend to focus on supporting their native services. If you’re aware of Oracle’s Cloud (OCI) messaging, then you’ll know that there is a strong recognition of the importance of multi-cloud. This extends not only to connecting apps across clouds but also to be able to observe and manage cloud-spanning solutions. Ultimately, most organizations want to headline observability-related views of their solutions.
Late last year, I presented these ideas, illustrating the ideas with the use of Fluent Bit and OCI’s Observability and Management products to visualize and analyze what is happening. I finally found the time to write how the very basic demo was built from a clean sheet over on the Oracle Devs blog on Medium.
This also highlights the fact that the Fluent Bit book, while I believe, once completed, will be through, can’t cover everything – and certainly not build end-to-end use cases like the Oracle Observability & Management example. To help address this, the book includes an appendix of helpful additional information, some of which I have included here, along with other content that we encounter – all of which can be found at Fluentd & Fluent Bit Additional stuff.
With the Christmas holidays happening, things slowed down enough to sit and catch up on some reading – which included reading Cloud Observability in Action by Michael Hausenblas from Manning. You could ask – why would I read a book about a domain you’ve written about (Logging In Action with Fluentd) and have an active book in development (Fluent Bit with Kubernetes)? The truth is, it’s good to see what others are saying on the subject, not to mention it is worth confirming I’m not overlapping/duplicating content. So what did I find?
Cloud Observability in Action by Michael Hausenblas
Cloud Observability In Action has been an easygoing and enjoyable read. Tech books can sometimes get a bit heavy going or dry, not the case here. Firstly, Michael went back to first principles, making the difference between Observability and monitoring – something that often gets muddied (and I’ve been guilty of this, as the latter is a subset of the former). Observability doesn’t roll off the tongue as smoothly as monitoring (although I rather like the trend of using O11y). This distinction, while helpful, particularly if you’re still finding your feet in this space, is good. What is more important is stepping back and asking what should we be observing and why we need to observe it. Plus, one of my pet points when presenting on the subject – we all have different observability needs – as a developer, an ops person, security, or auditors.
Next is Michael’s interesting take on how much O11y code is enough. Historically, I’ve taken the perspective – that enough is a factor of code complexity. More complex code – warrants more O11y or logging as this is where bugs are most likely to manifest themselves; secondly, I’ve looked at transaction and service boundaries. The problem is this approach can sometimes generate chatty code. I’ve certainly had to deal with chatty apps, and had to filter out the wheat from the chaff. So Michael’s approach of cost/benefit and measuring this using his B2I ratio (how much code is addressing the business problems over how much is instrumentation) was a really fresh perspective and presented in a very practical manner, with warnings about using such a measure too rigidly. It’s a really good perspective as well if you’re working on hyperscaling solutions where a couple of percentage point improvements can save tens of thousands of dollars. Pretty good going, and we’re only a couple of chapters into the book.
The book gets into the underlying ideas and concepts that inform OpenTelemetry, such as traces and spans, metrics, and how these relate to Observability. Some of the classic mistakes are called out, such as dimensioning metrics with high cardinality and why this will present real headaches for you.
As the data is understood, particularly metrics you can start to think about how to identify what normal is, what is abnormal, or an outlier. That then leads to developing Service Level Objectives (SLOs), such as an acceptable level of latency in the solution or how many errors can be tolerated.
The book isn’t all theory. The ideas are illustrated with small Go applications, which are instrumented, and the generated metrics, traces, and logs. Rather than using a technology such as Fluentd or Fluent Bit, Michael starts by keeping things simple and directly connecting the gathering of the metrics into tools such as Prometheus, Zipkin, Jaeger, and so on. In later chapters, the complexity of agents, aggregators, and collectors is addressed. Then, the choices and considerations for different backend solutions from cloud vendor-provided services such as OpenSearch, ElasticSearch, Splunk, Instana and so on. Then, the front-end visualization of the data is explored with tools such as Grafana, Kibana, cloud-provided tools, and so on.
As the book progresses, the chapters drill down into more detail, such as the differences and approaches for measuring containerized solutions vs. serverless implementations such as Lambda and the kinds of measures you may want. The book isn’t tied to technologies typically associated with modern Cloud Native solutions, but more traditional things like relational databases are taken into account.
The closing chapters address questions such as how to address alerting, incident management, and implementing SLOs. How to use these techniques and tools can help inform the development processes, not just production.
So I would recommend the book, if you’re trying to understand Observability (regardless of a cloud solution or not). If you’re trying to advance from the more traditional logging to a fuller capability, then this book is a great guide, showing what, why, and how to evaluate the value of doing so.
To come back to my opening question. The books have small points of overlap, but this is no bad thing, as it helps show how the different viewpoints intersect. I would actually say that the Observability in Action shows how the wider landscape fits together, the underlying value propositions that can help make the case for implementing a full observability solution. Then, Logging in Action and the new book, Fluent Bit with Kubernetes, give you some of the common context, and we drill into the details of how and what can be done with Fluent Bit and Fluentd. All Manning needs now is content to deep dive into Prometheus, Grafana, Jaeger, and OpenSearch to provide an end-to-end coverage of first principles to the art of the possible in Observability.
I also have to thank Michael for pointing his readers and sections of Logging in Action that directly relate and provide further depth into an area.
It’s been a busy time on the book front. With the book going into the Manning Early Access Program (MEAP). Since then, we have got three more chapters ready, and we’re waiting for the Manning Technical Editor to come back on the content. Once that feedback is addressed, we go into another peer review round, and the chapters will be added to the MEAP edition.
The peer review process is insightful and helps drive improvements to the content, as the reviewers are deeply informed – after all, some of them coded the functionality I’m writing about. So, the comments and questions that come back deserve careful consideration.
It isn’t just more chapters; the appendices have been moving forward, and we should see the first two appendices in the MEAP content soon.
If that isn’t enough for you – as we developed Chapter 8 (probably ready for Manning by the New Year) and worked on the appendices, we identified content that doesn’t neatly fit the book, so we’ve incorporated that content on this blog and in other places (we’re expecting a blog to be published early in the New Year here). We’ve created a page on this website to help collect together content here, and you can see all the blog content here.
But there is plenty of work still to be done, as Fluent Bit supports both YAML and classic configuration formats. We have focussed on the classic format in the book as more people use that at the moment. But, eventually, the YAML format will become more dominant, so we’re providing the YAML equivalent configurations for all the examples. These need to be finalized and checked.
We think Chapter 9 will come quickly, but the final chapter will take time, as we’re expecting the peer review feedback to land as we start Chapter 10. But there is also a lot more effort involved in the final chapter. Then, it’s another peer review round.
We’ve recently had several pieces published on other websites, so I thought we should link them together.
Here is a short piece on Ubuntu security on OCI here.
Another here (DZone) on the use of Solace for multi-cloud messaging.
We’re expecting another article to appear here in the New Year as well. Plus, the book is moving along at a very nice pace – we’ve got a separate post for that.
On the book front – watch out for the Manning and Packt festive promotions.
Phil (aka MP3Monster)'s Blog 
You must be logged in to post a comment.