Category Archives: Technology

Fluent Bit with Kubernetes – more MEAP chapters

06 Saturday Apr 2024

Posted by in Books, Fluentbit, manning, Technology

≈ Leave a comment

Tags

, , , ,

12th April Update – The last chapter, a use case Appendix, and a couple of chapter updates are heading to the MEAP release.

We’ve not been blogging too much as we’ve been very focused on the book. For the keen readers who have signed up for the MEAP (Manning Early Access Programme) of the book, another 2 chapters are in the process of being made available.

The last chapter has been submitted to our editor along with the appendix, which includes an enterprise use case that outlines a business scenario and illustrates how Fluent Bit can be applied.

We’ve received the feedback from the second peer review and have started to address it. I’m sure that every Manning author will testify as to how helpful the process is. While I recommended some of the reviewers to my editor, I didn’t know others. All the feedback comes back anomalously. So publicly, thank you to the reviewers. Constructive feedback is key to how we ensure that we are getting our points across, but also how details we may have overlooked or thought obvious get put right.

Unfortunately, authors can’t always address every comment. Sometimes, that is down to the fact that the layout has to work within the publisher’s guidelines. Sometimes, we simply can’t fit in suggested content, as we’re ultimately working to an agreed timeline, and people can be put off by 800-page books. For me, and I suspect other authors, those extras aren’t ignored; they’re fuel for blog ideas and content.

We’ve one more peer review cycle where the reviewers get pretty much the entire book, and once any edits for that are needed, we move into the copy editing, which is done by Manning, and I just need to confirm edits don’t accidentally change the meaning and emphasis. This will be a time when we can start blogging and sharing more.

Fluent Bit the engine to power ChatOps – update

17 Sunday Mar 2024

Posted by in chatbots, Fluentbit, General

≈ 1 Comment

Tags

, , , , , ,

The other month, I described a presentation and demo (Fluent Bit – Powering Chat Ops) we’ll be doing for the Cloud Native Rejekts conference, which is the precursor event to KubeCon in Paris this week. Since that post, we’re excited to say that, with Patrick Stephens’s contributions from Chronosphere, the demo is now in the Fluent GitHub repo. It has been nicely packaged with a Docker Compose, so everything runs in a couple of containers.

In addition, if you want to see the presentation and hear us discuss the solution and explain how it works, we recorded part of the presentation dry run, which can be heard here (Demo) and here (Code overview).

I couldn’t be in Paris in person, so Patrick took the job of presenting in Paris, we tried to enable my remote participation but had audio issues. Hopefully, you’ll see the recording of Pat’s physical presentation here. But I did manage to collaborate in the demo:

This means that the original repo I mentioned can be viewed as a beta or upstream version (it’s cluttered with some generated code from Helidon, which we will eventually get around to exploiting and making the utility a native binary executable).

Fluent Bit with Kubernetes book update

05 Tuesday Mar 2024

Posted by in Books, Fluentbit, manning, Technology

≈ Leave a comment

Tags

, , ,

A quick update on the book – very early this morning or late last night (depending on your perspective), we sent our development editor the final chapter of the Fluent Bit with Kubernetes book. There is still a way to go before we’re completed (with multiple reviews to happen, appropriate edits to be made, copy editing, etc. Still, it is an important milestone from an author’s perspective.

For the keen readers who have signed up for the MEAP (Manning Early Access Programme) of the book, I can confirm that the editorial team (preparation for eBook and website formatting, checking the edits to address the Technical Editor and Development Editor haven’t introduced any obvious issues) are working on the preparation of Chapter 7 – so that should be available soon. When this chapter is available, the content covering all the foundational aspects of Fluent Bit will be available. The remaining chapters reflect the advanced features.

Fluent Bit – Powering Chat Ops

26 Monday Feb 2024

Posted by in Cloud Native, development, Fluentbit, General, java, Technology, Website

≈ 2 Comments

Tags

, , , , ,

When it comes to observability, particularly logs, and traces, there is a historical tendency to process things in a batch manner or even only once the need to determine the root cause of an outage, often only using something in the metrics to indicate something might not be right. This misses a real opportunity given Fluent Bit can capture observability events in near real-time, whether that is a log, metric, or trace indicating something unhealthy; why not present the issue to those performing an ops role as soon as it is recognized by Fluent Bit. Not once the data is processed by a back end?

While we have solutions like PagerDuty, they tend to be integrated with back-end event analytics tools. Fluent Bit can talk to social channels such as Slack – so why not direct critical events to Slack and interact with the Ops team more directly. After all, if we’re told quickly about an imminent issue or as soon after something wrong occurs, the impact and effort involved in remediation and recovery are smaller. This is the basis of a presentation that Patrick Stephens (from Chronosphere and a committer to the Fluent Bit project) and I have put together. Patrick will be leading the session at the Cloud Native Rejekts conference in Paris (the ‘b side’ to Kube Con Europe), which takes place on the two days before Kubecon itself.

The session looks at the idea of what has been called ChatOps, why and how it can bring value, facilitated with a demo of using Fluent Bit to detect and share an event with Fluent Bit and also pick up and handle directions from the Ops team in the Slack channel.

We hope you’ll see from the session why we think the approach is worthy of consideration and how the potential security considerations can be mitigated. The MVP code is currently here but may, in due course, actually be migrated to the Fluent repos here.

We’ve bundled readme content and scripts to build and help test the additional functionality created to facilitate part of the operation.

We don’t want to spoil the presentation, so we won’t share too much. But it’ll also be worth checking with the blog, seeing as we’ll record a video and eventually record a session explaining the MVP’s ins and outs.

Fluent Bit with Kubernetes – book update

23 Tuesday Jan 2024

Posted by in Books, Fluentbit, General, manning

≈ Leave a comment

Tags

, , , , , ,

The exciting news is that Manning have released several more chapters of our Fluent Bit with Kubernetes book into the MEAP (Manning Early Access Program) – which means about two-thirds of the book is now available in MEAP form.

We’ve also been beefing up the supporting and related information on this website – as we can’t get everything into the book – for the static pages, the most relevant are here and here, and the blog post content can be seen here.

The sample configurations are in our GitHub repo here, and additional demos can be found here. We’ve got a pretty cool demo being built, which takes Fluent Bit into the world of ChatOps (and it isn’t just sending notifications) – it will eventually become visible in the repo – but to see it sooner, keep an eye out for our conference presentations.

Fluent Bit with Oracle Cloud

09 Tuesday Jan 2024

Posted by in Books, Fluentbit, Fluentd, General, Oracle

≈ Leave a comment

Tags

, , , , , , , , ,

The hyper scaler cloud vendors all offer Logging and monitoring capabilities. But they tend to focus on supporting their native services. If you’re aware of Oracle’s Cloud (OCI) messaging, then you’ll know that there is a strong recognition of the importance of multi-cloud. This extends not only to connecting apps across clouds but also to be able to observe and manage cloud-spanning solutions. Ultimately, most organizations want to headline observability-related views of their solutions.

Late last year, I presented these ideas, illustrating the ideas with the use of Fluent Bit and OCI’s Observability and Management products to visualize and analyze what is happening. I finally found the time to write how the very basic demo was built from a clean sheet over on the Oracle Devs blog on Medium.

Photo by Rafael AS Martins on Unsplash

Useful Resources for Fluent Bit and Observability

This also highlights the fact that the Fluent Bit book, while I believe, once completed, will be through, can’t cover everything – and certainly not build end-to-end use cases like the Oracle Observability & Management example. To help address this, the book includes an appendix of helpful additional information, some of which I have included here, along with other content that we encounter – all of which can be found at Fluentd & Fluent Bit Additional stuff.

Content Not here

22 Friday Dec 2023

Posted by in Books, General, manning, Technology

≈ Leave a comment

Tags

, , , ,

We’ve recently had several pieces published on other websites, so I thought we should link them together.

  • Here is a short piece on Ubuntu security on OCI here.
  • Another here (DZone) on the use of Solace for multi-cloud messaging.

We’re expecting another article to appear here in the New Year as well. Plus, the book is moving along at a very nice pace – we’ve got a separate post for that.

On the book front – watch out for the Manning and Packt festive promotions.

Fluent Bit config – Railroad Diagrams

13 Wednesday Dec 2023

Posted by in Fluentbit, General, railroad diagrams, Technology

≈ Leave a comment

Tags

, , , , ,

I’ve written about how railroad syntax diagrams (see here) are great for helping write code (or configuration files). Following the track through the diagram will give you the correct statement syntax, and generally, the diagrams don’t require you to jump around like BNF or eBNF representations.

As you may have seen, I’m currently writing a book on Fluent Bit, and guess what? We’re on the Stream Processor chapter. Looking broaching the use of syntax, it just felt right to have a railroad diagram. As the diagram is fairly large, it won’t print well, so here are the diagrams.

The master content is in GitHub here. If you want a quick reference to how the diagrams work, check here.

Stream Processor Configuration

Fluent Bit Stream Processor Configuration Syntax

While Fluent Bit’s core syntax is pretty straightforward, the syntax for the stream processing is a bit more complex, with a strong resemblance to SQL. As SQL is declarative in nature and can contain iterative and nested elements, RailRoad diagrams can really help.

The original BNF definition of the Stream SQL syntax is here.

Fluent Bit Configuration RailRoad Diagrams

When it comes to the core configuration files, RailRoad diagrams aren’t as effective because the configuration is more declarative in nature. But we’ve tried to capture the core essence here. The only issue is that representing things like the use of @include, which can show up in most parts of the file – arent so easy, and a list of attributes for each possible standard plugin would make the diagram enormously large and unwieldy.

Fluent Bit Classic Format Configuration

We know there are gaps in the current diagrams, which will be addressed. Including:

  • YAML format
  • @include
  • We should show that environment variables can be references as attributes
  • A better way to show the required indentation and line separation

It’s all about … Fluent Bit

06 Monday Nov 2023

Posted by in Books, development, ExternalWebPublications, Fluentbit, General, manning, Technology

≈ Leave a comment

Tags

, , , , ,

We can reveal why things have been quieter than usual on the blogging front. Logging in Action with Fluentd has a partner title … Fluent Bit with Kubernetes.

The new book focuses on Fluent Bit, given its significant advances, reflected by the fact it is now at Version 2 and is deserving of its own title. The new book is a free-standing book but is complimentary to the Logging In Action book. Logging in Action focuses on Fluentd but compliments by addressing more deeply deployment strategies for Fluent Bit and Fluentd. The new book engages a lot more with OpenTelemetry now it has matured, along with technologies such as Prometheus.

It is the fact that we’ve seen increasing focus in the cloud native space on startup speeds and efficiency in footprints that have helped drive Fluent Bit, as it operates with native binaries rather than using a Just-In-Time compilation like Ruby (used for Fluentd). The other significant development is the support for OpenTelemetry.

The book has entered the MEAP (Manning Early Access Program). The first three chapters have been peer-reviewed, and changes have been applied; another three are with the development editor. If you’ve not read a MEAP title before, you’ll find the critical content is in the chapters, but from my experience, as we work through the process, the chapters improve as feedback is received. In addition, as an author, when we have had time away from a chapter and then revisit it – it is easier to spot things that aren’t as clear as they could be. So, it is always worth returning as a reader and looking at chapters. Then, as we move to the production phases, any linguistic or readability issues that still exist are addressed as a copy editor goes through the manuscript.

I’d like to thank those involved with the peer review. Their suggestions and insights have been really helpful. Plus, the team at Calyptia is sponsoring the book (and happens to be employing a number of the Fluent Bit contributors).

We also have a discount code on the book, valid until 20th November – mlwilkins2

Speeding Ruby

30 Monday Oct 2023

Posted by in development, Fluentbit, Fluentd, General, languages, Technology

≈ Leave a comment

Tags

, , , , , ,

Development trends have shown a shift towards precompiled languages like Go and Rust away from interpreted or Just-In-Time (JIT) compiled languages like Java and Ruby as it removes the startup time of the language virtual machine and the JIT compiler as well as a smaller memory footprint. All desirable features when you’re scaling containerized solutions and percentage point savings can really add up.

Oracle has been leading the way with its work on GraalVM for some years now, and as a result, not only can GraalVM be used to produce native binary images from Java code, GraalVM also supports TuffleRuby and GraalPy, among others. As TruffleRuby is an open-source project, Oracle isn’t the only vendor contributing to it, work effort has also come from Shopify.

Helping Ruby move forward isn’t new for the Shopify engineering team, and part of that investment is that they have just announced the open-sourcing of a toolchain called Ruvy. Ruvy takes Ruby and creates a WebAssembly (WASM) from it the code. This builds on the existing project ruby.wasm. In doing so they’ve addressed the Ruby startup overhead of the language VM we mentioned. They have also simplified the process of deployment, eliminating the need for Web Assembly System Interface (WASI) arguments, and overcome constraints of class loading by reading files by having the code bundled within the assembly and then accessing the content using WASI-VFS, a simple virtual file system.

The published benchmarks show a massive performance boost in the process of executing where the Ruby code needs to be executed by the packaged JIT. For me, this is interesting as one of the related cloud-native trends is the shift from Fluentd to Fluent Bit. Fluentd was built with Ruby and has a huge portfolio of third-party extensions. But Fluent Bit is built using C to get those performance gains previously described. But it does support plugins through WASM. This raises an interesting question can we take existing Ruby plugins and wrap them so the required interfacing works – which should be minimal and more likely to be impacted by the fact Fluent Bit v2 has refined the internal data structure that was common to both Fluentd and Fluent Bit to allow Fluent Bit to more easily engaged with OpenTelemetry.

If the extra bit of wrapping code isn’t complex, then applying Ruvy should mean the core plugin can then work with Fluent Bit. If this can be templated, then Fluent Bit is going to make a big leap forward with the number of available plugins.