Fluent Bit with Chat Ops

12 Monday Aug 2024

Posted by mp3monster in Fluentbit, General, Technology

Tags

chatops, conference, Fluent, FluentBit, Open Source Monitoring Conference, osmc, osmc.de, Patrick Stephens, slack, tools

My friend Patrick Stephens and Fluent Bit committer will present at the Open Source Monitoring Conference in Germany later this year. Unfortunately, I won’t be able to make it, as my day job is closing in on its MVP product release.

The idea behind the presentation is to improve the ability to detect and respond to Observability events, as the time between detection and action is the period during which your application is experiencing harm, such as lost revenue, data corruption, and so on.

The stable configuration and code base version is in the Fluent GitHub repository; my upstream version is here. We first discussed the idea back in February and March. We applied simpler rules to determine if the log event was critical.

Advancing the idea

Now that my book is principally in the hands of the publishers (copy editing and print preparation, etc.), we can revisit this and exploit features in more recent releases to make it slicker and more effective, for example.

Stream processor, so a high frequency of smaller issues could trigger a notification using the stream processor.
We can also use the stream processor to provide a more elegant option to avoid notification storms.
The new processors will make it easier to interact with metrics, so any application natively producing metrics.

Other tooling

With the book’s copy editing done, we have a bit more time to turn to our other Fluent Bit project … Fluent Bit configuration converter, both classic to YAML, and implementing a Fluentd to Fluent Bit 1st stage converter. You can see this in GitHub here and here.

Fluent Bit – Powering Chat Ops

26 Monday Feb 2024

Posted by mp3monster in Cloud Native, development, Fluentbit, General, java, Technology, Website

≈ 2 Comments

Tags

chatops, conference, demo, FluentBit, Patrick Stephens, slack

When it comes to observability, particularly logs, and traces, there is a historical tendency to process things in a batch manner or even only once the need to determine the root cause of an outage, often only using something in the metrics to indicate something might not be right. This misses a real opportunity given Fluent Bit can capture observability events in near real-time, whether that is a log, metric, or trace indicating something unhealthy; why not present the issue to those performing an ops role as soon as it is recognized by Fluent Bit. Not once the data is processed by a back end?

While we have solutions like PagerDuty, they tend to be integrated with back-end event analytics tools. Fluent Bit can talk to social channels such as Slack – so why not direct critical events to Slack and interact with the Ops team more directly. After all, if we’re told quickly about an imminent issue or as soon after something wrong occurs, the impact and effort involved in remediation and recovery are smaller. This is the basis of a presentation that Patrick Stephens (from Chronosphere and a committer to the Fluent Bit project) and I have put together. Patrick will be leading the session at the Cloud Native Rejekts conference in Paris (the ‘b side’ to Kube Con Europe), which takes place on the two days before Kubecon itself.

The session looks at the idea of what has been called ChatOps, why and how it can bring value, facilitated with a demo of using Fluent Bit to detect and share an event with Fluent Bit and also pick up and handle directions from the Ops team in the Slack channel.

We hope you’ll see from the session why we think the approach is worthy of consideration and how the potential security considerations can be mitigated. The MVP code is currently here but may, in due course, actually be migrated to the Fluent repos here.

We’ve bundled readme content and scripts to build and help test the additional functionality created to facilitate part of the operation.

We don’t want to spoil the presentation, so we won’t share too much. But it’ll also be worth checking with the blog, seeing as we’ll record a video and eventually record a session explaining the MVP’s ins and outs.