The latest update was provided on:
Suggested additional reading and useful technical resources related to Fluent Bit and Fluentd. Intended to be complimentary to the books Fluent Bit with Kubernetes and Logging in Action with Fluentd books.
Additional Content on …
Fluent Bit
on Fluentd
- Fluent Bit Articles on TheNewStack – a Biproduct of a book sponsor?
- Fluentd Labels and Fluent Bit
- Migrating from Fluentd to Fluent Bit
- Checking your OpenTelemetry pipeline with Telemetrygen
- Fluent Bit with Oracle Cloud
3rd Party Resources
Logging Frameworks
Application Logging Frameworks can allow us to connect the generations and send logging messages directly to Fluent Bit and/or Fluentd, removing the I/O performance issues of going to a file and tailing the file. Many logging frameworks will talk about Fluentd – but will be capable of talking to Fluent Bit just as easily, as both support a number of common protocols such as HTTP and Forward, for example.
It is also worth looking at whether the framework has a means to template outputs or a formatting mechanism that can address the common issue of log events (particularly stack traces) containing new lines, which can add additional work to combine into a single log event.
- Better Stack article on Logging Frameworks – looks at both the factors to consider when choosing logging frameworks and covers a range of different frameworks for various languages.
- Logging in Action – chapter 11 — covers a look at Logging frameworks – the context is primarily Fluentd, but much of what is described will also work for Fluent Bit
- Calyptia blog on Fluent Bit with Java
- Mapped Diagnostic Content
- OpenTelemetry Logs SDK
- our own catalog of frameworks and how they can work with Fluentd and Fluent Bit.
Both Fluent Bit & Fluentd
- Forward Protocol Specification v1
- LogRotate Linux feature
- Fluent Slack
- W3C Spec for Trace Context
- Is it Observable? – Fluent Bit and more
- For creating SSL certificates to encrypt inter Fluent traffic – Let’s Encrypt
- Also checkout Minica
- Systemd with WSL
- Observability
- Observability Is Cultural
- Is It Observable – excellent YouTube channel covering Fluentd, Fluent Bit, and many of the other observability channels.
- Cloud Observability in Action
- CNCF Observability WhitePaper
- Logstash, Fluentd, Fluent Bit, or Vector? How to choose the right open-source log collector (Feb 2022)
- Effective Coding With Java Observability (July 23)
- OpenTelemetry
- Kublr support for Fluentd and Fluent Bit
- Oracle Cloud
- Kubernetes & Fluent
- How to expose K8s secrets as files inside a pod – a strategy for sharing certificates to Fluent Bit safely – explained here.
- Kubesphere Fluent Operator Walkthorugh – and another Fluent Operator Walkthrough
- Kube Logging – Logging Operator
Fluent Bit
- Docker Image
- Artifact Hub – Helm Chart
- Calyptia Learning Labs
- Calyptia Blog
- How to obfuscate logs using Fluent Bit in New Relic
- Connecting Rust to Fluent Bit
- OpenTelemetry Specification
- Fluent Bit packaged by Bitnami
- Using OCI OpenSearch to ingest Kubernetes logs using Fluentbit and Fluentd (Sep 22)
- Ingesting Data into an OpenSearch Cluster with Fluent Bit
- Fluent Bit Elastic Search Data Type Filter (July 19)
- Send Kapsule logs and metrics to the Observability Cockpit with Fluent Bit (Jun 23)
- Examples of using/extending Fluent Bit with TensorFlow
- Fluent Bit Stream Processing with Kubernetes
- EKS Workshop – Using Fluent Bit
- Fluent Bit Sidecar
- StackOverflow on Fluent Bit
- Reddit – Fluent Bit
- Book related GitHub repository and Postman collection
Plugins and commercial use cases
- Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs
- NVidia UFM
- Integration of Manticore with Fluentbit
- Integrating Fluent Bit and RTI Connext DDS to Manage Real-time Log Data
- Parsing Apache & Nginx logs with fluent-bit (June 22)
- Sending logs to Better Stack with Fluent Bit (uses HTTP plugin)
- Sending FluentBit logs to Uptrace
- Send Fluent Bit logs to Axiom
- Getting started with Fluent Bit and Azure Data Explorer
- VictoriaMetrics – Fluent Bit setup
- VKCloud Fluent Bit plugin
- Rafay Product Documentation – KOP Recipes – Fluent Bit
- Apica Ascent Observability Stack – Fluent Bit
- K8s logs to Upstash Kafka using Fluent Bit
- Circonus integration to Fluent Bit
- Couchbase – Tips and Tricks for Log Forwarding and Couchbase docs
- AWS for Fluent Bit – use cases
- VMWare install Fluent Bit for log forwarding
- Fluent Bit 2.0 and OpenSearch
- Learnings from the field – Running Fluent Bit on Azure Kubernetes Service Part1, Part 2
Fluentd
- Fluentd Docker image
- VSCode plugin for Fluentd
- Getting started with Fluentd and OpenSearch
- Ingest logs to OCI Logging Analytics using Fluentd
- Watching Vault’s Audit Logs Using FluentD
- SQL input plugin for Fluentd
- All Data Are Belong to AWS: Streaming upload via Fluentd
- GitHub repo for book and Postman collection
Lua
- Home of Lua, including the main docs.
- LuaJIT
- LuaRocks – Lua package manager (GitHub)
- VS Code plugin for Lua
- Lua Users Wiki
Zig
- Zig Language website
- Zig Language doc
- Road to Zig (video)
- Zig GitHub
- Zig community projects (from package managers to HTTP servers and web frameworks)
- Zig developer Forum
- Zig Resources for learning
- Zig recipes/blogs
Related Content
- Observability
- Return on Investment Driven Observability by Michael Hausenblas
- CNCF Observability Whitepaper
- Open Observability Talks – award-winning video/podcast
- Observability 360 – news etc
- Correlating Signals Efficiently in Modern Observability
- Debunking 3 Pillars of Observability Myth – Ben Sigelman
- Evolution of Observability
- K8s
- Configuring Docker Log Drivers
- National Cyber Security Centre – Logging and Protective Monitoring
- Logzio’s best practices
- Google’s SRE Guide on Monitoring
- RED (Rate, Errors, Duration) Method for metrics
- Utilization Saturation and Errors (USE) Method for performance analysis
- Incident management for high-velocity teams – MTBF, MTTR, MTTA, and MTTF
- What is eBPF? - standard becoming popular in the Observability space.
- Profiling Encyclopedia – applications of some observability to help optimize performance
- Metrics can support Service Level Objectives (SLOs)
- OpenSLO declarative definition of SLOs
- Generate SLOs from Prometheus with Sloth
- a practical guide for implementing SLO
- SLO Development Lifecycle
General Tooling that can help
- Calyptia Core – tooling for visualization through to managing deployment fleets
- Logging Operator – tool for managing deployments of Fluent Bit and Fluentd (home page)
- Log Simulator
- jq– ideal for formatting JSON
- YAML validation tools (Jan 21)
- Rubular – ERuby regular expression Editor (valid for both Fluentd and Fluent Bit)
- Creating CPU spikes
- WireMock and a WireMock UI
- OpenTelemetry Synthetic Log Generator
- Protoman – Postman for Protobuf APIs – ideal for working with OpenTelemetry and OTEL
- Uptrace – Open source APM
Railroad Syntax Diagrams
Stream Processor Configuration
While Fluent Bit’s core syntax is pretty straightforward, the syntax for the stream processing is a bit more complex, with a strong resemblance to SQL. As SQL is declarative in nature and can contain iterative and nested elements, RailRoad diagrams can really help.
The original BNF definition of the Stream SQL syntax is here.
Fluent Bit Configuration RailRoad Diagrams
When it comes to the core configuration files, RailRoad diagrams aren’t as effective because the configuration is more declarative in nature. But we’ve tried to capture the core essence here. The only issue is that representing things like the use of @include, which can show up in most parts of the file – arent so easy, and a list of attributes for each possible standard plugin would make the diagram enormously large and unwieldy.
We know there are gaps in the current diagrams, which will be addressed. Including:
- YAML format
- @include
- We should show that environment variables can be references as attributes
- A better way to show the required indentation and line separation
Phil (aka MP3Monster)'s Blog 
You must be logged in to post a comment.