Logging in Action with Fluentd, Kubernetes and More

Logging in Action with Fluentd, Kubernetes and More

This book looks at the open-source implementation of Fluentd and to an extent Fluent Bit.  As a result, the book will cover the Treasure Data version and will help with various applications of Fluentd (for example Oracle Cloud Logging).

Given that Log management is only as effective as the quality of log information, the book includes chapters on common logging frameworks and best practise for producing log entries.

The book can be purchased from Manning

Logging in Action with Fluentd

The book is supported with sample configurations, exercises available at https://github.com/mp3monster/UnifiedLoggingWithFluentd

In addition to the samples and exercise content, to help test Fluentd configurations properly, we’ve produced a Log Simulator which allows us to play into different log forms and over different protocols log events. Allowing for source configurations to be effectively tested.  This tool is available at https://github.com/mp3monster/LogGenerator

Fluentd Plugin – Redislist

As part of the book we’ve built a couple of plugins that make use of Redis. For details:

Fluentd Related Resources…

Over the years of working in the monitoring / logging space and with writing this book. Ive picked up useful resources and articles along the way. To those who have reviewed the book and offered more – thankyou. The book has an appendix which is a superset of this.

Fluentd Home Pagehttps://www.fluentd.org/The home page brings together a lot of official resources
Fluentd on Twitterhttps://twitter.com/fluentdOfficial Twitter feed
Git Hubhttps://github.com/fluentGit Hub repository for Fluentd and all the associated utilities etc
Fluent Bit home pagehttps://fluentbit.io/Fluentd’s baby brother
Docker Fluentd Driver Dockhttps://docs.docker.com/config/containers/logging/fluentd/Details of Fluentd as a docker log driver

Open Source / Non Commercial Related Resources …

Open Web Application Security Project (OWASP) cheat sheet for good logging
https://github.com/uken/fluent-plugin-elasticsearchComprehensive Elasticsearch plugin
https://community.oracle.com/tech/apps-infra/discussion/4467822/video-fluentd-introductionAn introductory video on Fluentd (curtesy of Oracle)
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdfGuide to Computer Security Log Management
Recommendations of the National Institute
of Standards and Technology (NIST)
https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-law-enforcement-processing/accountability-and-governance/logging/Information Commissioner’s Office (ICO in the UK) view on logging
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/Fluentd with Kubernetes
https://pubs.opengroup.org/dpbok/standard/DPBoK.html#IntroductionDigital Practitioner Body of Knowledge Standard
https://tools.ietf.org/html/rfc7807RFC on problem reporting
https://ionutbalosin.com/2019/11/logging-patterns-and-their-performance-impact/Look at logging performance
https://fluentular.herokuapp.com/Tool to help develop fluentd REGEX expressions


https://www.calyptia.com/Enterprise supported edition of Fluentd
Oracle Log Management support for Fluentd
OpenShift use of Fluentd
https://cloud.google.com/solutions/real-time/fluentd-bigqueryFluentd & BigQuery on GCP

Interesting articles: