Logging in Action with Fluentd, Kubernetes and More

Logging in Action with Fluentd, Kubernetes and More

This book looks at the open-source implementation of Fluentd and to an extent Fluent Bit.  As a result, the book will cover the Treasure Data version and will help with various applications of Fluentd (for example Oracle Cloud Logging).

Given that Log management is only as effective as the quality of log information, the book includes chapters on common logging frameworks and best practise for producing log entries.

The book can be purchased from Manning

The book is supported with sample configurations, exercises available at https://github.com/mp3monster/UnifiedLoggingWithFluentd

In addition to the samples and exercise content, to help test Fluentd configurations properly, we’ve produced a Log Simulator which allows us to play into different log forms and over different protocols log events. Allowing for source configurations to be effectively tested.  This tool is available at https://github.com/mp3monster/LogGenerator


Fluentd Plugin – Redislist

As part of the book we’ve built a couple of plugins that make use of Redis. For details:


Fluentd Related Resources…

TitleLinkDescription
Fluentd Home Pagehttps://www.fluentd.org/The home page brings together a lot of official resources
Fluentd on Twitterhttps://twitter.com/fluentdOfficial Twitter feed
Git Hubhttps://github.com/fluentGit Hub repository for Fluentd and all the associated utilities etc
Fluent Bit home pagehttps://fluentbit.io/Fluentd’s baby brother
Docker Fluentd Driver Dockhttps://docs.docker.com/config/containers/logging/fluentd/Details of Fluentd as a docker log driver

Open Source / Non Commercial Related Resources …

LinkDescription
https://cheatsheetseries.owasp.org/
cheatsheets/Logging_Cheat_Sheet.html
Open Web Application Security Project (OWASP) cheat sheet for good logging
https://github.com/uken/fluent-plugin-elasticsearchComprehensive Elasticsearch plugin
https://community.oracle.com/tech/apps-infra/discussion/4467822/video-fluentd-introductionAn introductory video on Fluentd (curtesy of Oracle)
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdfGuide to Computer Security Log Management
Recommendations of the National Institute
of Standards and Technology (NIST)
https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-law-enforcement-processing/accountability-and-governance/logging/Information Commissioner’s Office (ICO in the UK) view on logging
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/Fluentd with Kubernetes
https://pubs.opengroup.org/dpbok/standard/DPBoK.html#IntroductionDigital Practitioner Body of Knowledge Standard
https://tools.ietf.org/html/rfc7807RFC on problem reporting
https://ionutbalosin.com/2019/11/logging-patterns-and-their-performance-impact/Look at logging performance

Commercial

LinkDescription
https://www.calyptia.com/Enterprise supported edition of Fluentd
https://docs.oracle.com/en/cloud/paas/
management-cloud/logcs/use-fluentd-log-collection.html
Oracle Log Management support for Fluentd
https://docs.openshift.com/container-platform/
4.1/logging/config/efk-logging-fluentd.html
OpenShift use of Fluentd
https://cloud.google.com/solutions/real-time/fluentd-bigqueryFluentd & BigQuery on GCP