Logging in Action with Fluentd, Kubernetes and More

This book looks at the open-source implementation of Fluentd and to an extent Fluent Bit. As a result, the book will cover the Treasure Data version and will help with various applications of Fluentd (for example Oracle Cloud Logging).

Given that Log management is only as effective as the quality of log information, the book includes chapters on common logging frameworks and best practise for producing log entries.

The book can be purchased from Manning –

The book is supported with sample configurations, exercises available at https://github.com/mp3monster/UnifiedLoggingWithFluentd

In addition to the samples and exercise content, to help test Fluentd configurations properly, we’ve produced a Log Simulator which allows us to play into different log forms and over different protocols log events. Allowing for source configurations to be effectively tested. This tool is available at https://github.com/mp3monster/LogGenerator

How fluentd fits into the modern software landscape from Phil Wilkins

Fluentd Plugin – Redislist

As part of the book we’ve built a couple of plugins that make use of Redis. For details:

Over the years of working in the monitoring / logging space and with writing this book. Ive picked up useful resources and articles along the way. To those who have reviewed the book and offered more – thankyou. The book has an appendix which is a superset of this.

Title	Link	Description
Fluentd Home Page	https://www.fluentd.org/	The home page brings together a lot of official resources
Fluentd on Twitter	https://twitter.com/fluentd	Official Twitter feed
Git Hub	https://github.com/fluent	Git Hub repository for Fluentd and all the associated utilities etc
Fluent Bit home page	https://fluentbit.io/	Fluentd’s baby brother
Docker Fluentd Driver Dock	https://docs.docker.com/config/containers/logging/fluentd/	Details of Fluentd as a docker log driver

Link	Description
https://cheatsheetseries.owasp.org/ cheatsheets/Logging_Cheat_Sheet.html	Open Web Application Security Project (OWASP) cheat sheet for good logging
https://github.com/uken/fluent-plugin-elasticsearch	Comprehensive Elasticsearch plugin
https://community.oracle.com/tech/apps-infra/discussion/ 4467822/video-fluentd-introduction	An introductory video on Fluentd (curtesy of Oracle)
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf	Guide to Computer Security Log Management Recommendations of the National Institute of Standards and Technology (NIST)
https://ico.org.uk/for-organisations/guide-to-data-protection/ guide-to-law-enforcement-processing/accountability-and-governance/logging/	Information Commissioner’s Office (ICO in the UK) view on logging
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/	Fluentd with Kubernetes
https://pubs.opengroup.org/dpbok/standard/DPBoK.html#Introduction	Digital Practitioner Body of Knowledge Standard
https://tools.ietf.org/html/rfc7807	RFC on problem reporting
https://ionutbalosin.com/2019/11/logging-patterns-and-their-performance-impact/	Look at logging performance
https://fluentular.herokuapp.com/	Tool to help develop Fluentd REGEX expressions
https://regexr.com/	Tool for helping to develop and test Regular Expressions

Commercial

Link	Description
https://www.calyptia.com/	Enterprise supported edition of Fluentd
https://docs.oracle.com/en/cloud/paas/ management-cloud/logcs/use-fluentd-log-collection.html	Oracle Log Management support for Fluentd
https://docs.openshift.com/container-platform/ 4.1/logging/config/efk-logging-fluentd.html	OpenShift use of Fluentd
https://cloud.google.com/solutions/real-time/fluentd-bigquery	Fluentd & BigQuery on GCP