Category Archives: FluentD

LogSimulator new features

03 Friday Sep 2021

Posted by in development, FluentD, General, logsimulator, Technology

≈ Leave a comment

Tags

, , ,

The log simulator we’ve built and written about in the past has had a release made that lines up with the Logging In Action book (v0.1). I am now continuing to add improvements on the main line. Not best Git branching practice, but as I’m working on this solo it doesn’t represent a problem.

If you expect multi line events all you need to do, is add to the properties file a name value pair, with the name FIRSTOFMULTILINEREGEX and the value is a Java/Groovy regular expression which can be used to determine if a line in a log entry is the 1st line in a new log. Then all subsequent log lines are appended to the previous line until a line identifies as a new log entry. The log entry will be written with newline characters in the same place as the read.

In addition to this if the synthetic log events need to be set to be new line then using the ALLOWNL property to be set to true will result in any new line escape sequences (\n) to be made into proper new lines in the output.

The details are all included in the documentation in GitHub.

Continue reading →

New and coming to a screen near you soon

31 Monday May 2021

Posted by in Books, FluentD, General, manning, Oracle, Technology

≈ Leave a comment

Tags

, , , , , ,

Last night saw the final chapter of Logging in Action with Fluentd go back to my editor. The next step is that Chapter (and others I hope) will go to MEAP, so early readers not only get the final chapter, but also the raft of improvements we’ve made. Along with that, the manuscript goes for a full peers review. Once that’s back, its time for a round of edits as I address the feedback then into copy editing and Manning sign off review.

As you might have guessed, we’ve kept busy with an article in the 25th edition of OraWorld. This follows Part 1 talking about GraphQL with a look at considerations for API Security.

In addition to that we’re working on a piece around automation of OCI management activities such as setting up developers, allowing them a level of freedom to experiment without accidentally burning through all your credits by spinning up Exadata servers or 500 node Kubernetes clusters.

We might even have some time to write more about APIs and integration.

Unified Logging with Fluentd becomes Logging in Action using Fluentd, Kubernetes and more

08 Monday Feb 2021

Posted by in Books, FluentD, General, manning, Technology

≈ Leave a comment

Tags

, , , , ,

The book has had a title change as Manning found that links the book was clashing with other solutions using the term ‘Unified Logging’. With the name change it helps bring the book inline with the Manning naming with their In action series. This means the book website is now https://www.manning.com/books/logging-in-action.

With the name change we’ve agreed that there should an additional chapter added. As I’d written the book with a view that everything we cover applies to both modern solutions such as Microservices coming from the CNCF camp but equally relevant to more traditional IT landscapes. Within the book we have explianed how things are positioned and can be used in Kubernetes, but it was agreed with our editorial team that not tackling the configuration of Fluentd with Kubernetes and Docker was to an extent ignoring a key community that will be using Fluentd. So the new chapter will be introduced to address this aspect.

In terms of progress we’re into the 1’s – 1 Chapter to start (the new one), 1 Chapter back from the Technical Editor (Logging Best Practises) – some edits to be done, 1 Chapter now with the editor (How To Create Custom Plugins), 1 Chapter being finished (Logging Frameworks) and finally 1 peer review cycle to go.

Given the lovely review comments that have been quoted on the book’s page. I can only recommend if you have an interest in logging and monitoring then check it out through Manning Early Access Programme (MEAP).

Adventures in DevOps – Fluentd

20 Wednesday Jan 2021

Posted by in FluentD, General, Technology

≈ Leave a comment

Tags

, , , , , , , ,

I was fortunate enough to record a podcast with the team at Adventures In Dev Ops just before Christmas. The recording has been fine tuned and now available on their web site here. From my perspective, the discussion was really interesting and explored a wide range of areas around the challenges of monitoring.

As the podcast is linked to the book we’re writing for Manning (Unified Logging With Fluentd), there is a discount code currently running – poddevopsadv20.

Thanks to Charles Wood and Jeffrey Groman for having me on as a guest.

Other news …

I will be presenting at the online conference Blueprint LDN, check out the subjects being covered, looks very interesting.

Latest on book and APIs

15 Tuesday Dec 2020

Posted by in Books, FluentD, General, Technology

≈ Leave a comment

Tags

, , , , , , , ,

My blogging is way down compared with only a post about OKit – OCI Design (on Windows). It largely comes down to lots of work on our Fluentd book. Chapter 6 is now available in the MEAP. As the promo info says …

What’s new?

Chapter 6, “Filtering and Extrapolation”

Gain control and insight!

Last chapter, we touched on the use of the Filter directive. But that was just the tip of the iceberg! In Chapter 6, we’ll plunge below the surface, exploring the when, why, and how of applying filters to give us more insight and precise control over events.

Promo Email from Manning

Earlier chapters have been tweaked, with some additional improvements which will make the live reading experience better.

Another chapter and an appendix should be finding their way to MEAP very soon as it was handed over by our project editor. That will make it seven chapters available, and all the appendices.

Whilst the peer review is taking place the chapter covering plugin development is progressing. The development work has got the basics of the output plugin with log events being stored in Redis and the input being worked on as well. If you want a peak, keep an eye on my GitHub repository (here).

But is isn’t all writing…

I presented on Twitch – you can catch that at https://m.twitch.tv/videos/809295979 I’ve been offered the opportunity to present again, so keep an eye out for something next year.

We recorded a podcast with the excellent guys over at Adventures In DevOps. We don’t have the exact date for the podcast to be released, but I imagine it will sometime during Jan 2021. I’d recommend checking out the podcasts. I’ve been dipping into their back catalogue of recordings and the team ask some really thought provoking questions.

If that wasn’t enough, we’ve been fortunate enough to have some time to talk with leading members of the Fluentd and Fluent Bit projects which was a real pleasure. Hopefully, as we leave this horrendous year behind we’ll get to talk and possibly collaborate some more.

Fluentd Book Update

02 Monday Nov 2020

Posted by in Books, FluentD, General

≈ Leave a comment

Tags

, , , , , , ,

Things have been very hectic, so much so we’ve not really had much time to write a blog. Most of which has been related to my Fluentd book, so what has been happening (and keep an eye on my Twitter account for a promo code 🙂 ) …

  • Several virtual conferences (Open Source Summit, SPOUG and Oracle APAC Groundbreakers to name just a few)
  • Perhaps the biggest bit has been the book:
    • First 4 chapters went through a rigorous peer review, as a result a number of improvements have been made,
    • with Chapter 5 having been reviewed by our technical editor, and little bit of refinement applied it should be reaching MEAP very very soon along with updated appendices,
    • Chapter 6 has been reviewed by our development editor, so some revisions to apply and then onto the technical editor,
    • Chapter 7 writing in progress, with about 1/3rd complete including examples of applying scaling configuration that can be run on a desktop

So what is to follow:

  • We’re on Manning’s Twitch channel to do a session, which will cover Fluentd, some examples, the book and what it will cover,
  • Once Chapter 7 is done, then we go through a comprehensive review with external input. Depending on the feedback from this, we make another sweep through the existing chapters to make further improvements,
  • Chapter 8 I suspect will be the hardest to write, as we actually get into creating our own Plugin. So I it maybe a little while before this gets completed. The subsequent chapters will come more easily as we’ve got them part written in a rough draft already,
  • We have another round of external peer reviewing to come which will cover everything, so I’m sure we’ll be doing some refinements
  • A podcast recording is scheduled in December.

Talking of Manning on Twitch, this looks like a bit of a hidden Gem – worth looking at not only because of the live stream, but all the previous recordings with other Manning authors are available to watch.

If that isn’t enough with a day job, we have had some major work done on our house. Now we’re moved back in, there are lots of DIY jobs to do, get all the furniture back from storage. Every room apart from the kitchen needs to be painted. But that is my sob story.

I’m hoping to find time to experiment with Oracle’s new cloud native Log Management as this is built on Fluentd foundations and a little bit with API tech – but this is likely to be the Christmas break.

Book Project Unveiled – Unified Logging with Fluentd

03 Wednesday Jun 2020

Posted by in Books, FluentD, General, Technology

≈ Leave a comment

Tags

, , , , , , ,

Fluentd is both an open source solution for making log management so much easier to work with, particularly for distributed / multi component solutions. But not only that it is supported by many log analytics tools, and central to several cloud vendors log management services.

The goal of the book is to explain how Fluentd can help us and to use the tool. We can’t cover every possible plugin, so we walk through the use of enough plugins and the way features interact you can extrapolate to other plugins.

Cloud Native eParty

29 Friday May 2020

Posted by in Cloud, FluentD, General, Technology

≈ Leave a comment

Tags

, , , ,

We’ve been told because of current events in the US that this event is going to be rescheduled.

I am pleased to say that I will be talking about Fluentd at the Cloud Native eParty virtual conference on 2nd June 2020. I’ll be presenting at 4pm, and then hanging out on the conference slack channel to answer any more questions people might have.

https://cloudnativeeparty.com/#section-registration

Hope to see you online.

Mastering FluentD configuration syntax

19 Thursday Sep 2019

Posted by in Cloud, FluentD, General, Technology

≈ Leave a comment

Tags

, , , , , , , ,

Getting to grips with FluentD configuration which describes how to handle logging event(s) it has to process can be a little odd (at least in my opinion) until you appreciate a couple of foundation points, at which point things start to click, and then you’ll find it pretty easy to understand.

It would be hugely helpful if the online documentation provided some of the points I’ll highlight upfront rather than throwing you into a simple example, which tells you about the configuration but doesn’t elaborate as deeply as may be worthwhile. Of course, that viewpoint may be born from the fact I have reviewed so many books I’ve come to expect things a certain way.

But before I highlight what I think are the key points of understanding, let me make the case getting to grips with FluentD.

Why master FluentD?

FluentD’s purpose is to allow you to take log events from many resources and filter, transform and route logging events to the necessary endpoints. Whilst is forms part of a standard Kubernetes deployment (such as that provided by Oracle and Azure for example) it can also support monolithic environments just as easily with connections working with common log formats and frameworks. You could view it as effectively a lightweight (particularly if you use FluentBit variant which is effectively a pared-back implementation) middleware for logging.

If this isn’t sufficient to convince you, if Google searches are a reflection of adoption, then my previous post reflecting upon Observability -London Oracle Developer Meetup shows a plot reflecting the steady growth.  This is before taking into account that a number of cloud vendors have wrapped Fluentd/fluentbit into their wider capabilities such as Google (see here).

Not only can you see it as middleware for logging it can also have custom processes and adapters built through the use of Ruby Gems, making it very extensible.

FluentD

Remember these points

and mastering the config should be a lot easier…

Continue reading →

Observability -London Oracle Developer Meetup

10 Tuesday Sep 2019

Posted by in Dev Meetup, FluentD, General, Technology

≈ 1 Comment

Tags

, , , , , , , , , , ,

meetup-monitoringLast night was the London Oracle Developer Meetup’s sessions around observability.  Andrei Cioaca with a focus on the use of OpenTracing as provided by Jaeger, in a standard Kubernetes deployment with Istio – realized with Oracle Kubernetes Engine (OKE).  This was followed by my session on another pillar using logging via FluentD. Also incorporated into standard Kubernetes, but also able to support traditional monolithic use cases.

Andrei provided a great overview of the 3 pillars and the strengths and weaknesses of the different pillars. With the basics covered Andrei then dove into the configuration and execution of Istio combined with Jaeger and the corresponding insights available.  including a look at the kinds of visual insights that Jaeger and Kiali provide.  Some probing conversations followed about the relationship to Spring Cloud Sleuth, Open Zipkin and the OpenTracing as a concept more generally.

Andrei’s presentation material can be found in his GitHub repository here.

search-trend-fluentd

Google Analytics on Search Terms

My session followed a pizza break, as there was a delay in its arrival. With everybody having chatted over pizza about OpenTracing, we picked up on FluentD and the Logging aspect to Observability. FluentD, as an open-source project has been growing steadily, and actually baked into several Log Analytics products and services – as the above analytics from Google shows.

The presentation looked at the growing challenges of modern software in terms of making sense of logging.  We explored the capabilities of FluentD before drilling into real-world use cases and potential deployment models.

As you’ll see from the slides we ran a couple of demos. The configuration for the demos can be found at https://github.com/mp3monster/fluentd-demos along with an example payload.

The next meetup we have organized is around Blockchain, all the details can be found at https://www.meetup.com/Oracle-Developer-Meetup-London/events/264661742/.

Other related info …