• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Mindmaps Index
    • Oracle Integration Site
    • Useful Tech Resources …
      • Oracle Tech Resources inc Open Source
      • Useful Tech Resources
      • Python Setup & related stuff
  • Music

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Category Archives: logsimulator

Handling multi-line log entries with Fluentd

19 Tuesday Apr 2022

Posted by mp3monster in Fluentd, General, logsimulator, Technology

≈ Leave a comment

Tags

book, Fluentd, multiline, parser, regex

One aspect of logging I didn’t directly address with my Fluentd book was consuming multiline logs, such as those you’ll often see when a stack trace is included in the log output. Implementing the feature with Fluentd isn’t hugely complex as it leverages the use of regular expressions (addressed in the book in more depth) to recognize the 1st line in a multiline log entry and for subsequent lines.

I didn’t address it for a couple of reasons:

  • Using parsers is fairly inefficient, particularly when you’re using a parser to just decide how to then transform a line (this is why I’m not a huge fan of some of the 12 Factor App‘s recommendations when it comes to logging).
  • Incorporating into your Fluentd parser configurations for specific app log setups is arguably increasing the level of coupling.
  • Many logging frameworks can talk directly to Fluentd as we saw in the book. This is can be more efficient, which means that the log event is more likely to be passed over in a structured format (therefore less work to do).
  • Alternatively, frameworks like Log4J2 have the means to strip line feeds, etc at source – https://logging.apache.org/log4j/2.x/manual/layouts.html (see replace)

But let’s also be realistic, many applications will be configured to simply log to a file and aren’t likely to be changed. At which point we do need to process such situations, so is it done? The process remains largely the same as the tail plugin we illustrated. Except we introduce a different parser called multiline. The documentation provided by Fluentd includes several examples of multiline configurations that will work for default log formats (such as Log4J and Rails). If we took our most basic source setup:

<source>
  @type tail
  path ./Chapter3/basic-file.*
  read_lines_limit 5
  tag simpleFile
  <parse>
    @type none
  </parse>
</source>

Then assuming our log Simulator played back multiline logs (the provided configuration doesn’t do that) extended to consume standard Log4J2 logs we would have a configuration as follows:

<source>
  @type tail
  path ./Chapter3/basic-file.*
  read_lines_limit 5
  tag simpleFile
  <parse>
     @type multiline
     format_firstline /\d{4}-\d{1,2}-\d{1,2}/
     format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
  </parse>
</source>

As you can see we’ve set the parser type to multiline. Then there are two regular expressions, format_firstline is used to help recognize the start of a log event. Every line of the log is tested with this expression as we now assume unless this produces a valid result that the line will be part of a multiline event. If you look at the expression you’ll realize it is looking for a DateTime stamp in the form YYYY-MM-DD. This does mean if you generate a log that starts with the date even if it is part of a multiline output then you’ll trip up the parser. You could extend the expression – but the longer it is the slower the processing.

Following format_firstline we have in our example format1 which describes how to process the first line. This can be extended to define how to handle subsequent lines but this could be multiple format definitions. They do need to be presented in numerical order eg. format1, format2, format, and so on.

LogSimulator – Playing back multi-line logs

The Log simulator uses a very similar mechanism to Fluentd to understand how to playback multiple line logs. When it is reading the log lines in for replay it uses a regular expression to recognize the start of a new log entry (FIRSTOFMULTILINEREGEX) defined in the properties file. The simulator will concatenate lines together until it either hits the end of the file or has a new line that complies with the REGEX. It stores the line with an encoded /n (newline character). It will then print the log using the format specified and will allow the /n to create a newline (or not) based on another config parameter (ALLOWNL).

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

LogSimulator new features

03 Friday Sep 2021

Posted by mp3monster in development, Fluentd, General, logsimulator, Technology

≈ Leave a comment

Tags

Log, logging, tool, utility

The log simulator we’ve built and written about in the past has had a release made that lines up with the Logging In Action book (v0.1). I am now continuing to add improvements on the main line. Not best Git branching practice, but as I’m working on this solo it doesn’t represent a problem.

If you expect multi line events all you need to do, is add to the properties file a name value pair, with the name FIRSTOFMULTILINEREGEX and the value is a Java/Groovy regular expression which can be used to determine if a line in a log entry is the 1st line in a new log. Then all subsequent log lines are appended to the previous line until a line identifies as a new log entry. The log entry will be written with newline characters in the same place as the read.

In addition to this if the synthetic log events need to be set to be new line then using the ALLOWNL property to be set to true will result in any new line escape sequences (\n) to be made into proper new lines in the output.

The details are all included in the documentation in GitHub.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book

Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • TOGAF
  • xxRetired

Twitter

  • RT @RhonddaBryant: Why can’t we do this?Next Tweet: 2 days ago
  • Ready to improve your #CloudNative skills? Download these free ebooks for cloud native #developers. Learn about… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Registration for @Oracle #CloudWorld is now open! Save $200 on your pass by registering early. We might be biased,… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Apollo GraphQL – some pointers blog.mp3monster.org/2022/06/16/apo…Next Tweet: 1 week ago
  • K8s dashboard capability without needing to deploy K8s dashboard blog.mp3monster.org/2022/06/14/k8s…Next Tweet: 1 week ago
Follow @mp3monster

OraWorld

OraWorld

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 614 other followers

Blogs I Follow

  • MiddlewarEArchitecT
  • Site Title
  • All about modern software technologies & architectures
  • Rick's blog
  • A journey in development
  • Phil (aka MP3Monster)'s Blog
  • RedThunder.Blog
  • A millennial's musings
  • Shalindra's Blogs
  • BTplusMore
  • Creativenauts
  • PaaS Community Blog
  • RedStack
  • Musings of an Enterprise Software Technologist
  • The Open Group Blog
  • SutoCom Solutions
  • Rob's Wall Of Music
  • DataCentricSec.com
  • A World of Events

My Other Web Content & Contributions

  • All My Links
  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • GitHub
  • http://phil-wilkins.uk/
  • ICS Book Website
  • Mindmaps
  • Monster's Photos
  • my Capgemini Profile
  • OMESA
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

Calendar

June 2022
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
27282930  
« May    

Other Pages

  • About
    • Internet Profile
    • Presenting Activities
  • Books & Publications
    • API & API Platform
      • API Useful Resources
      • Useful Reading Sources
    • Logging in Action with Fluentd, Kubernetes and More
    • Oracle Integration
  • Mindmaps Index
  • Useful Tech Resources
    • Oracle Tech Resources inc Open Source
    • Python Setup & related tips

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Sound InstallationSound InstallationSound InstallationSound Installation
More Photos

History

Goodreads

OraNA

Aggregated by OraNA

Blogroll

  • A Journey in Development
  • A Neate Blog
  • Amy Simpson Grange blog
  • Blog by Robert van Mölken (co-author on ICS book)
  • Developer Community
  • Exigency In Specie
  • Flexagon (FlexDepoy)
  • GitHub
  • http://phil-wilkins.uk/
  • Ora World
  • PaaS Community
  • SOA4U

Calendar

June 2022
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
27282930  
« May    

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • TOGAF
  • xxRetired

Twitter

  • RT @RhonddaBryant: Why can’t we do this?Next Tweet: 2 days ago
  • Ready to improve your #CloudNative skills? Download these free ebooks for cloud native #developers. Learn about… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Registration for @Oracle #CloudWorld is now open! Save $200 on your pass by registering early. We might be biased,… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Apollo GraphQL – some pointers blog.mp3monster.org/2022/06/16/apo…Next Tweet: 1 week ago
  • K8s dashboard capability without needing to deploy K8s dashboard blog.mp3monster.org/2022/06/14/k8s…Next Tweet: 1 week ago
Follow @mp3monster

Flickr Pics

Sound InstallationSound InstallationSound InstallationSound Installation
More Photos

My Other Web Content & Contributions

  • All My Links
  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • GitHub
  • http://phil-wilkins.uk/
  • ICS Book Website
  • Mindmaps
  • Monster's Photos
  • my Capgemini Profile
  • OMESA
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 614 other followers

History

Other Pages

  • About
    • Internet Profile
    • Presenting Activities
  • Books & Publications
    • API & API Platform
      • API Useful Resources
      • Useful Reading Sources
    • Logging in Action with Fluentd, Kubernetes and More
    • Oracle Integration
  • Mindmaps Index
  • Useful Tech Resources
    • Oracle Tech Resources inc Open Source
    • Python Setup & related tips

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

Social

  • View @mp3monster’s profile on Twitter
  • View philwilkins’s profile on LinkedIn
  • View mp3monster’s profile on GitHub
  • View mp3monster’s profile on Flickr
  • View philmp3monster’s profile on Twitch
Follow Phil (aka MP3Monster)'s Blog on WordPress.com

Blog at WordPress.com.

MiddlewarEArchitecT

Site Title

All about modern software technologies & architectures

Cloud-native, technology and software architecture

Rick's blog

End-to-End OIC to SAP integration

A journey in development

A blog-post by blog-post journey of a ERP Cloud Solutions Degree Apprentice

Phil (aka MP3Monster)'s Blog

from Technology to Music

RedThunder.Blog

Demystifying cloud technologies...

A millennial's musings

Shalindra's Blogs

Technofunctional Blogs

BTplusMore

Business, Technology and more

Creativenauts

Personal, design, inspiration, interests.

PaaS Community Blog

by Jürgen Kress

RedStack

Oracle Database and Cloud Stuff

Musings of an Enterprise Software Technologist

My thoughts on Enterprise Software Technologies...and more.

The Open Group Blog

Achieving business objectives through technology standards

SutoCom Solutions

Success & Satisfaction with the Cloud

Rob's Wall Of Music

Thoughts of a lifelong music hoarder...

DataCentricSec.com

A World of Events

A Blog for Event and Data Analytics

  • Follow Following
    • Phil (aka MP3Monster)'s Blog
    • Join 614 other followers
    • Already have a WordPress.com account? Log in now.
    • Phil (aka MP3Monster)'s Blog
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d bloggers like this: