• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Mindmaps Index
    • Oracle Integration Site
    • Useful Tech Resources …
      • Oracle Tech Resources inc Open Source
      • Useful Tech Resources
      • Python Setup & related stuff
  • Music

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Category Archives: manning

Practical Steps when it comes to writing a technical book

11 Tuesday Oct 2022

Posted by mp3monster in Books, General, manning, Oracle Press, Packt, Technology

≈ 1 Comment

Tags

author, book, tips, writing

Following my article on Software Engineering Daily, here are some practical things that will help you if you’re considering taking on a technical book project.

Identifying a Publisher

While it is easy to self-publish today. The recognition comes from having worked with a traditional publisher as they have processes that ensure a level of quality. Not all publishers are equal, and some publishers are attributed with more prestige than others. In addition to this, some publishers are willing to take a risk on a subject and/or author. Have a look at the titles already published, and whether there are any publishers you can connect to.

When comes to contacting the publishers, most of their websites will have a page for recruiting authors. Some are easier to find than others. Here are a couple:

  • Packt
  • Manning
  • Pearson
  • APress

If, or when you get to talk to a publisher it is worth ensuring you understand how their editorial process works and what is expected from you? Plus what happens if you find yourself in the position of not being able to work to the original schedule. Day-to-day work can get in the way which you hadn’t expected.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Securing credentials in Fluentd configurations

07 Tuesday Jun 2022

Posted by mp3monster in development, Fluentd, General, manning, Technology

≈ Leave a comment

Tags

Conjur, env vars, environment variables, Fluentd, Hashicorp, open source, Ruby, secrets, Security, slack, token, Vault

When configuring Fluentd we often need to provide credentials to access event sources, targets, and associated services such as notification tools like Slack and PagerDuty. The challenge is that we don’t want the credentials to be in clear text in the Fluentd configuration.

Using Env Vars

In the Logging In Action with Fluentd book, we illustrated how we can take the sensitive values from environment variables so the values don’t show up in the configuration file. But, we’ve seen regularly the question of how secure is this, can’t the environment variable be seen by everyone on that machine?

The answer to this question comes down to having a deeper understanding of how environment variables work. There is a really good explanation here. The long and short of it is that environment variables can only be seen by the process that creates the variable and any child process will receive a copy of the parent’s variables.

This means that if we create the variable in a shell, only that shell and any processes launched by that shell can see the environment variable. So as long as we don’t set variables up as part of a system-level configuration then we already have a level of security. So we could wrap the start of Fluentd with a script that sets the environment variables needed. Then everything launches that script.

An even better way?

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Coding over Cocktails with Fluentd

06 Friday May 2022

Posted by mp3monster in Books, Fluentd, General, manning, Podcasts, Technology

≈ Leave a comment

Tags

book, Coding over Cocktails, Logging in Action, podcast, Toro Cloud

I’ve been fortunate enough to appear on a podcast with the excellent Coding Over Cocktails team from Toro Cloud. we got to talk about some of the ideas discussed in my Logging In Action book. You can check the podcast out via their website which includes all the episode details and links to all the platforms that host the podcast. There have been some great previous guests such as Luis Weir (my old boss), Chris Richardson of Microservices.io, Matthew Reinbold from Postman, Sam Newman to name just a few.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Handling multi-line log entries with Fluentd

19 Tuesday Apr 2022

Posted by mp3monster in Books, Fluentd, General, logsimulator, manning, Technology

≈ Leave a comment

Tags

book, Fluentd, multiline, parser, regex

One aspect of logging I didn’t directly address with my Fluentd book was consuming multiline logs, such as those you’ll often see when a stack trace is included in the log output. Implementing the feature with Fluentd isn’t hugely complex as it leverages the use of regular expressions (addressed in the book in more depth) to recognize the 1st line in a multiline log entry and for subsequent lines.

I didn’t address it for a couple of reasons:

  • Using parsers is fairly inefficient, particularly when you’re using a parser to just decide how to then transform a line (this is why I’m not a huge fan of some of the 12 Factor App‘s recommendations when it comes to logging).
  • Incorporating into your Fluentd parser configurations for specific app log setups is arguably increasing the level of coupling.
  • Many logging frameworks can talk directly to Fluentd as we saw in the book. This is can be more efficient, which means that the log event is more likely to be passed over in a structured format (therefore less work to do).
  • Alternatively, frameworks like Log4J2 have the means to strip line feeds, etc at source – https://logging.apache.org/log4j/2.x/manual/layouts.html (see replace)

But let’s also be realistic, many applications will be configured to simply log to a file and aren’t likely to be changed. At which point we do need to process such situations, so is it done? The process remains largely the same as the tail plugin we illustrated. Except we introduce a different parser called multiline. The documentation provided by Fluentd includes several examples of multiline configurations that will work for default log formats (such as Log4J and Rails). If we took our most basic source setup:

<source>
  @type tail
  path ./Chapter3/basic-file.*
  read_lines_limit 5
  tag simpleFile
  <parse>
    @type none
  </parse>
</source>

Then assuming our log Simulator played back multiline logs (the provided configuration doesn’t do that) extended to consume standard Log4J2 logs we would have a configuration as follows:

<source>
  @type tail
  path ./Chapter3/basic-file.*
  read_lines_limit 5
  tag simpleFile
  <parse>
     @type multiline
     format_firstline /\d{4}-\d{1,2}-\d{1,2}/
     format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
  </parse>
</source>

As you can see we’ve set the parser type to multiline. Then there are two regular expressions, format_firstline is used to help recognize the start of a log event. Every line of the log is tested with this expression as we now assume unless this produces a valid result that the line will be part of a multiline event. If you look at the expression you’ll realize it is looking for a DateTime stamp in the form YYYY-MM-DD. This does mean if you generate a log that starts with the date even if it is part of a multiline output then you’ll trip up the parser. You could extend the expression – but the longer it is the slower the processing.

Following format_firstline we have in our example format1 which describes how to process the first line. This can be extended to define how to handle subsequent lines but this could be multiple format definitions. They do need to be presented in numerical order eg. format1, format2, format, and so on.

LogSimulator – Playing back multi-line logs

The Log simulator uses a very similar mechanism to Fluentd to understand how to playback multiple line logs. When it is reading the log lines in for replay it uses a regular expression to recognize the start of a new log entry (FIRSTOFMULTILINEREGEX) defined in the properties file. The simulator will concatenate lines together until it either hits the end of the file or has a new line that complies with the REGEX. It stores the line with an encoded /n (newline character). It will then print the log using the format specified and will allow the /n to create a newline (or not) based on another config parameter (ALLOWNL).

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Istio In Action

25 Friday Mar 2022

Posted by mp3monster in Book Reviews, Books, General, manning

≈ Leave a comment

Tags

book, Istio, K8s, Kubernetes, manning, mesh, mindmap

Christian Posta and Rinor Maloku’s book with Manning, Istio In Action has just been published. I’ve previously said it’s a good book, and that’s not surprising given Christian’s role at solo.io. When the final chapters became available I started to go through it in more detail and built a mind map (As with the recent review of Kubernetes best practices). The map can be seen below.

As you can see the map is very substantial reflecting on the depth and value of the book. For those who look at the maps, may notice there are a couple of chapters not fully mapped. I will update the map to fill those gaps in, but given they focus on monitoring and observability, I was less concerned about those areas given my own writing. The book’s exercises are very much built around using Docker Desktop making it very easy to spin up the examples and exercises. If you want to know about Istio Service Mesh on K8s then I’d recommend it.

Reading through the book, I’ve learned details that I was not entirely aware of, for example the integration of non K8s workloads into the mesh. The tuning of Istio to keep it highly performant with a lot of workloads.

The book can be obtained from:

  • Manning
  • Amazon

And other retailers of course.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Logging in Action – has gone to the printers!

07 Monday Feb 2022

Posted by mp3monster in Books, Fluentd, General, manning, Technology

≈ 1 Comment

Tags

Amazon, Barnes & Noble, book, logging, Logging in Action, manning, print, Waterstones

The final leg of getting the book published has taken a lot longer than had been expected. But we’ve just been told that the book has been sent to the printers. This means:

  • final eBook will be available from Manning in about 1 week
  • Preordered print copies of the book will be dispatched in about 2 weeks
  • The alternative ebook formats for mobile readers e.g. kindle etc available in about 3 weeks
  • The book will become available to purchase from other book stores such as
    Amazon in 3-4 weeks
  • Safari Books Online and Apple stores will have the ebook in 4-5 weeks

Here are some links for buying the book …

  • Manning
  • Amazon UK – Amazon US – Amazon DE – Amazon Fr
  • Barnes & Noble, Waterstones
  • Safari Online Books

It also means the project from a writing perspective is complete. But we’re starting to look at the additional examples we’ll add to the GitHub repo. These will be dependent upon the book.

The complete cover artworks …

We’ve got print books now …

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Logging In Action – almost there

08 Thursday Jul 2021

Posted by mp3monster in Books, General, manning

≈ Leave a comment

Tags

CNCF, Docker, Fluentd, fluentd bit, Kubernetes, logging

We’ve got the peer review comments back on the completed 1st draft back of the book. So I’d like to take this opportunity to thanks those who have been involved as peer reviewers, particularly those involved in the previous review cycles. I hope the reviewers found it satisfying to see between iterations that their suggestions and feedback have been taken on board and where we can.

The feed back is really exciting to read. Some tweaks and refinements to do to address the suggestions made.

The work on the Kubernetes and Docker elements and the chapter which has become available on MEAP has helped round that aspect off. But importantly, the final chapters help address the wider challenges of logging, and some of the feedback positively reflects this.

To paraphrase the comments, we’ve addressed the issues of logging which don’t get the attention that they deserve. Which for me is a success.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

New and coming to a screen near you soon

31 Monday May 2021

Posted by mp3monster in Books, Fluentd, General, manning, Oracle, Technology

≈ Leave a comment

Tags

API, Fluentd, logging, Logging in Action, OCI, Oracle, OraWorld

Last night saw the final chapter of Logging in Action with Fluentd go back to my editor. The next step is that Chapter (and others I hope) will go to MEAP, so early readers not only get the final chapter, but also the raft of improvements we’ve made. Along with that, the manuscript goes for a full peers review. Once that’s back, its time for a round of edits as I address the feedback then into copy editing and Manning sign off review.

As you might have guessed, we’ve kept busy with an article in the 25th edition of OraWorld. This follows Part 1 talking about GraphQL with a look at considerations for API Security.

In addition to that we’re working on a piece around automation of OCI management activities such as setting up developers, allowing them a level of freedom to experiment without accidentally burning through all your credits by spinning up Exadata servers or 500 node Kubernetes clusters.

We might even have some time to write more about APIs and integration.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Unified Logging with Fluentd becomes Logging in Action using Fluentd, Kubernetes and more

08 Monday Feb 2021

Posted by mp3monster in Books, Fluentd, General, manning, Technology

≈ Leave a comment

Tags

book, Fluentd, In Action, Kubernetes, manning, progress

The book has had a title change as Manning found that links the book was clashing with other solutions using the term ‘Unified Logging’. With the name change it helps bring the book inline with the Manning naming with their In action series. This means the book website is now https://www.manning.com/books/logging-in-action.

With the name change we’ve agreed that there should an additional chapter added. As I’d written the book with a view that everything we cover applies to both modern solutions such as Microservices coming from the CNCF camp but equally relevant to more traditional IT landscapes. Within the book we have explianed how things are positioned and can be used in Kubernetes, but it was agreed with our editorial team that not tackling the configuration of Fluentd with Kubernetes and Docker was to an extent ignoring a key community that will be using Fluentd. So the new chapter will be introduced to address this aspect.

In terms of progress we’re into the 1’s – 1 Chapter to start (the new one), 1 Chapter back from the Technical Editor (Logging Best Practises) – some edits to be done, 1 Chapter now with the editor (How To Create Custom Plugins), 1 Chapter being finished (Logging Frameworks) and finally 1 peer review cycle to go.

Given the lovely review comments that have been quoted on the book’s page. I can only recommend if you have an interest in logging and monitoring then check it out through Manning Early Access Programme (MEAP).

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Latest on book and APIs

15 Tuesday Dec 2020

Posted by mp3monster in Books, Fluentd, General, manning, Technology

≈ 1 Comment

Tags

AdeventuresInDevOps, book, development, Fluentd, manning, MEAP, podcast, Redis, unified logging

My blogging is way down compared with only a post about OKit – OCI Design (on Windows). It largely comes down to lots of work on our Fluentd book. Chapter 6 is now available in the MEAP. As the promo info says …

What’s new?

Chapter 6, “Filtering and Extrapolation”

Gain control and insight!

Last chapter, we touched on the use of the Filter directive. But that was just the tip of the iceberg! In Chapter 6, we’ll plunge below the surface, exploring the when, why, and how of applying filters to give us more insight and precise control over events.

Promo Email from Manning

Earlier chapters have been tweaked, with some additional improvements which will make the live reading experience better.

Another chapter and an appendix should be finding their way to MEAP very soon as it was handed over by our project editor. That will make it seven chapters available, and all the appendices.

Whilst the peer review is taking place the chapter covering plugin development is progressing. The development work has got the basics of the output plugin with log events being stored in Redis and the input being worked on as well. If you want a peak, keep an eye on my GitHub repository (here).

But is isn’t all writing…

I presented on Twitch – you can catch that at https://m.twitch.tv/videos/809295979 I’ve been offered the opportunity to present again, so keep an eye out for something next year.

We recorded a podcast with the excellent guys over at Adventures In DevOps. We don’t have the exact date for the podcast to be released, but I imagine it will sometime during Jan 2021. I’d recommend checking out the podcasts. I’ve been dipping into their back catalogue of recordings and the team ask some really thought provoking questions.

If that wasn’t enough, we’ve been fortunate enough to have some time to talk with leading members of the Fluentd and Fluent Bit projects which was a real pleasure. Hopefully, as we leave this horrendous year behind we’ll get to talk and possibly collaborate some more.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...
← Older posts

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,540 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

January 2023
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  
« Dec    

Twitter

  • Learn how migrating your apps to #OCI can align with your overall application modernization strategy. Register for… twitter.com/i/web/status/1…Next Tweet: 7 hours ago
  • Join this demonstration and learn how banks can accelerate the account opening and onboarding process, improve aban… twitter.com/i/web/status/1…Next Tweet: 12 hours ago
  • IDC named @Oracle a leader in the MarketScape Worldwide #Hospitality Property Management Systems Vendor Assessment… twitter.com/i/web/status/1…Next Tweet: 1 day ago
  • Thank you for naming @Oracle the 'Best Enterprise Software Vendor' for 2022, @constellationr! social.ora.cl/60193bEs5Next Tweet: 1 day ago
  • Phoenix project blog.mp3monster.org/2023/01/21/pho…Next Tweet: 2 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 216 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d bloggers like this: