• Home
  • Site Aliases
    • www.cloud-native.info
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Log Generator
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
      • Fluent Bit Classic to YAML Format configurations
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • Python Setup & related stuff
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Ruby

Logging Frameworks that can communicate directly with Fluent Bit

13 Thursday Jun 2024

Posted by mp3monster in Fluentbit, General, Technology

≈ Leave a comment

Tags

.net, Erlang, Fluentbit, Go, Golang, java, languages, libraries, loggiong, node.js, OCAML, perl, PHP, python, Ruby, Scala

While the typical norm is for applications to write their logs to file or to stdout (console), this isn’t the most efficient way to handle logs (particularly given I/O performance for the storage devices). Many logging frameworks have addressed this by providing more direct outputs to commonly used services such as ElasticSearch and OpenSearch. This is fine, but the only downside is that there is no means for an intermediary layer to preprocess, filter, and route (potentially to multiple services). These constraints can be overcome by using an intermediary service such as Fluent Bit or Fluentd.

Many logging frameworks can work with Fluentd by supporting the HTTP or Forward protocols Fluentd supports out of the box. But as both Fluent Bit and Fluentd are interchangeable with these protocols and logging framework that supports Fluentd, by implication also supports Fluent Bit, not to mention Fluent Bit supports OpenTelemetry.

The following table identifies a range of frameworks that can support communicating directly with Fluent Bit. It is not exhaustive but does provide broad coverage. We’ll update the table as we discover new frameworks that can communicate directly.

Latest Version …

Logging Frameworks and Fluent Bit and Fluentd connectivity
LanguageFramework / LibraryProtocol(s)Commentary
JavaLog4J2HTTP AppenderSend JSON payloads over HTTP (use HTTP input plugin)
Javafluent-logger-javaForward
Pythoncore languageHTTP HandlerProvides the means to send logs over HTTP – means Fluent Bit input handler can manage
Pythonfluent-logger-python
Fluent Logger
ForwardUses the Forward protocol meaning it can gain the efficiencies from msgpack.
Maintained by the Fluent community
Node.jsfluent-logger-nodeForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Node.jsWinstonHTTP

Forward
Winston is designed as a simple and universal logging library supporting multiple transports.
Winston includes transport support for HTTP in its core. There is also a Transport implementation for native Fluent https://github.com/sakamoto-san/winston-fluent
Node.jsPino (Pino-fluent extension)Logger integrated into the Pino logging framework
Go (Golang)fluent-logger-golangForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
.Net (C# VB.Net etc)NLog (NLog.Targets.Fluentd)An NLog target – works with .Net
.Net (C# VB.Net etc)Log4NetLog4Net Appender
.NetSerilog (Fluent Sink)Forward and HTTPSupports both HTTP and nativbe Fluentd/FluentBit
Rubyfluent-logger-rubyForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
PHPfluent-logger-phpForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Perlfluent-logger-perlForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Scalafluent-logger-scalaForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Erlangfluent-logger-erlangForward
It uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
OCAMLfluent-logger-ocamlForward
It uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
RustRust Logging framework extension for Fluent BitRust crate for logging to Fluent Bit
DelphiQuickloggerHTTP

Share this:

  • Click to share on Facebook (Opens in new window) Facebook
  • Click to share on X (Opens in new window) X
  • Click to share on Pocket (Opens in new window) Pocket
  • Click to share on Reddit (Opens in new window) Reddit
  • Click to email a link to a friend (Opens in new window) Email
  • Click to share on WhatsApp (Opens in new window) WhatsApp
  • Click to print (Opens in new window) Print
  • Click to share on Tumblr (Opens in new window) Tumblr
  • Click to share on Mastodon (Opens in new window) Mastodon
  • Click to share on Pinterest (Opens in new window) Pinterest
  • More
  • Click to share on Bluesky (Opens in new window) Bluesky
  • Click to share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Speeding Ruby

30 Monday Oct 2023

Posted by mp3monster in development, Fluentbit, Fluentd, General, languages, Technology

≈ Leave a comment

Tags

Cloud, development, FluentBit, Fluentd, Ruby, Ruvy, Shopify

Development trends have shown a shift towards precompiled languages like Go and Rust away from interpreted or Just-In-Time (JIT) compiled languages like Java and Ruby as it removes the startup time of the language virtual machine and the JIT compiler as well as a smaller memory footprint. All desirable features when you’re scaling containerized solutions and percentage point savings can really add up.

Oracle has been leading the way with its work on GraalVM for some years now, and as a result, not only can GraalVM be used to produce native binary images from Java code, GraalVM also supports TuffleRuby and GraalPy, among others. As TruffleRuby is an open-source project, Oracle isn’t the only vendor contributing to it, work effort has also come from Shopify.

Helping Ruby move forward isn’t new for the Shopify engineering team, and part of that investment is that they have just announced the open-sourcing of a toolchain called Ruvy. Ruvy takes Ruby and creates a WebAssembly (WASM) from it the code. This builds on the existing project ruby.wasm. In doing so they’ve addressed the Ruby startup overhead of the language VM we mentioned. They have also simplified the process of deployment, eliminating the need for Web Assembly System Interface (WASI) arguments, and overcome constraints of class loading by reading files by having the code bundled within the assembly and then accessing the content using WASI-VFS, a simple virtual file system.

The published benchmarks show a massive performance boost in the process of executing where the Ruby code needs to be executed by the packaged JIT. For me, this is interesting as one of the related cloud-native trends is the shift from Fluentd to Fluent Bit. Fluentd was built with Ruby and has a huge portfolio of third-party extensions. But Fluent Bit is built using C to get those performance gains previously described. But it does support plugins through WASM. This raises an interesting question can we take existing Ruby plugins and wrap them so the required interfacing works – which should be minimal and more likely to be impacted by the fact Fluent Bit v2 has refined the internal data structure that was common to both Fluentd and Fluent Bit to allow Fluent Bit to more easily engaged with OpenTelemetry.

If the extra bit of wrapping code isn’t complex, then applying Ruvy should mean the core plugin can then work with Fluent Bit. If this can be templated, then Fluent Bit is going to make a big leap forward with the number of available plugins.

Share this:

  • Click to share on Facebook (Opens in new window) Facebook
  • Click to share on X (Opens in new window) X
  • Click to share on Pocket (Opens in new window) Pocket
  • Click to share on Reddit (Opens in new window) Reddit
  • Click to email a link to a friend (Opens in new window) Email
  • Click to share on WhatsApp (Opens in new window) WhatsApp
  • Click to print (Opens in new window) Print
  • Click to share on Tumblr (Opens in new window) Tumblr
  • Click to share on Mastodon (Opens in new window) Mastodon
  • Click to share on Pinterest (Opens in new window) Pinterest
  • More
  • Click to share on Bluesky (Opens in new window) Bluesky
  • Click to share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Securing credentials in Fluentd configurations

07 Tuesday Jun 2022

Posted by mp3monster in development, Fluentd, General, manning, Technology

≈ Leave a comment

Tags

Conjur, env vars, environment variables, Fluentd, Hashicorp, open source, Ruby, secrets, Security, slack, token, Vault

When configuring Fluentd we often need to provide credentials to access event sources, targets, and associated services such as notification tools like Slack and PagerDuty. The challenge is that we don’t want the credentials to be in clear text in the Fluentd configuration.

Using Env Vars

In the Logging In Action with Fluentd book, we illustrated how we can take the sensitive values from environment variables so the values don’t show up in the configuration file. But, we’ve seen regularly the question of how secure is this, can’t the environment variable be seen by everyone on that machine?

The answer to this question comes down to having a deeper understanding of how environment variables work. There is a really good explanation here. The long and short of it is that environment variables can only be seen by the process that creates the variable and any child process will receive a copy of the parent’s variables.

This means that if we create the variable in a shell, only that shell and any processes launched by that shell can see the environment variable. So as long as we don’t set variables up as part of a system-level configuration then we already have a level of security. So we could wrap the start of Fluentd with a script that sets the environment variables needed. Then everything launches that script.

An even better way?

Continue reading →

Share this:

  • Click to share on Facebook (Opens in new window) Facebook
  • Click to share on X (Opens in new window) X
  • Click to share on Pocket (Opens in new window) Pocket
  • Click to share on Reddit (Opens in new window) Reddit
  • Click to email a link to a friend (Opens in new window) Email
  • Click to share on WhatsApp (Opens in new window) WhatsApp
  • Click to print (Opens in new window) Print
  • Click to share on Tumblr (Opens in new window) Tumblr
  • Click to share on Mastodon (Opens in new window) Mastodon
  • Click to share on Pinterest (Opens in new window) Pinterest
  • More
  • Click to share on Bluesky (Opens in new window) Bluesky
  • Click to share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Mastering Puppet Review

09 Tuesday Sep 2014

Posted by mp3monster in Books, General, Technology

≈ Leave a comment

Tags

book, EasticSearch, Foreman, Kibana, Logstash, mCollective, Packt, Pulp, Puppet, review, Ruby, Splunk, Thomas Uphill, YAML

Packt’s Mastering Puppet kicks off with substantial first chapter on how to setup Puppet in a manner that can then scale. The core of this is driven by an explanation of the constituent parts of a Puppet solution and where the workload is. In terms of execution this is as much about understanding the configuration of things like Apache, Passenger and Ningx as it is Puppet. As part of the explanation there are indicative numbers in terms of supportable scale which reflects the knowledge of the product.

Looking at configuration distribution for headless deployments with Git is a solid well considered piece and the writing suggests considers all the needs of a solid deployment of a production quality solution such as access control, whilst supporting collaborative working etc. it would be interesting to have seen how that would have stacked against capabilities such as Zookeeper.

As we move through the chapters the books continues with more advanced themes such as using Hiera as a object hierarchical framework for managing configuration and on into leveraging Puppet forge and various Git repositories (and the challenges when linking to git repositories of the latest code vs a release). With the repositories we can draw in additional tooling and how to incorporate these capabilities into a deployment. This includes looking at several modules that practical experience from the author would recommend.

By chapter 6 we’re into writing our own custom modules and facts and deploying them. So you can do things such as create modules to manage your custom solutions.

The next natural step is to look at the reporting aspects of Puppet, orchestration through marionette collective (mCollective). Obviously to report you need to gather the activity information, so the book touches on the out of the box (OOTB) approach and moves onto the idea of using IRC; presentation via Foreman and Puppet Dashboard. Finally then with a reporting view, the next step is to dynamically query the nodes in Puppet environment which uses mcollective to communicate back & forth with the nodes.

So now we have a dynamically configurable set of Nodes, which can report and have dynamic querying against the nodes.  Final chapters cover the use of things like PuppetDB, roles & profiles and developing and debugging your puppet environment.

Reading the book, I get the feeling that a fair grasp of Linux system administration would help (i.e. a bit more than the average developer). There are a few useful touches that I think could have been included, such as external references such as man pages for RPM or site for the Pulp tool mentioned. But, as criticisms go, this as much me being too lazy to Google. The only other refinement would be inclusion of some diagrams to support the words. As they say a picture can tell a 1000 words, even if this was to just show the hierarchy or directory structures involved.

Compared to the recently reviewed Puppet Reporting book, this book isn’t for someone starting out with Puppet (but the Packt site says as much). You atleast need to have got some basic understanding or practical exposure to Puppet,  and exposure to a development environment is an added bonus.  So if you’re setting out with Puppet you might consider starting with the Puppet 3 Beginner’s Guide (Amazon) or Instant Puppet 3 Starter (Amazon).  Having got those under your belt, try this book to to really develop the use of Puppet configuration and deployment.  When it comes to reporting I’d look at this book along with reporting book (reviewed here).  This book feels like more options are on offer, but Puppet Reporting is a lot richer (but you’d expect that given the different book emphasis).

In summary – good solid book, full of practical experience and ideas.  But don’t try to use this as a jumpstart to Puppet.

Below are a few links I thought might be helpful as they aren’t in the book:

  • YAML – human readable serialization format
  • Pulp – software repository management app
  • Ruby – Open Source OO programming language
  • Foreman – tool capable of extending puppet to deliver PXE capabilities along with capabilities such as reporting
  • Splunk – BigData style analytics on log files etc
  • Elasticsearch / Logstash / Kibana (ELK) – set of tools to provide analytics against log files
  • ActiveMQ – Apache implementation of a JMS compliant messaging solution used my mcollective

Mastering Puppet at Amazon.

Share this:

  • Click to share on Facebook (Opens in new window) Facebook
  • Click to share on X (Opens in new window) X
  • Click to share on Pocket (Opens in new window) Pocket
  • Click to share on Reddit (Opens in new window) Reddit
  • Click to email a link to a friend (Opens in new window) Email
  • Click to share on WhatsApp (Opens in new window) WhatsApp
  • Click to print (Opens in new window) Print
  • Click to share on Tumblr (Opens in new window) Tumblr
  • Click to share on Mastodon (Opens in new window) Mastodon
  • Click to share on Pinterest (Opens in new window) Pinterest
  • More
  • Click to share on Bluesky (Opens in new window) Bluesky
  • Click to share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Puppet Reporting & Monitoring Book Review

27 Friday Jun 2014

Posted by mp3monster in Book Reviews, Books, General, Packt, Technology

≈ Leave a comment

Tags

book, Michael Duffy, monitoring, Packt, packtpub, Puppet, reporting, review, Ruby

So the Packt book (Puppet Reporting and Monitoring) focuses down on a couple of aspects of the Puppet Toolset, as a result this is a relatively short book with only a couple hundred pages. As an enterprise architect I am no expert Puppet practitioner, my knowledge of Ruby is high level (part of the reason I reviewed this book is I wanted to better understand the art of the possible in these areas).  But despite this the book does an exceptionally good job defining a context and then explaining and showing what could be done, down to code examples.  In doing so, the author Michael Duffy introduces a number of open source libraries that can be leveraged to provide dashboard views, presentation of report content whilst maximising the leveraging of the Puppet ecosystem such as the Puppet DB (an abstracted database with a REST + JSON API).  The book goes beyond just implementation of monitoring and reporting but also engages with considerations such as deployment.  without ‘boiling the ocean’ the book provides a very good illustrations of the art of the possible and provides plenty of references to source information so working how you want to implement you own solutions.

My only criticism of the book, and it is a minor one at that is a few more diagrams to help illustrate ideas (particularly in the first chapter when discussing deployment considerations) would help get ideas across easily.

On the strength of this  book,  I hope that Michael considers taking on other authoring projects as this has been one of the best written technical books I’ve read in sometime.

Puppet Reporting & Monitoring
Useful Links:

  • Book – http://bit.ly/1qbSxKC
  • Puppet Labs – http://puppetlabs.com/
  • Puppet DB – http://docs.puppetlabs.com/puppetdb/
  • Ruby – https://www.ruby-lang.org/
  • Michael Duffy – http://www.stunthamster.com/, http://uk.linkedin.com/pub/michael-duffy/40/809/17a

Share this:

  • Click to share on Facebook (Opens in new window) Facebook
  • Click to share on X (Opens in new window) X
  • Click to share on Pocket (Opens in new window) Pocket
  • Click to share on Reddit (Opens in new window) Reddit
  • Click to email a link to a friend (Opens in new window) Email
  • Click to share on WhatsApp (Opens in new window) WhatsApp
  • Click to print (Opens in new window) Print
  • Click to share on Tumblr (Opens in new window) Tumblr
  • Click to share on Mastodon (Opens in new window) Mastodon
  • Click to share on Pinterest (Opens in new window) Pinterest
  • More
  • Click to share on Bluesky (Opens in new window) Bluesky
  • Click to share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

    I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

    • About
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    Oracle Ace Director Alumni

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
      • Book Reviews
      • manning
      • Oracle Press
      • Packt
    • Enterprise architecture
    • General
      • economy
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
          • java
          • node.js
      • drone
      • Fluentbit
      • Fluentd
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • Helidon
        • ITSO & OEAF
        • Java Cloud
        • NodeJS Cloud
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF
    • xxRetired
    • AI
    • API Platform CS
    • APIs & microservices
    • App Ideas
    • Book Reviews
    • Books
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
    • drone
    • economy
    • Enterprise architecture
    • ExternalWebPublications
    • Fluentbit
    • Fluentd
    • General
    • Helidon
    • ITSO & OEAF
    • java
    • Java Cloud
    • languages
    • LinkedIn
    • logsimulator
    • manning
    • mindmap
    • Music
    • Music Resources
    • Music Reviews
    • node.js
    • NodeJS Cloud
    • OIC – ICS
    • OMESA
    • Oracle
    • Oracle Cloud Native
    • Oracle Press
    • OUG
    • Packt
    • Photography
    • Podcasts
    • railroad diagrams
    • Technology
    • TOGAF
    • tools
    • Website
    • xxRetired

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,555 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    December 2025
    M T W T F S S
    1234567
    891011121314
    15161718192021
    22232425262728
    293031  
    « Nov    

    Twitter

    Tweets by mp3monster

    History

    Speaker Recognition

    Open Source Summit Speaker

    Flickr Pics

    Turin Brakes Acoustic Tour 24 @ The Maltings FarnhamTurin Brakes Acoustic Tour 24 @ The Maltings FarnhamTurin Brakes Acoustic Tour 24 @ The Maltings FarnhamTurin Brakes Acoustic Tour 24 @ The Maltings Farnham
    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      • Phil (aka MP3Monster)'s Blog
      • Join 233 other subscribers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Subscribe Subscribed
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d