• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Tracing

LogGenerator becomes SignalGenerator

11 Wednesday Nov 2026

Posted by mp3monster in development, Fluent Observability, General, logsimulator, OpAMP, Technology

≈ Leave a comment

Tags

Fluent Bit, FluentBit, Fluentd, log simulator, loggenerator, logs, metrics, observability, Open Telemetry, OpenTelemetry, OTel, python, signal generator, signals, Technology, testing, tools, Traces, Tracing, unified logging

When I wrote Logging In Action, I found that to help with setting up and testing logging pipelines, having a utility that could just take existing logs and play them so it looked like a real application was generating the logs was helpful – not only for the book, but in everyday work.

The natural evolution was: can I also create and run synthetic logs through observability systems so we can ensure log thresholds, filters, alerts, etc. worked as expected. Not to mention taking synthetic or real logs and applying some basic reformatting so we could mimic Apache or Nginx web servers, for example.

So the LogGenerator (sometimes called Log Simulator) was born and made available. At the time, my development skills were centred on Java (and I loved Groovy for its convenience and development speed). So we built the solution using Groovy and Java -the net result is that you could create a deployable JAR or run it without the need to first establish a Maven pipeline (without sacrificing the ability to keep polymorphism and abstract layers).

The LogGenerator has had a number of enhancements, so we could use it to check not only tools like Fluentd and Fluent Bit, but also cloud observability tooling such as OCI. This was made to be a pluggable framework so you could add custom targets easily.

Time moves on …

Python continues to grow in popularity and is the default language bundled with Linux. I’m comfortable developing in Python now, and we have the tools I enjoyed with Groovy and Java (no need to mess with build pipelines for quick changes, but it can also be packaged and easily deployed for mainstream adoption).

But it isn’t programming language evolution that has triggered change. OpenTelemetry standards and the golden signals of Logs, Traces and Metrics are the norm. So applying the same principles to the Traces and metrics that the LogGenerator used for logs is a natural evolution.

Deployment

It was pointed out to me that, even if people don’t take the full OpAMP solution we built, the simulator component may be attractive in its own right. This got us to thinking about whether there was value in pairing the SignalGenerator with the OpAMP simulator.

As a result, we created an optional feature that lets us deploy SignalGenerator as an extension of the OpAMP simulator. This gives us multiple deployment options.

  • A free-standing solution in its own right, with no need for anything else. Ideal for dropping into test environments to exercise your Observability configuration and tuning without needing start-up applications that require large deployments with complex inputs to generate unhappy path signals.
  • Deploy in a supervised manner using OpAMP – ideal if you want to drive synthetic Observability exercises in a distributed manner.
  • Plug into the OpAMP simulator – fewer processes running, easier to create custom commands through the OpAMP framework to get the generator to do things from pushing out new data sources to be replayed, to having signal generators simulate Observability data that reflects a distributed cascading failure as means to evaluate smarter AI-driven processes that should be looking for correlations and cause and effect.

The last part with custom messaging over OpAMP is not yet in place, but it’s a candidate for enhancement.

New home

As the Logging in Action book uses the LogGenerator, we didn’t want to make this level of change in the current repository. So we have a new repository within the OpAMP-Observe GitHub organization which has feature parity, but also has a mechanism for simulating metrics (pushing, or allowing the data to be scraped).

The documentation to link the repositories and the GitHub documentation together is a little bit behind, and we need to figure out how to create a manageable playback structure for Traces before we enable that feature.

Resources

  • Logging In Action
  • LogGenerator aka LogSimulator
  • Maven
  • SignalGenerator
  • OpAMP-Observe (repository)
  • OpAMP.info
Signal Generator Logo

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Observability -London Oracle Developer Meetup

10 Tuesday Sep 2019

Posted by mp3monster in Dev Meetup, Fluentd, General, Technology

≈ 1 Comment

Tags

Fluentd, Istio, Jaeger, Kiali, logging, meetup, monitoring, observability, OKE, OpenTracing, Oracle, Tracing

meetup-monitoringLast night was the London Oracle Developer Meetup’s sessions around observability.  Andrei Cioaca with a focus on the use of OpenTracing as provided by Jaeger, in a standard Kubernetes deployment with Istio – realized with Oracle Kubernetes Engine (OKE).  This was followed by my session on another pillar using logging via FluentD. Also incorporated into standard Kubernetes, but also able to support traditional monolithic use cases.

@andreicioaca starts talking about Oracle #Kubernetes #OKE #istio and #OpenTracing at the #OracleDeveloperMeetup #London @PaaSCommunity pic.twitter.com/HISzpmxjaN

— Phil Wilkins (@PhilConsultant) September 9, 2019

Andrei provided a great overview of the 3 pillars and the strengths and weaknesses of the different pillars. With the basics covered Andrei then dove into the configuration and execution of Istio combined with Jaeger and the corresponding insights available.  including a look at the kinds of visual insights that Jaeger and Kiali provide.  Some probing conversations followed about the relationship to Spring Cloud Sleuth, Open Zipkin and the OpenTracing as a concept more generally.

Andrei’s presentation material can be found in his GitHub repository here.

search-trend-fluentd

Google Analytics on Search Terms

My session followed a pizza break, as there was a delay in its arrival. With everybody having chatted over pizza about OpenTracing, we picked up on FluentD and the Logging aspect to Observability. FluentD, as an open-source project has been growing steadily, and actually baked into several Log Analytics products and services – as the above analytics from Google shows.

The presentation looked at the growing challenges of modern software in terms of making sense of logging.  We explored the capabilities of FluentD before drilling into real-world use cases and potential deployment models.

As you’ll see from the slides we ran a couple of demos. The configuration for the demos can be found at https://github.com/mp3monster/fluentd-demos along with an example payload.

The next meetup we have organized is around Blockchain, all the details can be found at https://www.meetup.com/Oracle-Developer-Meetup-London/events/264661742/.

Other related info …

  • Mastering Distributed Tracing – book review
  • Article direct to LinkedIn – OpenTracing and API Gateways

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

Mastering Distributed Tracing – book review

29 Saturday Jun 2019

Posted by mp3monster in Books, Fluentd, General, Technology

≈ 4 Comments

Tags

CNCF, Jaeger, monitoring, open tracing, Tracing

So recently we have been working on ‘knowing what I don’t know’ when it comes to Open Tracing and how such tech may intersect with traditional logging and the use of FluentD.

As part of that, I have read the Packt book Mastering Distributed Tracing written by Yuri Shkuro who has been key in the OpenTracing API and Jaeger and is the technical lead for Uber’s tracing team.

Whilst I have a good relationship with Packt, the fact they published the book is pretty much coincidental.

Understanding tracing over traditional logging is very important when moving into the world of microservices and reactive frameworks such as Node.js where threads are picked up and put down, you don’t know where and when the next service in a solution will pick up the next related activity. When you add to this solutions are more polyglot than ever – not only in the sense of different languages that may be used but a more diverse source of middle features e.g. historically you’d probably use JMS based messaging if you’re a Java developer and MSMQ for .net. Now you may be using AWS SNS as easily as Kafka. This means the mechanisms for passing and tracing events through these services need to be more unifying than ever.

Complexity of Observability

Continue reading →

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d