• Home
  • About
    • Background
    • LinkedIn
    • Presenting Activities
    • Internet Profile
    • About
    • www.cloud-native.info
    • AI Disclosure
  • Observability, Fluent(d) Bit & OpAMP
    • OpAMP Product
    • OpAMP Overview
    • Log Generator
    • Fluent Bit Classic to YAML Format configurations
  • My Books
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: loggenerator

LogGenerator becomes SignalGenerator

11 Wednesday Nov 2026

Posted by mp3monster in development, Fluent Observability, General, logsimulator, OpAMP, Technology

≈ Leave a comment

Tags

Fluent Bit, FluentBit, Fluentd, log simulator, loggenerator, logs, metrics, observability, Open Telemetry, OpenTelemetry, OTel, python, signal generator, signals, Technology, testing, tools, Traces, Tracing, unified logging

When I wrote Logging In Action, I found that to help with setting up and testing logging pipelines, having a utility that could just take existing logs and play them so it looked like a real application was generating the logs was helpful – not only for the book, but in everyday work.

The natural evolution was: can I also create and run synthetic logs through observability systems so we can ensure log thresholds, filters, alerts, etc. worked as expected. Not to mention taking synthetic or real logs and applying some basic reformatting so we could mimic Apache or Nginx web servers, for example.

So the LogGenerator (sometimes called Log Simulator) was born and made available. At the time, my development skills were centred on Java (and I loved Groovy for its convenience and development speed). So we built the solution using Groovy and Java -the net result is that you could create a deployable JAR or run it without the need to first establish a Maven pipeline (without sacrificing the ability to keep polymorphism and abstract layers).

The LogGenerator has had a number of enhancements, so we could use it to check not only tools like Fluentd and Fluent Bit, but also cloud observability tooling such as OCI. This was made to be a pluggable framework so you could add custom targets easily.

Time moves on …

Python continues to grow in popularity and is the default language bundled with Linux. I’m comfortable developing in Python now, and we have the tools I enjoyed with Groovy and Java (no need to mess with build pipelines for quick changes, but it can also be packaged and easily deployed for mainstream adoption).

But it isn’t programming language evolution that has triggered change. OpenTelemetry standards and the golden signals of Logs, Traces and Metrics are the norm. So applying the same principles to the Traces and metrics that the LogGenerator used for logs is a natural evolution.

Deployment

It was pointed out to me that, even if people don’t take the full OpAMP solution we built, the simulator component may be attractive in its own right. This got us to thinking about whether there was value in pairing the SignalGenerator with the OpAMP simulator.

As a result, we created an optional feature that lets us deploy SignalGenerator as an extension of the OpAMP simulator. This gives us multiple deployment options.

  • A free-standing solution in its own right, with no need for anything else. Ideal for dropping into test environments to exercise your Observability configuration and tuning without needing start-up applications that require large deployments with complex inputs to generate unhappy path signals.
  • Deploy in a supervised manner using OpAMP – ideal if you want to drive synthetic Observability exercises in a distributed manner.
  • Plug into the OpAMP simulator – fewer processes running, easier to create custom commands through the OpAMP framework to get the generator to do things from pushing out new data sources to be replayed, to having signal generators simulate Observability data that reflects a distributed cascading failure as means to evaluate smarter AI-driven processes that should be looking for correlations and cause and effect.

The last part with custom messaging over OpAMP is not yet in place, but it’s a candidate for enhancement.

New home

As the Logging in Action book uses the LogGenerator, we didn’t want to make this level of change in the current repository. So we have a new repository within the OpAMP-Observe GitHub organization which has feature parity, but also has a mechanism for simulating metrics (pushing, or allowing the data to be scraped).

The documentation to link the repositories and the GitHub documentation together is a little bit behind, and we need to figure out how to create a manageable playback structure for Traces before we enable that feature.

Resources

  • Logging In Action
  • LogGenerator aka LogSimulator
  • Maven
  • SignalGenerator
  • OpAMP-Observe (repository)
  • OpAMP.info
Signal Generator Logo

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

OCI Notifications through the LogSimulator

04 Friday Nov 2022

Posted by mp3monster in development, General, logsimulator, Oracle, Technology

≈ Leave a comment

Tags

events, loggenerator, logsimulator, Notifications, OCI, open-source, Oracle, tool

We’ve been busy putting together a number of Oracle Architecture Center assets over the last week. This has included building LogSimulator extensions that can either be run in a very simple manner using just a single file, but limited in the payloads that can be sent to OCI (if you take the appropriate custom file from the LogSimulator you do need to make one minor tweak. But the code has also been added to the Oracle GitHub repository here in a manner that doesn’t require the full tool. There of course a price to pay for the simplified implementation. This comes in the form of the notifications being sent and received being hardwired into the code rather than driven through the insulator’s configuration options.

The decoupling has been done by implementing the interface for the custom methods in a class without the implements declaration, and then we extend the base class and apply the implements declaration at that level.

While notifications could take log events, it is more suited to JSON payloads. But as the simulator can tailor the content being sent using some formatting, it does not care if the provided events to send are pre-formatted as JSON objects making it an easy tool to test the configuration of OCI Notifications.

Unit testing as well

In addition to the new channel, as previously mentioned we have been making some code improvements. To support that we have started to add unit tests, and double checking code will compile under Java. To keep the dependencies down we’re making use of Java assert statements rather than a pretty JUnit. But the implementation ideas are very similar. As the tests use Java asserts the use of asserts does need to be enabled in the command line; for example:

Groovy test.groovy -enableasserts

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...

LogSimulator New Feature – Custom Targets with OCI Logging example

14 Friday Oct 2022

Posted by mp3monster in Cloud, development, General, logsimulator, Oracle, Technology

≈ Leave a comment

Tags

book, loggenerator, logging, logsimulator, OCI, Prometheus

Those who have been using my Logging in Action book will know that to help test the configuration of monitoring tools including Fluentd we have built a LogGenerator that can very easily play and replay logging events into a variety of destinations and formats. all written in Groovy to make the utility easy to run as a script and extend without needing to set up a proper Java development environment.

With the number of different destinations built into the script and the logic to load the source log events and format them the utility is getting rather large for a single file. Rather than letting it continue to grow as we add more destinations to pump log events too, I’ve extended the implementation so you can point to a Groovy file that implements the logic to send the log events. It only requires three simple methods to be implemented.

To demonstrate the feature we have created a custom extension and fully documented it. The extension allows you to send log events to the OCI Logging service. This includes an optional crude aggregation mechanism as sending individual log events is a little inefficient over REST. By doing this we can send synthetic or playback logs as if we’re an application in real-life to ensure that any alerting or routing for the logging works properly before we get anywhere production and do not need to run the application and induce error events.

Beyond this, we’re also thinking about creating a plugin to fire log events at Prometheus so we can send events using the Prometheus pushgateway. As a result, we can tune Prometheus’ configuration.

More improvements – refactoring the existing code

We will refactor the existing code to use the same approach which should make the code more maintainable, but the changes won’t stop the utility from working as it always has (so we won’t break out the existing output channels from the core).

We have also started to improve the code commenting – so hopefully it will make the code a bit more navigable.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Share on LinkedIn (Opens in new window) LinkedIn
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • Share on Bluesky (Opens in new window) Bluesky
Like Loading...
    • About
      • AI Disclosure
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • OpAMP explained: the control plane for observability agents
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Ace Director Alumni

    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
    • Enterprise architecture
    • General
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Security
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • ITSO & OEAF
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,628 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    November 2026
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    30  
    « Oct    

    Speaker Recognition

    Open Source Summit Speaker

    Twitter

    Tweets by mp3monster

    History

    Flickr Pics

    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      Phil (aka MP3Monster)'s Blog
      Join 232 other subscribers

      Have a WordPress.com account? Log in now.

    • Phil (aka MP3Monster)'s Blog
      View site in Reader
      Manage subscriptionsSign upLog in
      Report this content
      Collapse this bar
    Loading Comments...

    You must be logged in to post a comment.

    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d