Tag Archives: java

Logging Frameworks that can communicate directly with Fluent Bit

13 Thursday Jun 2024

Posted by in Fluentbit, General, Technology

≈ Leave a comment

Tags

, , , , , , , , , , , , , , ,

While the typical norm is for applications to write their logs to file or to stdout (console), this isn’t the most efficient way to handle logs (particularly given I/O performance for the storage devices). Many logging frameworks have addressed this by providing more direct outputs to commonly used services such as ElasticSearch and OpenSearch. This is fine, but the only downside is that there is no means for an intermediary layer to preprocess, filter, and route (potentially to multiple services). These constraints can be overcome by using an intermediary service such as Fluent Bit or Fluentd.

Many logging frameworks can work with Fluentd by supporting the HTTP or Forward protocols Fluentd supports out of the box. But as both Fluent Bit and Fluentd are interchangeable with these protocols and logging framework that supports Fluentd, by implication also supports Fluent Bit, not to mention Fluent Bit supports OpenTelemetry.

The following table identifies a range of frameworks that can support communicating directly with Fluent Bit. It is not exhaustive but does provide broad coverage. We’ll update the table as we discover new frameworks that can communicate directly.

Latest Version …

Logging Frameworks and Fluent Bit and Fluentd connectivity
LanguageFramework / LibraryProtocol(s)Commentary
JavaLog4J2HTTP AppenderSend JSON payloads over HTTP (use HTTP input plugin)
Javafluent-logger-javaForward
Pythoncore languageHTTP HandlerProvides the means to send logs over HTTP – means Fluent Bit input handler can manage
Pythonfluent-logger-python
Fluent Logger		ForwardUses the Forward protocol meaning it can gain the efficiencies from msgpack.
Maintained by the Fluent community
Node.jsfluent-logger-nodeForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Node.jsWinstonHTTP

Forward		Winston is designed as a simple and universal logging library supporting multiple transports.
Winston includes transport support for HTTP in its core. There is also a Transport implementation for native Fluent https://github.com/sakamoto-san/winston-fluent
Node.jsPino (Pino-fluent extension)Logger integrated into the Pino logging framework
Go (Golang)fluent-logger-golangForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
.Net (C# VB.Net etc)NLog (NLog.Targets.Fluentd)An NLog target – works with .Net
.Net (C# VB.Net etc)Log4NetLog4Net Appender
.NetSerilog (Fluent Sink)Forward and HTTPSupports both HTTP and nativbe Fluentd/FluentBit
Rubyfluent-logger-rubyForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
PHPfluent-logger-phpForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Perlfluent-logger-perlForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Scalafluent-logger-scalaForwardIt uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
Erlangfluent-logger-erlangForward
It uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
OCAMLfluent-logger-ocamlForward
It uses the Forward protocol, meaning it can gain efficiencies from msgpack.
Maintained by the Fluent community
RustRust Logging framework extension for Fluent BitRust crate for logging to Fluent Bit
DelphiQuickloggerHTTP

Java 21 & GraalVM — lots to be excited about

19 Tuesday Sep 2023

Posted by in General, java, Technology

≈ Leave a comment

Tags

, , , ,

Today, Java 21 has reached General Availability (GA) with some important new features in the language mainstream (i.e., not requiring preview flags enabled), and Oracle will be supporting Java 21 as a Long long-term support (guaranteed at least 3years of free support (2years to the next LTS + 1 yr overlap) and then for at least an additional 5 years under support subscription). Everyone is talking about virtual threads. Interestingly the new virtual threads mean, in the majority of cases, we no longer need to handle the complexities of reactive programming – not my point of view, but a view expressed earlier today by Tomas Langer, the architect for Helidon. For old hands like myself – this is a blessing as the old-style threading comes more naturally. There are a lot of other smaller features coming through in the language with this, such as records, Z Garbage Collector, and better support for Key Encapsulation management. All the fine details can be found on the Oracle Java blog.

Java.dev has a new Playground, which allows you to write some Java code in the browser and run it. No local JDK or IDE is needed. Great for trying out code, like pattern matching for switch statements.

GraalVM gets a new release with Java 21. Along with some other cool features. Including being able to deploy Graal’s polyglot features with just the support for the languages you want, meaning that the GraalVM footprint is kept as small as you need in containers. This decoupling is supported with Maven and Gradle configurations. With this are enhancements for the Just-in-Time (JIT) and Ahead-of-Time (AOT) performance – read more about this in Alina Yurenko‘s blog.

Running Groovy scripts as JARs

03 Monday Aug 2020

Posted by in General

≈ Leave a comment

Tags

, , , , , ,

For those who regularly follow my blog, I like Groovy as a way of scripting given its portability, proximity to Java, its use of the JVM means its really easy to work with. With the extra language features it makes it easy to work with without needing to setup Maven builds to manage dependencies for something simple. As a result, the utils I’ve produced to support Oracle API Platform CS (here) for example are written with Groovy.

However, not everyone is such a fan, or sees Groovy as niche and, to be honest in the last few years Python has made huge in roads in the scripting space. So I can appreciate the preference for Python to be used. This was really brought home to me by a peer review comments for my latest book project. This got me to thinking what options do I have to remove Groovy from the equation, without resulting in needing to mess with maven POM files. The options as I saw it are:

  • Replace any use of Groovy extensions with Java libraries
  • Switch to using Java 11 and the shebang feature (more in a minute)
  • Compile code with Groovyc and package the jars manually including dependencies.

From Java 11, the ability for a single Java file to be executed without compiling etc was introduced – known as shebang (more here). The issue here is a lot of applications have been certified against Java 8, and in an Enterprise environment this is important. Which means either installing Java 11 locally without changing environment variables, or you need to add code to switch Java versions as necessary. The upside being that you can run your code as a script, and then if necessary build it into a jar to distribute. Aside from the need to switch between JDK versions, Java’s language gets ever richer, but Groovy has a nice set of extensions that make JSON object navigation really nice, and these aren’t necessarily in core Java.

Groovy provides a tool to compile Groovy to bytecode for the JVM (i.e. create class files). From which we can create a JAR file with using the javac command. But this doesn’t bundle the Groovy dependencies necessary to run the jar, as a result Groovy still needs to be installed. You could do this through the use of a maven, Gradle or other build tools. But we’re back to creating a POM file or equivalent., and to be honest creating a POM file from scratch is slow going unless everything lines up neatly with file structures, which it won’t because when you produce a script you’re not expecting to need to worry about packages and associated folder structures.

In looking to see if someone had simplified the process I came across this excellent tool scriptjar (more here). Script jar takes the location of the class files, creates a jar file and loads it with the Groovy jar dependencies, and assembles the manifest file etc. Net result, a jar file created in 2 lines that has no Groovy deployment dependencies to use, and will work with older Java versions such as Java 8.

I started out referring to the API tools, but the same mechanism can be applied to the LogSimulator tool I’ve created that allows you to generate or replay existing logs in real-time or faster than real-time, making it easy to test log monitoring setups from a monitored source all the way through (today it only handles log file replaying or the use of the Java native logging).

Exploring Helidon – Part 1

01 Tuesday Jan 2019

Posted by in General, Helidon

≈ 4 Comments

Tags

, , ,

So I recently blogged (here) about the announcement of Helidon – the open source project from Oracle to provide a microservice app server that includes optional support for the J2EE Microprofile.

This is the first of what will probably become a series of blogs about Helidon, particularly in its SE form (non J2EE Micro-Profile) as Micro Profile and the wider J2EE model in general will have been documented more widely.

Hello World

Helidon comes with a quick start example app implemented both in SE and MP forms.  It is worth following the very simple instructions provided by the Helidon site to instantiate both versions of the Hello World app as it provides a good way to start to understand the differences in the way Helidon can be used.

The thing that really jumps out when you compare the code (for me at least) is the fact that the SE code being driven from values loading from configuration is more dynamic. The configuration can be sourced in a number of different ways from YAML files to etcd.So for our for first experiment we took the hello World app, and made the path /greet dynamic by loading the path from from some additional configuration. Enhancing the main with :

private static Routing createMultiRouting() {
Map greetingConfig = Config.create().get(“greeting”).asMap();
Routing.Builder routing = Routing.builder();
if (!greetingConfig.isEmpty())
{
greetingConfig.forEach((k,v)->
{
System.out.println (“Read config value>” + k + “=” + v + “<“);
// as the key is the fully qualified name, I just want the last piece,
// so let’s strip it to be a substring
String key = (String)k;
key = key.substring(key.lastIndexOf(“.”)+1);
// each different URI should have its own instance of the Greet Service
// with its tailored key which will mean it responds with the key e.g. France
routing.register(JsonSupport.get())
.register(“/”+v, new GreetService(key));
});
}
else
{
System.out.println(“No config\n”);
}
return routing.build();
}
 

We can create URLs for greetings in different languages, and see the different instances of the Service object responding to the web calls. Whilst many may associate this approach with Node.js for me it felt more like the webserver multiplexer (MUX) such as Gorilla used with Go (you can see what I mean here).

Helidon and On the Road

Two of the leading figures Tomas Langer and Dmitry Kornikov will be presenting at a number of meet-ups in Europe, including the London Meetup (go here)

Part 1?

Yes, I will be blogging more about Helidon as soon as I can, but presently wrapping up a white paper and running an API Design training session soon.

GraalVM – why a different VM?

09 Thursday Aug 2018

Posted by in development, General, Oracle, Technology

≈ Leave a comment

Tags

, , , ,

If you read press around Java you’ll have come across references to GraalVM. So what is it and why would I use it?

There is an excellent podcast from Software Engineering Daily that digs into the subject and can be found here and here. But let draw out some of the reasons as to why GraalVM is interesting.

Whilst multiple languages on top of the JVM is nothing new, such as Scala , Kotlin, and Groovy to name a few, GraalVM through the use of its Truffle framework takes it a new level. Truffle provides an Abstract Syntax Tree (AST) which describes the language syntax (more here about AST). The net result is any language can be described and therefore executed using the GraalVM. To this end the GraalVM team have got Node and JavaScript ported in addition to defining existing languages using this approach. Not all of this is proven robustly in production, but some of the languages VM certainly is, for example Twitter have been using the Scala port.

Because the languages are described through the same framework this means the work to optimise the VM performance becomes a lot easier.

It would be easy to assume that using the framework would mean the execution of languages using this mechanism would slow be slower. But, Truffle works by translating the code to standard byte code before execution, so ‘ported’ languages are now no less efficient than Java come runtime.

There is an interesting bi-product of this model, that at runtime with the right object exposures it is possible for multiple languages to interact with the same object easily, no JNI or dropping to the lowest common denominator such as a JSON+REST. This does raise interesting possibilities for thick client solutions or polyglot monoliths!

Probably one of the biggest pay offs for using GraalVM and its ability to run multiple languages is that the base Container images can be simplified as you don’t need different container images. This makes the work of patching and testing configurations of these container a lot simpler as the permutations will drop, particularly for organisations that have wholehearted embraced polyglot micro-service ideas.

One common reason for changing implementations of the JVM particularly at the more performance sensitive use cases (checkout Azul as an example) is how the JVM is optimised and the JIT algorithms and processes particularly the Garbage Collector work (checkout this list of JVMs. For example GraalVM will provide better performance for processes that less heap hungry than Oracle’s JVM.

It is interesting that Oracle are investing in a new VM when it wasn’t that long ago that JRockit was wound down. Given the legal dispute between Oracle and Google (see here) the new VM would give Google a means to escape from the copyright breaches and retain support for Java.

Oracle PaaS – the Good and opportunities to get better

19 Saturday Mar 2016

Posted by in General, Java Cloud, Oracle, Technology

≈ Leave a comment

Tags

, , , , , ,

IMG_0180Although Oracle have been late to the cloud party they are certainly making up for it, by bringing products to the cloud at an amazing pace, and using their core products to build out new offerings at a rate that will mean they will at least catch  all the competition across  the breadth of PaaS very quickly.

When it comes to taking on Oracle PaaS  it does have  some quirks, some relate to Oracle’s normal licensing approach, and others I’m told relate to the way US accounting has to work when it comes to realised revenue. A couple of other characteristics I suspect are linked to the fact that the infrastructure for Oracle’s cloud is still being rolled out and grown for capacity.

So firstly the carry over – well outside of a trial account you need to agree and sign a general agreement which provides an overarching legal framework defining terms, conditions and liabilities. This makes dealing with each subsequent purchase a little simpler. Rather than purchasing services as you go, you then purchase credits from Oracle which have a maximum life of 1 year. This does mean you’re not got a pure OPEX spend model – although you do stand a chance of negotiating a better deal as the numbers are naturally bigger. As part of the agreement you’ll get a rate card, so different services cost different amounts – for example a standard edition database will cost x and an enterprise high performance version will cost a bit more. The credits are for specific product families such as SaaS products, products in the PaaS domain for example document cloud, Java cloud, SOA and so on. But make sure the products you might want are in the families you get credits for, there is the odd surprise for example MBaaS isn’t in the same family as the integration products.

In addition your negotiation you need to consider whether  services are in metered or unmetered models.  Unmetered means you agree a level of capacity for the year. This will obviously work out cheaper than a metered model where you can use up your credits as you choose, with different metering rates – for example hourly and monthly. When this was first explained it looked really good for dealing with the situation of having a baseline demand which could be unmetered and then purchasing metered services to capacity burst. Sadly this isn’t possible out of the box. I suspect because of the way Oracle cloud allocates workload to different work domains. So bursting workload would have to be done as if you’re bursting in 2 different clouds. So if you have a dynamic load you either go unmetered to your maximum demand or metered for everything. Either way you’re not getting the best in terms of cost management.  I have to admit I don’t know whether the likes of AWS and Azure when you enter into long term agreements have the same challenges.

One the positive side, with the credits you can then purchase a broad range of configurations of products from just ADB schema all the way a full size  Exadata setup. So performing PoCs is pretty easy and figuring out scaling just means burning your credits quickly and instantiating more capacity.

Before getting into instantiating your cloud instances you’d best  setup access controls to allow people access controls to creating instances. Then you can start creating instances of the products you want. Make sure you protect your credentials as the way things are setup anyone else recovering them will be a problem.

With services such as SOA and Java you do need to go through the process of creating the different layers, storage, then the database and so on. But unlike building on premise each step only requires a couple of clicks and your done. To put it into context the first time I built a small footprint 11g environment took me a couple of days to work my way through on my own create a DB, deploy RCU,Weblogic, SOA and AIA foundation (no load balancing or security etc) and was no way near secure as a cloud instance. Oracle PaaS in three hours we:

  • Meet our Customer Success Manager (more on this shortly)
  • Got the utilities such as putty installed on my laptop
  •  Walked through putty’s key generation quirks and how to avoid the gotchyas
  • being walked through the process, of setting up management rights to our credits and instance creation
  • Instaniated storage, debated on the DB option to use, created the SOA CS instance with OSB, a load balancer and configured SSH security and web access routes to our cloud. Plus setup my developers
  • Had a couple cups of coffee and ordered lunch

With SOA CS and atleast some of the other cloud offerings you also get SSH access to the OS so you can tinker and tune your SOA container and Weblogic etc. Some would argue that totally undermines the ideal of PaaS and that exploiting such a capability means you can end up customising your deployment to the point it will break the moment an update or patch comes along. So it is very double edged. In my mind (but I’m a techie at heart so seeing the engine running is always interesting) it’s good, but must be handled with great care. As they say – with great freedom comes great responsibility.

One of the real wins is that Oracle allocate customers a Cloud Success Manager who are tasked with enabling you to use the Oracle cloud – any problems, guidance needed can be addressed through these people. A cynic might say they exist to help you spend money which becomes released revenue. But our experience is the CSMs are genuinely enthusiastic and helpful  – answering questions at 6pm on a Friday (despite my school boy error).

So in our experience so far I’d suggest Oracle could do two things to really make a big advancement – commercially atleast:

  • Allow payments to be made on a reoccurring model as an alternative to the credits model, perhaps this approach restricts you to metered only services
  • Allow metered and unmetered services to be utilised together – perhaps as a stretched cluster mentality.

This was first made available at https://community.oracle.com/groups/united-kingdom-user-group/blog/2016/03/25/starting-with-oracles-soa-cs 

Oracle Node.js cloud service

31 Thursday Dec 2015

Posted by in General, Java Cloud, NodeJS Cloud, Oracle, Technology

≈ 1 Comment

Tags

, , , , ,

A while back I posted about using the use of Node.js cloud service Oracle had marked as coming soon (Blog post here). Well we have checked back to see if the free trial is openly available and it still appears not yet to be the case.  But more than that, Oracle have reorganised the capability here to form what they are now calling Application Container Cloud (ACC). The application container cloud provides a number of options for running Node.js or a pure play J2SE solution. 
   
The good news is that there is a lot more detail of what the options are with this cloud which includes just Node.js – the details can be seen here. So node 0.10 and 0.12 are supported and JDKs 7 & 8 are supported. With the JDK you also get the use of cruise control. The metering periods go down to the hour as well which is great for PoC activities. The level of detail provided, suggests that these cloud solutions are currently available to partners and paying customers (the JDK service is certainly the case based on discussions I have had with my account manager). So hopefully as Oracle rollout their cloud offerings into data centres and capacity grows we should see public trial access.

Exclamation Mark Note: ACC has now been superseded by Oracle Kubernetes Engine (OKE)

Java EE Development with Eclipse 2nd Edition

06 Tuesday Oct 2015

Posted by in Book Reviews, Books, Packt, Technology

≈ Leave a comment

Tags

, , , ,

I have just been told that the 2nd Edition of Java EE Development with Eclipse has now been published.  This is another book that I have supported as a technical reviewer.  Looking forward to receiving my print copy so I can see how my suggestions and feedback have carried through to the final copy.

You can see the book on the Packt site with  https://www.packtpub.com/application-development/java-ee-development-eclipse-second-edition  or Amazon here

 

IMG_2232

Here is the finished article!

Push Notifications with a Bit of Java

24 Saturday Jan 2015

Posted by in Java Cloud, Oracle, Technology

≈ Leave a comment

Tags

, , , , , , , , , , , , , , , , ,

So continuing from my previous posts:

We’re going to use the Prowl API and create the equivalent classic “Hello World” App using the push framework – but cutting out the need for Growl etc.  For this blog post we’re not going to use the Oracle Java Cloud as we need to see the code working locally and get ready to promote the code to the cloud.  Once we’ve got some code working we can look at setting up the Java Cloud environment, package and promote what we have here using our IDE into the cloud environment.

As we’re cutting Java code now – you can obviously use your own preferred IDE, I’m going to use JDeveloper 12c if for no other reason than it being a huge improvement on 11g (download here) and I’ve become somewhat disappointed with Eclipse.  Whilst talking about IDEs;  you should be aware that Oracle provide an Oracle Cloud SDK which integrates with a number of IDEs to make some of the interactions with the cloud straight forward.

The SDK provides Ant and Maven scripts to help the build and deploy process – so we will be using those later, plus command line tools to help manage other activities, a number of code examples and HTML documentation.  To setup the SDK you will need to unpack the file into a folder and add that folder into your PATH environment variables. The bundle includes a readme that contains just enough to show what is required to get unpacked and make the command line tools work.

To download from Oracle you will need to setup an Oracle Technology Network (OTN) account – so if you don’t have one now is the time to create one – there is no cost to this, we’re going to need

We could use the REST based API that is provided by the ProwlApp, but at least to start with we’re going to follow the approach using a library to make using the API very simple. By using the API provided on SourceForge (jProwlAPI). Using an API will allow us to show the use of 3rd party libraries in the cloud deployment but also follows some of the Oracle ideas of offering ‘adaptors’ to simplify integration.

So you need to download:

So with this downloaded we are going to:

Setup an new Maven based project (or copy my file structure into place and import) using a maven quickstart archtetype (org.apache.maven.archetypes:maven-archetype-quickstart) . We will probably need to modify this later to leverage the full cloud capable archetype. This will build your project environment and retrieve a bunch of plugins you might need.

Next lets take a peak inside of the jProwlAPI download. You’ll see an example bit of java that shows how to fire the API.  Rather than tinker with this we have created a small package and JUnit test as we would if writing a proper solution created with the maven archetype.

PushedHelloWorld directory structure

We also need to make the JProwlAPI jar file available to the project. So we use maven pattern, and create a folder called lib and copy the jar file into it. We then add the lib folder to project setup.

 

 

pushedhello-RunConfig

To be able to create the deployable artefact we need to load the jar file into the local repository, which we can do with a command line instruction (presuming maven is also available by your PATH variable).

mvn install:install-file -Dfile=./lib/JProwlAPI-0.5.jar -DgroupId=prowl -DartefactId=JProwlAPI -Dversion=12.1.3-0-0 -Dpackage=jar

We’ll come back to the command line in a bit, but within JDeveloper the code I have provided needs 1 change from yourself – replace the references to –YourAPIKey– in the run execute command and in the JUnit class with your own key.

JDEveloper 12c Run Config

In the the ProwlProcessor class I have included a man in method so we can just execute the class to see things working. So having done that we can then repeat by running the class via the JUnit test. You should see the same result. When we we’ve deployed or class to the cloud we can use the JUnit test to invoke the cloud.

ProwlProcessor - main method

The last step, within the IDE we have been compiling to get the class, but not creating a deployable jar file. We could do this with the IDE but we would also in a real development condition be creating artefacts via a Continuous Integration tooling which will effectively fire the maven command line like interface. So let’s do that to create the jar files, using the following command:

mvn clean package

You should then see a folder created if not already there and a jar file reflecting the values in the POM file, that we can see below.

JDeveloper 12C - view of a simple POM

So we have enough now we could in theory deploy a jar to a weblogic container and fire it from a Unit test. In the next post we’ll deploy and execute the unit test, and throw a crude front end into the mix.

Oracle Java & Node.JS Cloud Blogs Coming

23 Tuesday Dec 2014

Posted by in General, Java Cloud, NodeJS Cloud, Oracle, Technology

≈ 1 Comment

Tags

, , ,

Over the coming months I’m planning on running a series of blogs on getting setup and using Oracle’s Java Cloud and the upcoming Oracle Node.JS Cloud when we can get access to it.

In both cases we’re aiming to demo the capabilities, setup eyc around the classic Hello World – but with a bit of a twist. Rather than simply sending to the console we’re going to use a mobile push notification – without resorting to having to build a mobile app.

By taking this approach to hello world aside from keeping it simple we can see how to bring 3rd party APIs into the mix.  The first couple of posts wont need to much on the Oracle front – as we’ll walk through getting things setup and running a proof without the cloud stuff. As they say keep it simple stupid.

So that you can see the blog entries for each of this two stories, I specifically setup in my blog two entry categories:

A few posts will obviously be common to both.