The last couple of weeks have been pretty exciting. Firstly, we have Fluent Bit 3.1 released, which brings further feature development to Fluent Bit, making it even more capable with Fluent Bit handling of Open Telemetry (OTel).
We’ve been progressing the utility, testing and stabilizing it, and making several releases accordingly. The utility is packaged as a Docker image, and the regression test tool also runs as a Docker image.
Moving forward, we’ll start branching to develop significant changes to keep the trunk stable, including experimenting with the possibility of extending the tool to help port Fluentd to Fluent Bit YAML configurations. The tools won’t be able to do everything, but I hope they will help address the core structural challenges and flag differences needing manual intervention.
Book
The Fluent Bit book has moved into its last phase with the start of copy editing. We have also had a shift in the name to Logs and Telemetry using Fluent Bit, Kubernetes, streaming, and more, or just Logs and Telemetry using Fluent Bit. The book fundamentally hasn’t changed. There is still a lot of Kubernetes-related content, but it helps focus on what Fluent Bit is all about rather than being just another Kubernetes book.
Logs and Telemetry using Fluent Bit, Kubernetes, streaming and more
Fluent Bit supports both a classic configuration file format and a YAML format. The support for YAML reflects industry direction. But if you’ve come from Fluentd to Fluent Bit or have been using Fluent Bit from the early days, you’re likely to be using the classic format. The differences can be seen here:
[SERVICE]
flush 5
log_level debug
[INPUT]
name dummy
dummy {"key" : "value"}
tag blah
[OUTPUT]
name stdout
match *
#
# Classic Format
#
Beyond having a consistent file format, the driver is that some new features are not supported by the classic format. Currently, this is predominantly for Processors; it is fair to assume that any other new major features will likely follow suit.
Migrating from classic to YAML
The process for migrating from classic to YAML has two dimensions:
Change of formatting
YAML indentation and plugins as array elements
addressing any quirks such as wildcard (*) being quoted, etc
Addressing constraints such as:
Using include is more restrictive
Ordering of inputs and outputs is more restrictive – therefore match attributes need to be refined.
None of this is too difficult, but doing it by hand can be laborious and easy to make mistakes. So, we’ve just built a utility that can help with the process. At the moment, this solution is in an MVP state. But we hope to have beefed it up over the coming few weeks. What we plan to do and how to use the util are all covered in the GitHub readme.
A quick update to say that we now have a container configuration in the repository to make the tool very easy to use. All the details will be included in the readme, along with some additional features.
Update 7th July
We’ve progressed past the MVP state now. The detected include statements get incorporated into a proper include block but commented out.
We’ve added an option to convert the attributes to use Kubernetes idiomatic form, i.e., aValue rather than a_value.
The command line has a help option that outputs details such as the control flags.
Update 12th July
In the last couple of days, we pushed a little too quickly to GitHub and discovered we’d broken some cases. We’ve been testing the development a lot more rigorously now, and it helps that we have the regression container image working nicely. The Javadoc is also generating properly.
We have identified some edge cases that need to be sorted, but most scenarios have been correctly handled. Hopefully, we’ll have those edge scenarios fixed tomorrow, so we’ll tag a release version then.
We’ve been busy putting together a number of Oracle Architecture Center assets over the last week. This has included building LogSimulator extensions that can either be run in a very simple manner using just a single file, but limited in the payloads that can be sent to OCI (if you take the appropriate custom file from the LogSimulator you do need to make one minor tweak. But the code has also been added to the Oracle GitHub repository here in a manner that doesn’t require the full tool. There of course a price to pay for the simplified implementation. This comes in the form of the notifications being sent and received being hardwired into the code rather than driven through the insulator’s configuration options.
The decoupling has been done by implementing the interface for the custom methods in a class without the implements declaration, and then we extend the base class and apply the implements declaration at that level.
While notifications could take log events, it is more suited to JSON payloads. But as the simulator can tailor the content being sent using some formatting, it does not care if the provided events to send are pre-formatted as JSON objects making it an easy tool to test the configuration of OCI Notifications.
Unit testing as well
In addition to the new channel, as previously mentioned we have been making some code improvements. To support that we have started to add unit tests, and double checking code will compile under Java. To keep the dependencies down we’re making use of Java assert statements rather than a pretty JUnit. But the implementation ideas are very similar. As the tests use Java asserts the use of asserts does need to be enabled in the command line; for example:
The log simulator we’ve built and written about in the past has had a release made that lines up with the Logging In Action book (v0.1). I am now continuing to add improvements on the main line. Not best Git branching practice, but as I’m working on this solo it doesn’t represent a problem.
If you expect multi line events all you need to do, is add to the properties file a name value pair, with the name FIRSTOFMULTILINEREGEX and the value is a Java/Groovy regular expression which can be used to determine if a line in a log entry is the 1st line in a new log. Then all subsequent log lines are appended to the previous line until a line identifies as a new log entry. The log entry will be written with newline characters in the same place as the read.
In addition to this if the synthetic log events need to be set to be new line then using the ALLOWNL property to be set to true will result in any new line escape sequences (\n) to be made into proper new lines in the output.
The details are all included in the documentation in GitHub.
For those who regularly follow my blog, I like Groovy as a way of scripting given its portability, proximity to Java, its use of the JVM means its really easy to work with. With the extra language features it makes it easy to work with without needing to setup Maven builds to manage dependencies for something simple. As a result, the utils I’ve produced to support Oracle API Platform CS (here) for example are written with Groovy.
However, not everyone is such a fan, or sees Groovy as niche and, to be honest in the last few years Python has made huge in roads in the scripting space. So I can appreciate the preference for Python to be used. This was really brought home to me by a peer review comments for my latest book project. This got me to thinking what options do I have to remove Groovy from the equation, without resulting in needing to mess with maven POM files. The options as I saw it are:
Replace any use of Groovy extensions with Java libraries
Switch to using Java 11 and the shebang feature (more in a minute)
Compile code with Groovyc and package the jars manually including dependencies.
From Java 11, the ability for a single Java file to be executed without compiling etc was introduced – known as shebang (more here). The issue here is a lot of applications have been certified against Java 8, and in an Enterprise environment this is important. Which means either installing Java 11 locally without changing environment variables, or you need to add code to switch Java versions as necessary. The upside being that you can run your code as a script, and then if necessary build it into a jar to distribute. Aside from the need to switch between JDK versions, Java’s language gets ever richer, but Groovy has a nice set of extensions that make JSON object navigation really nice, and these aren’t necessarily in core Java.
Groovy provides a tool to compile Groovy to bytecode for the JVM (i.e. create class files). From which we can create a JAR file with using the javac command. But this doesn’t bundle the Groovy dependencies necessary to run the jar, as a result Groovy still needs to be installed. You could do this through the use of a maven, Gradle or other build tools. But we’re back to creating a POM file or equivalent., and to be honest creating a POM file from scratch is slow going unless everything lines up neatly with file structures, which it won’t because when you produce a script you’re not expecting to need to worry about packages and associated folder structures.
In looking to see if someone had simplified the process I came across this excellent tool scriptjar (more here). Script jar takes the location of the class files, creates a jar file and loads it with the Groovy jar dependencies, and assembles the manifest file etc. Net result, a jar file created in 2 lines that has no Groovy deployment dependencies to use, and will work with older Java versions such as Java 8.
I started out referring to the API tools, but the same mechanism can be applied to the LogSimulator tool I’ve created that allows you to generate or replay existing logs in real-time or faster than real-time, making it easy to test log monitoring setups from a monitored source all the way through (today it only handles log file replaying or the use of the Java native logging).
The last week or two I have been working on a new API Platform utility to add to my existing tools (see here). This tool addresses the question of generating documentation. Much as been said about API documentation and the quality of it, check out these articles :
If you look at these articles and others, there are some common themes, which are:
Document the URI / payload
Describe error handling
Describe contracts such as how many API calls
How the API is authenticated
Apiary covers the first theme to a first class standard, and you will see Apiary called out for its ability to document APIs in a lot of articles. Well written API Blueprints will cover the bulk of the second bullet. But the other points tend to fall outside of a Blueprint and fit more the API Policies and their use.
Not everyone is so commited or enjoys writing documentation. The other driver for going beyond the use of Apiary is that some organizations feel the need to have a traditional word style document to capture/define an API’s contract in detail. With the API Platform the management portal enables an API to be published into the developer portal with the Apiary definition and a markdown file for further documentation.
With version 11 of Sparx‘ Enterprise Architect tool a new cloud feature was introduced to support team working, which previously had been achieved using a shared Database.
When we heard about EA Cloud, both myself and my colleagues got rather excited, thinking that this would be the opportunity to offload the effort of looking after a central DB (making sure backups happened, fine tuning the DB settings and so on) plus maintaining the platform’s patching for security etc. Not only that through the cloud capability we could host the repository that made it very easy for the team to access the repository on the move without needing to have another whole in our corporate firewall etc.
Unfortunately, EA Cloud provides all the software to establish a cloud based repository – which can be used through firewalls etc – HTTPS traffic rather than DB connectors on unusual ports but not the hosting. This seems to a bit of a missed opportunity for Sparx who already have to deal with all of these points to host 3 demo cloud servers. So the next step of instantiating a server for a regular on going fee doesn’t seem too challenging, not to mention promotes customer tie in, plus the ability to capture some potentially interesting metrics about its users (e.g. which modelling techniques are most popular etc). Having looked at Sparx partners they don’t offer the capability either which is a shame.
Phil (aka MP3Monster)'s Blog 
You must be logged in to post a comment.