My blogging is way down compared with only a post about OKit – OCI Design (on Windows). It largely comes down to lots of work on our Fluentd book. Chapter 6 is now available in the MEAP. As the promo info says …
Last chapter, we touched on the use of the Filter directive. But that was just the tip of the iceberg! In Chapter 6, we’ll plunge below the surface, exploring the when, why, and how of applying filters to give us more insight and precise control over events.
Promo Email from Manning
Earlier chapters have been tweaked, with some additional improvements which will make the live reading experience better.
Another chapter and an appendix should be finding their way to MEAP very soon as it was handed over by our project editor. That will make it seven chapters available, and all the appendices.
Whilst the peer review is taking place the chapter covering plugin development is progressing. The development work has got the basics of the output plugin with log events being stored in Redis and the input being worked on as well. If you want a peak, keep an eye on my GitHub repository (here).
But is isn’t all writing…
I presented on Twitch – you can catch that at https://m.twitch.tv/videos/809295979 I’ve been offered the opportunity to present again, so keep an eye out for something next year.
We recorded a podcast with the excellent guys over at Adventures In DevOps. We don’t have the exact date for the podcast to be released, but I imagine it will sometime during Jan 2021. I’d recommend checking out the podcasts. I’ve been dipping into their back catalogue of recordings and the team ask some really thought provoking questions.
If that wasn’t enough, we’ve been fortunate enough to have some time to talk with leading members of the Fluentd and Fluent Bit projects which was a real pleasure. Hopefully, as we leave this horrendous year behind we’ll get to talk and possibly collaborate some more.
OKit is a tremendous tool for the visual design and development for your Oracle Cloud environment. Visualizing your networks, positioning of service gateways and so on makes it a lot easier than filling in web forms or writing Terraform files as you can see the relationships between the different parts far more easily. For the same reason really that a lot of people use Visio and other tools for this work. The real beauty is that OKit can generate the Terraform and Ansible scripting that can then be used to deliver the implementation.
Okit for visual design of Oracle Cloud
The tool isn’t currently an official Oracle product, but something built by the Oracle A-Team (a small team of gurus who have a role blending developer advocacy, architect supporting customers for the special edge cases and providing thought leadership). But we can hope that someone brings it into the fold and perhaps even incorporates it securely into the cloud dashboard. In the mean time, the code in its entirety is available on GitHub.
Things have been very hectic, so much so we’ve not really had much time to write a blog. Most of which has been related to my Fluentd book, so what has been happening (and keep an eye on my Twitter account for a promo code 🙂 ) …
Several virtual conferences (Open Source Summit, SPOUG and Oracle APAC Groundbreakers to name just a few)
Perhaps the biggest bit has been the book:
First 4 chapters went through a rigorous peer review, as a result a number of improvements have been made,
with Chapter 5 having been reviewed by our technical editor, and little bit of refinement applied it should be reaching MEAP very very soon along with updated appendices,
Chapter 6 has been reviewed by our development editor, so some revisions to apply and then onto the technical editor,
Chapter 7 writing in progress, with about 1/3rd complete including examples of applying scaling configuration that can be run on a desktop
So what is to follow:
We’re on Manning’s Twitch channel to do a session, which will cover Fluentd, some examples, the book and what it will cover,
Once Chapter 7 is done, then we go through a comprehensive review with external input. Depending on the feedback from this, we make another sweep through the existing chapters to make further improvements,
Chapter 8 I suspect will be the hardest to write, as we actually get into creating our own Plugin. So I it maybe a little while before this gets completed. The subsequent chapters will come more easily as we’ve got them part written in a rough draft already,
We have another round of external peer reviewing to come which will cover everything, so I’m sure we’ll be doing some refinements
A podcast recording is scheduled in December.
Talking of Manning on Twitch, this looks like a bit of a hidden Gem – worth looking at not only because of the live stream, but all the previous recordings with other Manning authors are available to watch.
If that isn’t enough with a day job, we have had some major work done on our house. Now we’re moved back in, there are lots of DIY jobs to do, get all the furniture back from storage. Every room apart from the kitchen needs to be painted. But that is my sob story.
I’m hoping to find time to experiment with Oracle’s new cloud native Log Management as this is built on Fluentd foundations and a little bit with API tech – but this is likely to be the Christmas break.
Manning have made section of my book freely available. The excerpt from the first draft of my book Unified Logging with Fluentd illustrates the Fluentd take on Hello World – the extract can be found at http://mng.bz/nzm8. This is from the 1st chapter to help set the scene of how Fluentd can be configured. The following quote comes from one of the peer reviewers:
The extract includes the use of the log simulator tool – https://github.com/mp3monster/LogGenerator which takes some configuration and can either play synthetic data or replay real logs as current log events in what ever format you want to simulate – for example standard Log4J through to Apache Server logs with the relevant time separation between events.
Book Discount …
If this isnt enough temptations, then perhaps saying that on September 22: Deal of the Day book is my book Unified Logging with Fluentd. Use code dotd092220au at https://bit.ly/3mBRLK2
Oracle’s data integration product landscape outside of GoldenGate has with the arrival of Oracle Cloud been confusing at times. This has meant finding the right product documentation can be challenging, and knowing which product to use in your own technology road-map can be harder to formulate. I believe the landscape is starting to settle now. But to understand the position, let’s look at the causes of disturbance and the changes that have occurred.
Why the complexity?
This has come from I think a couple of key factors. The organizational challenges triggered by Thomas Kurian’s departure which has resulted in rather than the product organization being essentially in three parts aligning roughly to Infrastructure, Platform and Applications to being two Infrastructure and Apps. Add to this Oracle’s cloud has gone through two revolutions. Generation 1 now called Classic was essentially a recognition that they needed an answer to Microsoft, Google and AWS quickly (Oracle are now migrating customers off classic). Then came Generation 2, which is a more considered strategy which is leveraging not just the lowest level stack of virtualization (network and compute), but driving changes all the way through the internals of applications by having them leverage common technologies such as microservices along with a raft of software services such as monitoring, logging, metering, events, notifications, FaaS and so on. Essentially all the services they offer are also integral to their own offerings. The nice thing about Gen2 is you can see a strong alignment to CNCF (Cloud Native Compute Foundation) along with other open public standards (formal or de-facto such as Microprofile with Helidon and Apache). As a result despite the perceptions of Oracle, modern apps are standard a better chance of portability.
Impact on ODI
Oracle’s Data Integration capabilities, cloud or otherwise have been best known as Oracle Data Integrator or ODI. The original ODI was the data equivelent of SOA Suite implementing Extract Load Transform (ELT) rather than ETL as it meant the Oracle DB was fully leveraged. This was built on the WebLogic server.
Along Came Cloud
Oracle cloud came along, and there is a natural need for ODI capabilities. Like SOA Suite, the first evolution was to provide ODI Cloud Service just like SOA Suite had SOA Cloud Service. Both are essentially the same on-premises product with UIs to manage deployment and configuration.
ODI’s cloud transformation for the cloud lead ODI CS evolving into DIPC (Data Integration Platform Cloud). Very much an evolution, but with a more web centered experience for designing the integrations. However, DIPC is no longer available (except possible to customers already using it).
Whilst DIPC had been evolving the requirement to continue with on-premises ODI capabilities is needed. Whilst we don’t know for sure, we can speculate that there was divergent development happening creating overhead as ODI as an on-prem solution remained. We then saw the arrival of ODI Marketplace, this provides an easier transition, including taking into account licensing considerations. DIPC has been superseded by ODI Marketplace.
Marketplace
Oracle has developed a Marketplace just like the other major players so that 3rd party vendors could offer their technologies on the Oracle cloud, just as you can with Azure and AWS. But Oracle have also exploited it to offer their traditional products normally associated with on-premise deployments in the cloud. As a result we saw ODI Marketplace. A smart move as it offers the possibility of exploiting on-prem licensing into the cloud along with portability.
So far the ODI capabilities in its different forms continued to leverage its WebLogic foundations. But by this time the Gen2 Oracle Cloud and the organizational structures behind it has been well established, and working up the value stack. Those products in the middleware space have been impacted by both the technology strategies and organization. As a result API for example have been aligned to the OCI native space, but Integration Cloud has been moved towards the Apps space. in many respects this reflects a low code vs code native model.
OCI ODI
Earlier this year (2020) Oracle launched a brand new ODI product, to use its full name Oracle Cloud Infrastructure Data Integration. This is an OCI native (i.e. Gen2 solution leveraging microservices technologies).
This new product appears to be a very much ground up build as it exploits Apache Spark and Function as a Service (FaaS) as foundational elements. As a ground up build, it doesn’t inherit all the adapters the original ODI can offer. This does mean as a solution today it is very focused on some specific needs around supporting the data movement between the various Oracle Cloud storage and Database as a Service solutions rather than general ingestion and extraction processes.
Conclusion
Products are evolving, but the direction of travel appears to be resolving. But we are still in that period where there are capability gaps between the Gen2 native solution and the traditional ODI via Marketplace solution. As a result the question becomes less which product, but when and if I have to invest in using ODI Marketplace how to migrate when the native product catches up.
As some may know we have continued to support our book on Oracle Integration through our site Oracle-Integration.cloud with monthly a roundup of new articles and posts coming from the community. Today we posted the August roundup. This compliments our own writing, and the possibility of adding a cookbook to compliment the original book. There are two sources that we don’t take look at too deeply, the use of videos resources and courses. Primarily because these sources cost, or take time to determine what the content covers. Whilst I’m not providing guarantees, two members of the Ace community that I have had regular contact and respect have developed course ware and video content these are:
For those who regularly follow my blog, I like Groovy as a way of scripting given its portability, proximity to Java, its use of the JVM means its really easy to work with. With the extra language features it makes it easy to work with without needing to setup Maven builds to manage dependencies for something simple. As a result, the utils I’ve produced to support Oracle API Platform CS (here) for example are written with Groovy.
However, not everyone is such a fan, or sees Groovy as niche and, to be honest in the last few years Python has made huge in roads in the scripting space. So I can appreciate the preference for Python to be used. This was really brought home to me by a peer review comments for my latest book project. This got me to thinking what options do I have to remove Groovy from the equation, without resulting in needing to mess with maven POM files. The options as I saw it are:
Replace any use of Groovy extensions with Java libraries
Switch to using Java 11 and the shebang feature (more in a minute)
Compile code with Groovyc and package the jars manually including dependencies.
From Java 11, the ability for a single Java file to be executed without compiling etc was introduced – known as shebang (more here). The issue here is a lot of applications have been certified against Java 8, and in an Enterprise environment this is important. Which means either installing Java 11 locally without changing environment variables, or you need to add code to switch Java versions as necessary. The upside being that you can run your code as a script, and then if necessary build it into a jar to distribute. Aside from the need to switch between JDK versions, Java’s language gets ever richer, but Groovy has a nice set of extensions that make JSON object navigation really nice, and these aren’t necessarily in core Java.
Groovy provides a tool to compile Groovy to bytecode for the JVM (i.e. create class files). From which we can create a JAR file with using the javac command. But this doesn’t bundle the Groovy dependencies necessary to run the jar, as a result Groovy still needs to be installed. You could do this through the use of a maven, Gradle or other build tools. But we’re back to creating a POM file or equivalent., and to be honest creating a POM file from scratch is slow going unless everything lines up neatly with file structures, which it won’t because when you produce a script you’re not expecting to need to worry about packages and associated folder structures.
In looking to see if someone had simplified the process I came across this excellent tool scriptjar (more here). Script jar takes the location of the class files, creates a jar file and loads it with the Groovy jar dependencies, and assembles the manifest file etc. Net result, a jar file created in 2 lines that has no Groovy deployment dependencies to use, and will work with older Java versions such as Java 8.
I started out referring to the API tools, but the same mechanism can be applied to the LogSimulator tool I’ve created that allows you to generate or replay existing logs in real-time or faster than real-time, making it easy to test log monitoring setups from a monitored source all the way through (today it only handles log file replaying or the use of the Java native logging).
Earlier this week I was fortunate enough to be invited to participate in the bol.com’s TechLab podcast on logging. I understand this is the first time that Bol have had an external participant in one of their podcasts, so feeling very honored, not to mention having enjoyed the whole recording process.
bol.com are a European etailer, with a platform based capability. It was really interesting to talk about how they have approached logging in comparison to other organizations I’ve worked with over the years.
I’d recommend checking out not just this podcast (obviously) but others as well given that the guys have been very open about their platform and their experiences.
You can listen to the podcasts from the following links:
The Corona Virus pandemic has had some very odd impacts to the usual technology conferences. We all know that a great number of events have moved from physical to virtual. From a presenter’s perspective has meant the opportunity to present to communities we would normally be less likely submit papers for. For example, the All India Oracle User Group Yatra conference is a significant event in the Oracle Ace/ Groundbreakers calendar – where I have had two papers accepted.
For me to present there, is a challenge as it requires that I lose a week of fee earning time (once you account for the travel). But since the event has gone virtual, I’ve been able to submit a paper, which has been selected …
In addition, the recently postponed Camp Cloud Native which has been lead from the US, has been rescheduled because of the events triggered by the death of George Floyd. As the event is virtual, it wasn’t difficult for the organizers to do the right thing.
With meetups going virtual participating in European events is a lot easier, for example contributing to Aces@Home Episode 3 …
Whilst presenting virtually is the current norm, we are still seeing calls for papers for physical events for October/November this year. It creates a bit of a dilemma. Will it be safe enough to travel? will my family feel comfortable with me travelling (even today we’re hearing news of a serious new outbreak in Beijing)?, will I have to isolate at either end of the journey? Difficult, because I don’t want to submit a paper, and then withdraw at the last minute if circumstances look less secure. Having been involved in user group conference planning, whilst last minute changes will always happen, they can be difficult to cover, so would prefer not to put people in that position.
Whilst I’m not an extrovert by nature, being physically present at a conference, has its plusses – the opportunity to network/meet other Aces to catchup.
So until the physical conference again, be online, be distant, and be safe.
With the impact of Covid19 on the Meetup community really hitting, the Oracle Developer Meetups across Europe have got together and gone on-line. We’re several events in now, whilst Lucas Jellema (AMIS) and Rolando Carrasco (SPS) are leading the charge with a great series of sessions focusing Oracle Cloud Native services complete with practical labs.
Today was one of my sessions, whilst I only co-hosted, we got to hear a great presentation with a heart warming story which in this challenging times seems all the more appropriate. Christian McCabe (Steltix) and Filip Huysmans (Contribute) presented on how a multinational hackerthon spanning South Africa to Belgium was put together to only help children of Christel House (a charity who work to provide education to those who would not normally get access to it). Not only was the hackerthon engineered to given the students a chance to learn and experience software development in a pretty realistic context, it also provided the school with some software to help their everyday activities, in this case managing books in their library.
The hackerthon yielded a lot of successful outcomes (Steltix wrote about it here), but, what was really interesting is that whilst working with the school, children and interns (from both Steltix and Contribute) took a lot lessons away as well.
Finally we should thank Geertjan Wielenga who facilitated and supported the development of the hackerthon.
If you want to know more about the virtual meetups then check our meetup page (https://www.meetup.com/Oracle-Developer-Meetup-London) for both virtual events, and when things change so we can all be a lot closer and be sharing pizza again.
