Avoid creating an event storm when using social outputs with Fluentd

14 Thursday Apr 2022

Posted by in Fluentd, General, Technology

≈ Leave a comment

Tags

, , , , , ,

Fluentd has an incredible catalogue of plugins including notification and collaboration channels from good old-fashioned email through to slack, teams, and others.

The thing to remember if you use these channels is that if you’re sending errors, from application logs it isn’t unusual for there to be multiple error events as a root event can cause a cascade of related issues. For example, if your code is writing transactions to a database and the database goes down with no failover mechanism, then your code will most likely experience an error, roll back the transaction perhaps to some sort of queue, and then try to process the next event. Which will again fail. This is the classic situation where multiple errors will get reported for the same issue. This problem is often referred to as a mail storm given that there was a time when we didn’t have social collaboration tools and everyone used email.

There are several ways to overcome this problem. But the most simple and elegant of these is using the suppress plugin in its filter mode.

<filter **>
  @type suppress
  interval 60       # period in seconds when the condition to supress is triggered
  num 2             # number of occurences of a value before suppressing
  attr_keys source  # the element of the event to consider.
</filter>

In this example if we encounter an event with an attribute called source containing the same value twice then the suppression will kick in for 60 seconds. If you want the key to the valuebeing checked to be the tag then simply omit the attr_keys parameter.

Of course, we don’t want the suppression to kick in if the same value in the attribute keys occured once every few hours. To address this the occurence count is applied over not a time period, but a number of events received by the configuration of max_slot_num which defaults to 10k, but resets

In the filter mode, this plugin is best positionbed immediately before the match block. This means we don’t accidentally suppress messages before they are routed anywhere else.

For the purposes of a demo this is less of an issue. But for a realworld use case would probably benefit from some tuning. All the documentation for this plugin is at https://github.com/fujiwara/fluent-plugin-suppress

Web Assembly – Free O’Reilly Book

13 Wednesday Apr 2022

Posted by in Books, General

≈ Leave a comment

Tags

, , , ,

Red Panda is currently making the O’Reilly book by Brian Sletten – WebAssembly the Definitive Guide free to download (well, the price of your email address).

So why is worth snagging a download? Well, WebAssembly (Wasm) is the new way to deliver browser-based solutions to a client without having to develop with JavaScript. As a result, you have the chance to reuse more of your existing code. It also has the benefit of having been precompiled to a binary form, offering code security and compact download. But Wasm also has been seeing adoption in back-end server solutions as well, particularly for feature extensions, etc.

Knowledge Resources from Oracle

05 Tuesday Apr 2022

Posted by in Cloud, General, Oracle, Technology

≈ Leave a comment

Tags

, , ,

Before joining Oracle I used to typically refer to a couple of key resources from Oracle – docs.oracle.com, and occasionally developer.oracle.com and ateam-oracle.com. We’d obviously use cloud.oracle.com and the main oracle.com to be able to reference published stats, success references etc. Now I’m part of the company and working in the OCI product team with an outbound side of things, I needed to gem up on all the assets that exist. So that we can help contribute, and ensure that they are up to date etc. In doing so, the number of resources available is so much more than I’d realized.

Upon reflection, this may have been from the fact we didn’t drill down deeply enough, also in part that Capgemini has its own approaches and strategies as well.

This in part is linked to the organizational structures e.g. OCI Product Management’s outbound work overlaps with the Marketing Developer Relations, for example, something that is inevitable in an organization that provides such a diverse portfolio of products.

For my own benefit, and for others to exploit, the following table summarises the different areas of information. The nature of the content and – where content overlaps or is presented in different ways.

We’ve moved this content so it can be easily revised to here (and accessible from the site menu). But also available here …

oracle.com/goto/ …Domain AddressDescription
oracle.comThis is the commercial side of the Oracle content. But includes a landing page for a lot of Oracle knowledge/documentation.
docs.oracle.comThe home of the core Oracle documentation for all products.
/blogblogs.oracle.comThe members of this team are the ‘gurus’ of product applications. These cover a range of domains – structured in a similar way to blogs.oracle.com with different posts. These posts represent patterns and solutions to problems encountered by the team. How to, or not to implement things.
This can overlap with some blogs in so far as both product blogs and A-Team blogs may address how to leverage product features.
A-TeamDeveloper relations lead, which covers not only Oracle products but also the application of open source. By its very name, there is a strong emphasis on coding (rather than low-code) covering not just Java but .Net languages such as C#, Node, JavaScript, and so on.
There is some content overlap here with the Architectural Center, where Architecture Centre provides reference solutions.
/devdeveloper.oracle.comDeveloper relations lead, which covers not only Oracle products but also the application of open source. By its very name, there is a strong emphasis on coding (rather than low-code) covering not just Java, but .Net languages such as C#, Node, JavaScript, and so on.
There is some content overlap here with the Architectural Center, where Architecture Centre provides reference solutions.
/ref-archs
/playbooks
/deployed		docs.oracle.com/solutions
(AKA Architecture Center)		This is the Architecture Center which provides reference solutions. But these aren’t exclusive to the SaaS products (which would be easy to interpret). A lot of examples cover deploying and running open-source solutions on IaaS, for example, Drupal, WordPress, and Magento to name just a couple. A lot of these are backed up with scripts, Terraform, and code to achieve the deployment and configuration.
In addition to this, there are use cases of what customers have deployed into production (known as built and deployed).
docs.oracle.com/learn/This contains a lot of free tutorials and labs that can be taken a run to implement different things, from deploying a Python with Flask solution on Kubernetes to Creating USB Installation Media for Oracle Linux with Fedora Media Writer. As you can see from these examples, the tutorials cover both Oracle products and open source.
These resources interlink with the Architecture Centre and can overlap with developer.oracle.com.
/gh-devrelgithub.com/oracle-devrelThis contains the code artifacts developed by Developer Relations and the Architecture Center team. So covers Reference Architectures, tutorials, and Live Labs all freely available to use.
/gh-samplesgithub.com/oracle-samplesThis repository contains example solutions and Terraform logic developed by groups such as the A-Team.
/openopensource.oracle.comThis provides a catalog of links to the various open-source repositories available. This includes oracle-sample and devrel but also the many other projects including, but not limited to Helidon, Fn, Verrrazano, GraalVM, Apiary
/communitycommunity.oracle.com/hubThis is where a lot of the Q&A for Oracle Partners and users takes place. Community participation is gamified to encourage participation.
academy.oracle.comThis is really for the educational community (Universities, Colleges & Schools) and provides resources to take you from zero to certified skills for Java and Oracle Database.
education.oracle.comPrimarily for the formal Oracle certification resources, but there are some free training resources here as well.
/apexapex.oracle.comThis provides a catalog of links to the various open-source repositories available. This includes oracle-sample and devrel but also many other projects including, but not limited to Helidon, Fn, Verrrazano, GraalVM, Apiary
/gh-oraclegithub.com/oracleThis is the home for the majority of all Oracle free code, including the Open source projects such as Helidon, JET, Kubernetes Operators, and so on.
Helidon, Fn, and other source projectsAPEX has its own area of dedicated resources. This includes solutions, templates, other development resources, and community.
/youtubewww.youtube.com/oracleYouTube training videos. With multiple channels based on different technologies.
Java (OpenJDK and Oracle JDK)A number of Oracle open-source projects have their own independent web resources as well. Helidon includes additional technical resources.
The ones we know more about are :
Helidon, Fn, Verrrazano, GraalVM, Apiary
But it includes references to Java core language etc.
/paasPaaS CommunityManaged by Jurgen Kress (Prod Mgr for Oracle PaaS). It acts as an aggregator for contributions from the community and shares news about what is happening within Oracle to support customers and partners in the PaaS space.
Developer CommunityPartner site to the PaaS Community with a focus on more development activities around Weblogic, Helidon, etc.
Tutorials etc., relating to the use of Linux and related Cloud Native things such as Verrazano, OCNE (Kubernetes), etc.LunaOracle propel not just their main JDK /JRE, which has license constraints, but also Open JDK. With this comes a dedicated Developer zone as dev.java.
Oracle PodcastsPerhaps not access usable as documentation, how-to, etc. but Podcasts can yield a lot of broad picture insights. Oracle has a range of podcasts covering a diverse range of subjects. Not all podcasts are active at any one time. But the site provides a catalog and episode list.

blogs.oracle.com – last time for Capgemini

02 Saturday Apr 2022

Posted by in General

≈ Leave a comment

A final blog that I co-wrote with Andy Bell whilst I was at Capgemini has gone live on blogs.oracle.com (here). Future posts will be as an Oracle employee.

This blog post follows up on a webinar that Capgemini did with Oracle and Flexagon reflecting on the application of FlexDeploy within Capgemini’s AIP offering.

Istio In Action

25 Friday Mar 2022

Posted by in Books, General

≈ Leave a comment

Tags

, , , , , ,

Christian Posta and Rinor Maloku’s book with Manning, Istio In Action has just been published. I’ve previously said it’s a good book, and that’s not surprising given Christian’s role at solo.io. When the final chapters became available I started to go through it in more detail and built a mind map (As with the recent review of Kubernetes best practices). The map can be seen below.

As you can see the map is very substantial reflecting on the depth and value of the book. For those who look at the maps, may notice there are a couple of chapters not fully mapped. I will update the map to fill those gaps in, but given they focus on monitoring and observability, I was less concerned about those areas given my own writing. The book’s exercises are very much built around using Docker Desktop making it very easy to spin up the examples and exercises. If you want to know about Istio Service Mesh on K8s then I’d recommend it.

Reading through the book, I’ve learned details that I was not entirely aware of, for example the integration of non K8s workloads into the mesh. The tuning of Istio to keep it highly performant with a lot of workloads.

The book can be obtained from:

And other retailers of course.

New Challenge – New Job

21 Monday Mar 2022

Posted by in General

≈ Leave a comment

Tags

, , , , ,

Those I interact with more regularly will know I have this last week started a new job. You could say I’ve followed a common path for Ace Directors that don’t have instrumental roles within a company and joined Oracle. But some things won’t change; my new role as Cloud Developer Evangelist means I will still be producing blog content. My writing will appear on Oracle’s websites through the Blog, Community, Developer, and Architecture parts of the site, plus potential contributing to other high profile sites. But We’ll continue to add posts here, including referencing contributions in other locations.

I will continue submitting papers/presentations to conferences and presenting. Hopefully, we’ll start to get out and see people whilst presenting as well.

This does mean a change to my Ace Director status, as I will become an Alumni of the community, and we’ll be updating logos etc. But I have developed many friendships and contacts within the ace community. Not to mention, I believe and value what the Ace Community does, so while I may not be officially part of the community anymore, I will endeavour to support my friends and the wider community.

Leaving Capgemini has been uncomfortable, as I have left behind many great people that I’ve enjoyed working with (although some of those have also preceded me into Oracle). But to do more of what I have learnt over the years as the most rewarding (talking and writing about applying technology to solve problems, sharing insights and knowledge), particularly as part of the Ace community, meant a need for a new challenge.

To those who have contributed and influenced my journey – as ever my thanks.

gRPC, GraphQL and more …

01 Tuesday Mar 2022

Posted by in APIs & microservices, General, Technology

≈ Leave a comment

Tags

, , ,

I got into a discussion with several people about the use of GraphQL and related API technologies and discovered that a presentation I’ve been using and evolving for a while now, didn’t appear in my blog. So here is a version of it used at an API Conference …

The presentation may appear again in the future as the perspective of API technologies evolves the presentation will need to evolve. For example, AsyncAPI is starting to make an impression now. Other variants to API technologies such as DRPC are showing up.

If you’re new to GraphQL you might find a couple of other posts on the subject helpful:

Logging in Action – has gone to the printers!

07 Monday Feb 2022

Posted by in Books, Fluentd, General, Technology

≈ 1 Comment

Tags

, , , , , , ,

The final leg of getting the book published has taken a lot longer than had been expected. But we’ve just been told that the book has been sent to the printers. This means:

  • final eBook will be available from Manning in about 1 week
  • Preordered print copies of the book will be dispatched in about 2 weeks
  • The alternative ebook formats for mobile readers e.g. kindle etc available in about 3 weeks
  • The book will become available to purchase from other book stores such as
    Amazon in 3-4 weeks
  • Safari Books Online and Apple stores will have the ebook in 4-5 weeks

Here are some links for buying the book …

It also means the project from a writing perspective is complete. But we’re starting to look at the additional examples we’ll add to the GitHub repo. These will be dependent upon the book.

The complete cover artworks …

We’ve got print books now …

Kubernetes Best Practises – Review & Mindmaps

13 Thursday Jan 2022

Posted by in APIs & microservices, Books, Cloud, Cloud Native, development, General, Technology

≈ 2 Comments

Tags

, , , , ,

I’ve had some time to catch up on books I’d like to read, including Kubernetes Best Practises in the last few weeks. While I think I have a fair handle on Kubernetes, the development of my understanding has been a bit ad-hoc as I’ve dug into different areas as I’ve needed to know more. This meant reading a Dummies/Introduction to entry style guide would, to an extent, likely prove to be a frustrating read. Given this, I went for the best practises book because if I don’t understand the practises, then there are gaps in my understanding still, and I can look at more foundation resources.

As it goes, this book was perfect. It quickly covered the basics of the different aspects of Kubernetes helping to give context to the more advanced aspects, and the best practices become almost a formulated summary in each section. The depth of coverage and detail is certainly very comprehensive, explaining the background of CNI (Container Network Interface) to network-level security within Kubernetes.

The book touched upon Service Meshes such as Istio and Linkerd2 but didn’t go into great depth, but again this is probably down to the fact that Service Mesh ideas are still maturing, and you have initiatives like SMI (Service Mesh Interface still in the CNCF’s sandbox).

In terms of best practices, that really stood out for me:

  • Use of Taints and Tolerations for refined control of pod deployment (Allowing affinity to be controlled to optimise resilience, or direct types of pod deployment to nodes with specialist capabilities such as GPU).
  • There are a lot more differences and options then you might realize in terms of ingress controller capabilities, so take time to identify what you may need from an ingress controller.
  • Don’t forget pods can be scaled vertically with the VPA (Vertical Pod Autoscaler)as well as horizontally through the HPA.
  • While using a managed persistence service will make statement storage a lot easier, stateful sets will give you a very portable solution.

As with a lot of technical books I read. As I go through the book I build up a mind map of what I think are the key points. Doing so leaves me with a resource I can use as a quick reference, but creating the mind map helps reinforce the learning. So here is the mind map …

Greening our work compute needs

13 Monday Dec 2021

Posted by in Cloud, General, Technology

≈ Leave a comment

Tags

, , , ,

As a consultant working with clients, we always need to address security considerations for clients, their networks and data. Typically this might mean ensuring I could connect to the correct network through a VPN with the secure client software installed. Then work through a Citrix set-up for the tools we’re allowed to use.

Since the start of the pandemic, there seems to be a marked shift towards issuing consultants with customer provided laptops that have been configured and locked down. This means I can’t use the client laptop to connect to my employer’s network to interact with our own systems – making it easy to leverage our existing resources to support the customer and conversely no trust or contractual position that might allow our company devices connecting to a VPN or ring-fenced part of a network.

Interestingly there seems to have been a drift away from the ideas of BYOD (Bring Your Own Device) which may come from the fact that outside of smaller very tech-savvy organizations, BYOD can be seen as challenging to support.

As this Google Trends report shows over the last five years the trend has been until the last couple of months showing a generally downward trend. Not authoritative proof, but hints that it hasn’t accelerated as you might expect given remote working.

By the customer supplying a laptop, there is an effort to control intrusion and other security risks. But the problem is, now I have a device that I could easily take off-line and work to defeat the security setup and the client would be non the wiser, or worse it is another laptop that could ‘get lost’ or ‘stolen’ with a greater chance of having sensitive material. Every new device is without a doubt an elevated risk for the client and a cost to support (this of course is also an argument for not applying BYOD).

Continue reading →