• Home
  • Site Aliases
    • www.cloud-native.info
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Log Generator
    • Logs and Telemetry using Fluent Bit
      • Fluent Bit book
      • Book Resources in GitHub
      • Fluent Bit Classic to YAML Format configurations
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Fluentd & Fluent Bit Additional stuff
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
      • Recommended Tech Podcasts
      • Official Sources for Product Logos
      • Java and Graal Useful Links
      • Python Setup & related stuff
      • DevTips
  • Music
    • Monster On Music
    • Music Listening
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Technology

Fluent Bit and AI: Unlocking Machine Learning Potential

30 Monday Dec 2024

Posted by mp3monster in Fluentbit, General, Technology

≈ Leave a comment

Tags

AI, artificial-intelligence, Cloud, Data Drift, development, Fluent Bit, GenAI, Machine Learning, ML, observability, Security, Technology, Tensor Lite, TensorFlow

These days, everywhere you look, there are references to Generative AI, to the point that what have Fluent Bit and GenAI got to do with each other? GenAI has the potential to help with observability, but it also needs observation to measure its performance, whether it is being abused, etc. You may recall a few years back that Microsoft was trailing new AI features for Bing, and after only having it in use for a couple of days, it had been recorded generating abusive comments and so on (Microsoft’s Tay is such an example).

But this isn’t the aspect of GenAI (or the foundations of AI with Machine Learning (ML)) I was thinking about. Fluent Bit can be linked to GenAI through its TensorFlow plugin. Is this genuinely of value or just a bit of ‘me too’?

There are plenty of backend use cases once the telemetry has been incorporated into an analytics platform, for example:

  • Making it easy to query and mine the observability data, such as natural language searching – to simplify expressing what is being looked for.
  • Outlier / Anomaly detection – when signals, particularly metrics, diverge from the normal patterns of behavior, we have the first signs of a problem. This is more Machine Learning than generative AI.
  • Using AI agents to tune monitoring thresholds and alerting scenarios

But these are all backend, big data style use cases and do not center on Fluent Bit’s core value of getting data sources to appropriate destination systems for such analysis or visualization.

To incorporate AI into Fluent Bit pipelines, we need to overcome a key issue – AI tends to be computationally heavy – making it potentially too slow for streams of signals being generated by our applications and too expensive given that most logs reflecting ‘business as usual’ are, in effect, low value.

There are some genuine use cases where lightweight AI can deliver value. First, we should be a little more precise. The TensorFlow plugin is the TensorFlow Lite version, also known as LiteRT. The name comes from the fact that it is a lite-weight solution intended to be deployable using small devices (by AI standards). This fits the Fluent Bit model of having a small footprint.

So, where can we put such a use case:

  • Translating stack traces into actionable information can be challenging. A trained ML or AI model can help classify and characterize the cause of a stack trace. As a result, we can move from the log to triggering appropriate actions.
  • Targeted use cases where we’ve filtered out most signal data to help analyze specific events – for example, we want to prevent the propagation of PII data downstream. Some PII data can be easily isolated through patterns using REGEX. For example, credit card IDs are a pattern of 4 digits in 4 groups. Phone numbers and email addresses can also be easily identified. However, postal addresses aren’t easy, particularly when handling multinational addresses, where the postal code/zip code can’t be used as an indicative pattern. Using AI to help with such checks means we must filter out signals to only examine messages that could accidentally carry such information.

When adopting AI into such scenarios, we have to be aware of the problems that can impact the use of ML and AI. These use cases are less high profile than the issues of hallucinations but just as important. As we’re observing software, which will change over time. As a result, payloads or data shifts (technically referred to as data drift) and the detection rate can drop. So, we need to measure the efficacy of the model. However, issues such as data drift need to be taken into account, as the scenario being detected may change in volume, reflecting changes in software usage and/or changes in how the solution works.

There are ways to help address such considerations, such as tracking false positive outcomes, and if the model can provide confidence scoring, is there a trend in the score?

Conclusion

There are good use cases for using Machine Learning (and, to an extent, Artificial Intelligence) within an observability pipeline – but we have to be selective in its application as:

  • The cost of the computation can outweigh the benefits
  • The execution time for such computation can be notably slower than our pipeline, leading to risks of back pressure if applied to every event in the pipeline.
  • The effectiveness and how much data drift might occur (we might initially see very good results, but then things can fall off).

Possibly, the most useful application is when the AI/ML engine has been trained to recognize patterns of events that preceded a serious operational issue (strictly, this is the use of ML).

Forward-looking

The true potential for Gen AI is when we move beyond isolating potential faults based on pattern recognition to using AI to help recommend or even trigger remediation processes.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Speaker Upgrade – how I decided what was good

08 Tuesday Oct 2024

Posted by mp3monster in Music

≈ Leave a comment

Tags

Acoustic Energy, AE500, audio, Beth Orton, Bowers & Wilkins, Chord, Cosmotron, Elbow, GoGo Penguin, Music, new-music, Peter Gabriel, reviews, Rush, Technology, Tori Amos

With some recent good news from work, I decided to treat myself to a speaker upgrade – Acoustic Energy 500s sat on some IsoAcoustic Aperta stands. While these would be considered audiophile – they’re still at the lower end – we’re not talking audio exotica like B& Nautilus at nearly hundred thousand pounds or the Cosmotron 130 at around the million pound mark.

Bowers & Wilkins – Nautilus Speaker – a snip at £90,000

So how can I decide and justify the expenditure, even if it’s a fraction of the loose change from the back of the sofa from buying these monsters? As friends have said to me in the past, the Samsung speakers on my stereo are just as good. Well there are a raft of things that will prevent speakers from performing well, from positioning, to the quality of their source.

Million Pound Cosmotron speaker
Cosmotrom priced at £1M

The source material is often one of the biggest issues, particularly for rock and pop pushing the envelope with CDs. We saw what has become known as the loudness wars – where the dynamic range of the music was reduced. But music with a wide dynamic range with good speakers is great. A couple characteristics of good speakers is the containment of distortion – so if you have a song that is often quiet with occasional moments of loudness, the speaker drivers (cones) will be able to react properly to another sudden spike in signal occurs the sudden movement in the magnet moving the cone is handled rather than causing the speaker surface straining against its mounts.

Better speakers will result in better control of the cone (the visible bit of the speaker), making the cone’s movements more precisely revealing detail in the music. You’ll go from hearing a cymbal, to being able to tell how the cymbal was struck, a drum is no long a thump, but you’ll start to hear it resonate.

The cone moves backward and forwards to move the air, which affects air inside the speaker, not just outside. We don’t want the speaker casing to behave as a suction cup, preventing air movement and inhibiting the cone’s movement.

Improvements in speaker performance can help you recognize little details. For example, with a vocal performance, you’ll start to hear fine details, such as air drawn over the microphone as the singer inhales. You can also hear changes as a singer moves close to or away from the microphone, even if they alter their vocal volume.

I was experimenting with a loaned hi-fi kit once, listening to a Jamie Cullum live performance, and a detail that leapt out as I swapped in and out a piece of equipment was what sounded like background ambient noise, such as air conditioning. But suddenly, it became clear I wasn’t picking up ambient noise but the fan that was positioned behind Jamie.

It is always useful to have some good go-to pieces of music for trying out hi-fi. Being familiar with the music and knowing the production values applied means that if there are improvements, you’ll pick them up. So, what are my go-to pieces at the moment?

  • Tori Amos – Me and a Gun — although any part of Little Earthquakes is good. This song is an acapella performance, recounting a rape. With just a voice, the miking of the vocal is very close, and you can hear the inhalation and the rawness of the performance.
  • Beth Orton – Weather Alive — probably Beth’s best album to date. Here is another incredible voice, but also more delicate than Tori Amos, so the better the HiFi, the purer the performance will sound.
  • GoGo Penguin – Branches Break from Man Made Objects – although just about any of their work will be good. This is a trio of piano, bass, and drums in a jazz/minimalist classical/chill beat crossover. This is a recording that should feel like it’s being performed in a big live sounding room. But you’ll hear each instrument clearly, particularly down to recognizing the loudness, varying attack, and decay of each note played.
  • Rush – Red Sector A from Grace Under Pressure, perhaps not the best-produced album in the world, but before the loudness wars really took hold. Rush were a real bunch of prog rock musos with the late Neil Peart, who many considered to be one of the best ever drummers. This track will test the HiFi in terms of control – the drumming has a huge range of very fine cymbal work, some really deep bass drums, and tom-tom runs that make Phil Collin’s In The Air Tonight sound like child’s play.
  • Elbow – One Day Like This – The Seldom Seen Kid (Live At Abbey Road Studios) — with a high-quality recording (Abbey Road’s special Half Speed Mastered edition), you’ll get a sense of staging and as the song grows scale with the choir. The strings will be natural and nuanced, in the early parts of the performance of the performance you’ll hear how dry Guy’s voice is – not a hint of vibrato or sibilance.
  • Peter Gabriel – the Book Of Love — from Scratch My Back — another performance that should give a sense of staging and breadth with great dynamics and the strings swell and subside. Fronted by Peter’s voice which should weathered and world warn.

The list of music could go on. But, ultimately, it’s a very individual choice.

Final anecdote

Buying Hi-Fi is a law of diminishing returns. As you get better and better, the parts needed are more expensive and produced in fewer numbers, making the R&D more expensive, with costs to be covered by a small number of sales. But still, these esoteric, bank-crushing systems are amazing.

Some years back, I went to a HiFi show; if you’ve never been to such a show then picture this. A corridor of rooms is stripped of the beds and furnishings other than some chairs. Each company has a room and typically sets up its demo kit where the head of the bed would usually be. Everything would be positioned and mounted on professional hi-fi tables, etc, for the absolute best performance. The classic layout for a hotel room means as you walk into the room, you won’t see what is set, so the seconds it takes to walk past what is normally the bathroom is almost a blind test as you can’t see the HiFi, but you’ll be able to hear it.

So here we are, as we start to walk into a room that was pretty busy, so you didn’t see the main space for a minute or so, and we hear a performance of a beautifully played unaccompanied double bass. I could have sworn there was a musician in the room performing – the performance had that warmth, depth, and volume you’d expect. No hint of any recording artifacts. When we got to the main part of the room, we were stunned to see two speakers, big and rather boxy – no audio exotica beauty like Nautilus or Cosmotron — definitely all function, and little thought to form. With them, 3 large pieces of silver HiFisat on big chunky slabs of marble on the floor – what I assume to be a pre-amp and a power amp for each speaker. Plus a source – which might have been a turntable – but honestly, I can’t remember – whatever it was, the sound was breathtakingly natural sounding.

Chord Ultima Monoblock Power Amplifier £35,000 per unit
Chord Monobloc Power Amplifier £350,000 per bloc – you’d need two, plus a pre-amp for a basic arrangement.

I do remember the price tags, and at the time, prices were around 50k a component- so little change out of a quarter of a million. It left me wishing I’d won the national lottery.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Cloud Native Architecture book

31 Friday May 2024

Posted by mp3monster in Books, General

≈ Leave a comment

Tags

BPB, Cloud Native, CNCF, Fernando Harris, Kubernetes, organization, people, process, Technology

It’s a busy time with books at the moment. I am excited, and pleased to hear that Fernando Harris‘ first book project has been published. It can be found on amazon.com and amazon.co.uk among sites.

Having been fortunate enough to be a reviewer of the book, I can say that what makes this book different from others that examine cloud-native architecture is its holistic approach to the challenge. Successful adoption of cloud-native approaches isn’t just technical (although this is an important element that the book addresses); it also considers organizational, processes, and people dimensions. Without these dimensions, the best technology in the world will only be successful as a result of chance rather than by intent.

As a result, this book guides and connects content to the Kubernetes technical content (technical how-to books we typically see from publishers like Manning and O’Reilly) and the more organizational leadership books that you might expect from IT Revolution (Gene Kim et al.).

A read I’d recommend to any architect or technical lead who wants to understand the different aspects of achieving cloud-native adoption rather than just the mastery of an individual technology.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Simplifying the escaping of JSON strings

08 Thursday Jun 2023

Posted by mp3monster in development, General, Technology

≈ Leave a comment

Tags

CLI, development, JQ, JSON, software, Technology, utility

when you’re testing apps, it is pretty common to want to send JSON via CURL to a local endpoint. The problem is that this usually means that the string you provide curl needs to have characters escaped, such as quote marks. By hand, this can be irritating to sort out, particularly if you’re using an IDE to make sure the JSON is correct. I’d concluded this is hardly a new problem; someone must have produced a nice little multiple-platform command line utility that can do it for you. The result was a bit more surprising.

There are plenty of online utils that solve it, but if you’re working with data, you don’t want to publicly share (or the fiddling around with copy-pasting to your browser). Nothing wrong with these tools, but you can’t script them without resorting to RPA (Robotic Process Automation) either. Here are a couple of services I found that are straightforward, and when I’ve tried them, not plagued by annoying ads.

  • https://wtools.io/json-escape-unescape
  • https://www.jsonescaper.com/
  • https://toolslick.com/text/escaper/json
  • https://codebeautify.org/json-escape-unescape
  • https://appdevtools.com/json-escape-unescape

But finding command line tools, well, finding an answer, has proven a bit more challenging. For removing escaped characters, you could use jq, but we actually want to go the other way to use curl with JSON that has been escaped. I have come across conversations covering the use of bash (making use of awk and sed. Plus, details about how the manipulation could be done in various languages (so you could code your own solution if so inclined. Coding is unlikely to take much effort, but testing permutations is going to demand effort).

The one solution I have found that meant I could escape (or reverse) JSON locally is a plugin for VS Code called appropriately JSON-escaper, which does what is needed in a nice and clean manner. All credit to Joshua Poehls for the tool.

The solution JSON-escaper built on top of a more generic JavaScript utility which addresses escaping special characters which can be found here.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Node (npm) package licensing

05 Tuesday Jul 2022

Posted by mp3monster in development, General, node.js, Technology

≈ Leave a comment

Tags

code, developer, development, Licensing, node.js, package, Technology

When building Node solutions, even if you’re not going to publish the code to a public repository you’re likely to be using package.json to declare the dependencies for your app. Doing this makes it easier to build and deploy a utility. But if you’re conversant with several languages there is a tendency to just adapt your existing skills to work with others. The downside of this is small tooling nuances can catch you off guard and consume time while figuring them out. The workings of packages with NPM (as shown below) is one possible case.

{
  "name": "graph-svr",
  "version": "1.0.0",
  "description": "packages needed for this service",
  "main": "index.js",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@graphql-tools/graphql-file-loader": "^7.3.11",
    "@graphql-tools/load-files": "^6.5.4",
    "@graphql-tools/schema": "^8.3.10",
    "@graphql-yoga/node": "^2.4.1",
    "apollo-datasource-rest": "^3.5.2",
    "apollo-server": "^3.6.7",
    "graphql": "^16.4.0",
    "graphql-tools": "^8.2.8"
  },
  "author": "Phil Wilkins",
  "license": "MIT"
}

If you create the package.json using npm init to create the initial version of the file, it is fairly common to set values to default. In the case of the license, this is an ISC license. This is easily forgotten. The problem here is twofold:

  • Does the license set reflect the constraints of the dependencies and their licenses
  • Does the default license reflect the position you want?

Looking at the latter point first, This is important as organizations have matured (and tooling greatly improved) when it comes to understanding how open source licensing can impact. This is particularly important for any organizations leveraging open source as part of their revenue generating activities either ‘as a service’ but also selling software solutions. If you put the wrong license here the license checking tools often protecting code repositories may reject your code, even in internal only use cases (yes this tripped me up).

To help overcome this issue you can install a tool that will analyze the dependencies and optionally their dependencies and report back on your license exposure. This tool is called license-report. Once installed (npm install -g license-report) we just need to point the tool to the package.json file. e.g. license-report package.json. We can make the results a lot more consumable by outputting the content in a number of formats. For example a simple text value:

From this, you could set your license declaration in package.json or validate that your preferred license won’t conflict,

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Apollo GraphQL – some pointers

16 Thursday Jun 2022

Posted by mp3monster in development, General, languages, node.js, Technology

≈ 2 Comments

Tags

API, code, development, GraphQL, javascript, node.js, Technology

I’ve designed a variety of GraphQL schemas and developed microservice backends. But not done much with configuring the Apollo implementation of a GraphQL server until recently. This may reflect the fact my understanding of JavaScript doesn’t extend into the world of Node.JS as much as I’d like (the problem with being a multi-language developer is you’re likely to find your way around many languages but never be a master of one). Anyway, the following content is about the implementation within a GraphQL server part of a solution. It may be these pointers are just for my benefit you might find them helpful as well.

Read more: Apollo GraphQL – some pointers

To make it easy to reference the code, we’ve added entries (n) into the code, where n is a number. This is not part of the code. But there to make the different lines referenceable. Where code should go but is not relevant to the point being made I’ve added ellipsis (…)

Dynamic loading and server configuration

import { ApolloServer } from 'apollo-server';
import { loadFilesSync } from '@graphql-tools/load-files';
import { resolvers } from './resolvers.js';   (1)
import ProviderInternalAPI from './ProviderInternalAPI.js'; (1)
import EventsInternalAPI from './EventsInternalAPI.js';  (1)
const server = new ApolloServer({
  debug : true,    (2)
  typeDefs: loadFilesSync('./schema.graphql'),   (3)
  resolvers,
  dataSources: () => {
    return {
      eventsInternalAPI: new EventsInternalAPI(),    (4)
      providerInternalAPI: new ProviderInternalAPI() (4)
      pro
    };
  }});

There is the potential to dynamically load the resolvers rather than importing each JavaScript file as we see on lines (1). The mechanics to do this is documented here. It would be cool if an opinionated implementation was provided. As shown by (3) we can take a independent schema file being loaded. The Apollo example approach for this didn’t seem to work for us, although both approaches make use of graphql-tools in a synchronous manner.

We can switch on debugging (2) for the GraphQL server, although the level of information published doesn’t appear to be significant. Ideally this setting is changed for production.

Defining the resolvers

The prefix for each resolver (1) must correlate to the name in the schema of the mutator or query (not the type as you would expect with Java). Often we don’t need all the parameters for the resolver. The documentation describes replacing each unused parameter with one or more underscores (i.e _, __ ). The underscore denoting the field not in use. However we can satisfy the indication of not being used, but keep the meaning of each position by using the underscore then a name (i.e. _parent, _args ) as shown in (2).

By taking the response into a variable (3) we can optionally log it. Trying to return using invocation line would result in the handler object rather than the payload itself. By taking the result into a variable we can log the content if desired and return the content.

The use of the backward quote is a node feature. It allows us to incorporate variables into a string by referencing it within ${} (4).

We need to supply the GraphQL server with instances with a layer of code that will interact with the resolvers. We can instantiate the instances in the declaration. The naming of the object is important (4) to the resolver.js (declarations).

import { useLogger } from "@graphql-yoga/node";
...
latestEvent (1): async (_parent, _args, { dataSources }, _info) (2)   => {
      if (log) { console.log("resolvers - get latest event"); }
      let responseValue = await dataSources.eventsInternalAPI.getLatestEvent(); (3)
      if (log) { console.log(`(4)  Resolver response for latest event:\n ${responseValue}`); }
      return responseValue;
    },

Resolver declarations

 Query: {  ...
 },
  
Mutation: {...
},
  Event: {  (1)
    providers: (event, args, { dataSources }, info) => {
      if (log) { console.log(`going to locate ${event.sources}`) }
      let responseValue = await (2) dataSources.providerInternalAPI.getProviders(event.sources);
      return responseValue;
    }

To handle the use of resolvers within a larger resolver we need to declare the resolution outside of the Query and Mutator blocks (but inside the whole declaration block)(1). The name provided needs to match the parent entity that the query resolver contributes to.

To then provide values from the outer resolution we need to prover to the chained resolution use the naming as represented in the GraphQL schema as shown by (2). The GraphQL engine will resolve the mapping values.

Web resolver URL

  // GET
  async getProvider(code) {
    console.log("getProvider (%s) directing to %s",code,this.baseURL);
    return this.get(`provider?code=${code} (1)`);
  }

The URL parameters need to be appended to the base URL path for the parent class to use in the invocation as shown by (1). The Apollo examples showed a setter option but we didn’t see the URI being addressed properly. This approach produces the relevant requirement.

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Container Registry – pushing and storing containers

12 Thursday May 2022

Posted by mp3monster in Cloud Native, General, Oracle, Oracle Cloud Native, Technology

≈ 2 Comments

Tags

container, development, Docker, Kubernetes, OCIR, OKE, Oracle, registry, Technology

A container registry is as essential as a Kubernetes service as you want to manage the deployable resources. That registry could be the public Docker repository or something else. In most people’s cases, the registry needs to be private as you don’t want to expose your product assets to potential external tampering. As a result, we need a service such as Oracle’s container registry OCIR.

The re of this blog is going to walk through how to push a container you’ve built into OCIR and a gotcha that can trip up users if you make assumptions about how the registry works.

Build container

Let’s assume you’re building your microservices locally or retrieving vetting 3rd party services for use. In both cases, you want to manually push your assets into OCIR manually rather than have an automated build pipeline do it for you.

To make it easier to see what is happening, we can exploit some code from Oracle’s Github repo (such as this piece being developed) or you could use the classic hello world container (https://github.com/whotutorials/docker-busybox-hello-world/blob/master/Dockerfile). For the rest of the post, we’ll assume it is the code developed for the Oracle Architecture Center-provided code.

docker build -t event-data-svc .

This creates a container locally, and we can see the container listed using the command:

docker images

Setup of OCIR

We need an OCIR to target so the easiest thing is to manually create an OCIR instance in one of the regions, for the sake of this illustration we’ll use Ashburn (short code is IAD). To help with the visibility we can put the registry in a separate compartment as a child of the root. Let’s assume we’re going to call the registry GraphQL. So before creating your OCIR set up the compartment as necessary.

fragment of the compartment hierarchy

In the screenshot, you can see I’ve created a registry, which is very quick and easy in the UI (in the menu it’s in the Developer Services section).

The Oracle meu to navigate to the OCIR service
the UI to create a OCIR

Finally, we click on the button to create the specific OCIR.

Deployment…

Having created the image, and with a repo ready we can start the steps of pushing the container to OCIR.

The next step is to tag the created image. This has to be done carefully as the tag needs to reflect where the image is going using the formula <region name>/<tenancy name/<registry name>:<version>. All the registries will be addressed by <region short code>.ocir.io In our case, it would be iad.ocir.io.

docker tag graph-svr:latest iad.ocir.io/ociobenablement/graphql-svr:v0.1-dev

As you may have realized the tag being applied effectively tells OCI which instance of OCIR to place the container in. Getting this wrong can be the core of the gotcha previously mentioned and we’ll elaborate upon it shortly.

To sign in you’ll need an auth token as that is passed as the password. For simplicity, I’ve passed the token in the docker command, which Docker will warn you of as being insecure, and suggest it is passed in as part of a prompt. Note my token will have been changed by the time this is published. The username is built on the structure of <cloud tenancy name>/identitycloudservice/<username>. The identitycloudservice piece only needs to be included for your authentication is managed through IDCS, as is the case here. The final bit is the URI for the appropriate regional OCIR address, as we’ve used previously.

docker login -u ociobenablement/identitycloudservice/philip.wilkins@oracle.com -p XXXXXXXXXXX  iad.ocir.io

With hopefully a successful authentication response we can push the container. It is worth noting that the Docker authenticated connection will timeout which is why we’ve put everything in place before connecting. The push command is very simple, it is the tag name assigned to the artifact including the version number.

docker push iad.ocir.io/ociobenablement/graphql/graph-svr:v0.1-dev
OCIR with several versions of a container

Avoiding the gotcha

When we deal with repositories from Git to SVN or Apache Archiva to Nexus we work with a repository that holds multiple different assets with multiple versions of those assets. as a result, when we identify an asset uniquely we would expect to name things based on server/location, repository, asset name, and version. However, here each repository is designed for one type of asset but multiple versions. In reality, a Docker repository works in the same manner (but the extended path impact is different).

This means it becomes easy to accidentally define a tag with an extra element. Depending upon your OCI tenancy privileges if you get the path wrong, OCI creates a new root compartment container repository with a name that is a composite of the name elements after the tenancy and puts your artifact in that repository, not the one you expected.

We can address this in several ways, first and probably the best option is to automate the process of loading assets into OCIR, once the process is correct, it will remain correct. Another is to adopt a principle of never holding repositories at the root of a tenancy, which means you can then explicitly remove the permissions to create repositories in that compartment (you’ll need to explicitly grant the permissions elsewhere in the compartment hierarchy because of policy inheritance. This will result in the process of pushing a container to fail because of privileges if the tag is wrong.

Visual representation of structure differences

Repository Structure
Registry Structure

Condensed to a simple script

These steps can be condensed to a simple platform neutral script as follows:

docker build -t event-data-svc .
docker tag event-data-svc:latest iad.ocir.io/ociobenablement/event-data-svc:latest

docker login -u ociobenablement/identitycloudservice/philip.wilkins@oracle.com -p XXXXX  iad.ocir.io
docker push iad.ocir.io/ociobenablement/event-data-svc:latest

This script would need modifying for each container being built, but you could easily make it parameterized or configuration drive.

A Note on Registry Standards

Oracle’s Container Registry has adopted the Open Registries standard for OCIR. Open Registries come under the Linux Foundation‘s governance. This standard has been adopted by all the major hyperscalers (Google, AWS, Azure, etc). All the technical spec information for the standard is published through GitHub rather than the main website.

References

  • Push an Image to Oracle Cloud Infrastructure Registry
  • Notes about Repository Creation
  • Creating a Container Registry
  • Open Registries
  • Policies to Control Repository Access

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

The Air Gap a Security Fallacy?

12 Saturday Jun 2021

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

Security, software, Technology

Securing systems through an air gap is an idea goes back decades, and through the 50s to as recently as the 2000s the idea that you could safely and successfully run and maintain systems by simply not connecting to a network protects them from vulnerabilities may have been true. But in the last ten or more years, I would argue it is a fallacy, that can lull people into a false sense of security and therefore more likely to take more risks (or at least not be as careful as we should be). This idea is a well established piece of psychology with modern cars (here for example). This is known as Risk Compensation (also more here) or an aspect of behaviour adaptation.

Whilst this is rather theoretical, perhaps we should illustrate in practical terms why it is both a fallacy, and in the modern day simply impractical.

Continue reading →

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

EMEA PaaS Forum 2019 in review

15 Monday Apr 2019

Posted by mp3monster in General

≈ 1 Comment

Tags

Ace, Capgemini, Cloud, conference, development, drone, Luis Weir, meetup, Oracle, PaaS, Presentation, Technology

Image by @motivcx

Image by @motivcx

Another Spring means another excellent Oracle EMEA PaaS Forum for Oracle partners. Every Year Juergen Kress organizes the event, finding really nice venues to host several hundred people over four and half days.

The event is split into several parts,  Monday afternoon normally involves Oracle Ace’s presenting on best practices, insights on applying the various technologies etc.  For me this meant presenting on the London Developer Meetup, looking at how it worked, what has been successful, and what hasn’t.  For those know have read my blogs on the subject (here) will know about our Drone initiative.

Meetups – The Oracle Ace Way from Phil Wilkins

Picture by @AmyGrangeX

Then Tuesday is a single stream day where Juergen has managed to pull in SVPs and Senior Product Managers from around the globe to provide a high-level view of what has been going on with their products. For anyone consulting in the Oracle domain, this is incredibly useful. For example, there is a clear strategy coalescing around AI and Machine Learning both as a service proposition to users, but also how these technologies are being made available and used within other products.  Other areas such as OIC and SOA CS have stability and maturity, and the road map is about maximising connectivity with the newer products.

But before the sessions start, Juergen starts with opening remarks, and demos’ something engaging.  In previous years this has been things like Digital Assistants/Chatbots and so on.  This year, we have been fortunate to be an active contributor by demoing the drone through the use of APIs and talking about the ideas.  The dry runs of the demo on Monday went without a problem, but when it came to the main show, the drone was a little uncooperative – we think because the air-con had really kicked in.  But importantly, even not achieving the desired result, the message of engagement made it home.

Wednesday is split into streams with in-depth sessions from the different Product Managers, he amount of insight gained from these sessions is tremendous, some of which is very much protected by safe harbour statements or not for public disclosure such is the honest and open discussions. The day closes with an Ace Director initiative which demonstrates the application of Oracle Cloud products to a plausible use case, and Luis Weir (Capgemini Oracle CTO) is part of. This session has become something of a tradition now.

The day’s business concludes awards, and for a second year the UK Capgemini team have taken home two awards for APIs and PaaS Contribution.

Luis Weir with his API award

The final two days are then a choice of Hackerthon or 1/2 day training sessions on different products with the relevant Product Managers, and an excellent opportunity to pick the brains of the presenters as well as get hands-on experience with the different products.

The week isn’t without it’s social and networking activities of course …

39.506174
2.532125

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...

Building Evolutionary Architectures

16 Saturday Mar 2019

Posted by mp3monster in Book Reviews, Books, General, Technology

≈ Leave a comment

Tags

architecture, book, evolutionary, mindmap, review, Technology

I have been working my way through Building Evolutionary Architectures by Neal Forward, Rebecca Parsons and Patrick Kua. Three senior and respected members of Thoughtworks (also the home of Martin Fowler). Having read and listened to Neal and Rebecca’s presentations and writing I had expected a deeply thought-provoking read, but have to admit to being disappointed. There are some good points without a doubt, but the book pretty much focuses on one idea, the application of fitness functions. But I’m not convinced it warrants several hundred pages of a book as a result the point does at times feel laboured.

There are some arguments made, that leaves me thinking that there is a view that the only answer is microservices in the conventional model of Kubernetes, Docker etc, which I agree is a powerful paradigm to allow solutions to evolve, but it isn’t a silver bullet and not always right in every case (if you have a team lacking the underlying appreciation of the goals, or put in to place in an ad-hoc manner (see Chris Richardson‘s work) it isn’t going to help.

Alongside this, there is little said about the interface definition for microservices (typically APIs of one form or another). Whilst mention of leaky abstractions are made, the material illustrations such as code lead API definitions are omitted (risk being, code changes, the API changes and the impact cascades).

What surprised me the most is the on more than one occasion the books points to ERPs not being sufficiently customisable. Yet, anyone working with ERPs will tell you that ERPs are at their best when you use them to leverage industry best practices rather than crowbar them to fit unconventional ways of operating. If you’re a manufacturer, is fiscal reporting part of your differentiator; probably not, so why not take best practice OOTB.

As usual, I have mind mapped things as I read through the book.  The dynamic/interactive version is here, the image (but not in full detail) is below.

evolutionary architectures.png

 

Share this:

  • Share on Facebook (Opens in new window) Facebook
  • Share on X (Opens in new window) X
  • Share on Reddit (Opens in new window) Reddit
  • Email a link to a friend (Opens in new window) Email
  • Share on WhatsApp (Opens in new window) WhatsApp
  • Print (Opens in new window) Print
  • Share on Tumblr (Opens in new window) Tumblr
  • Share on Mastodon (Opens in new window) Mastodon
  • Share on Pinterest (Opens in new window) Pinterest
  • More
  • Share on Bluesky (Opens in new window) Bluesky
  • Share on LinkedIn (Opens in new window) LinkedIn
Like Loading...
← Older posts
Newer posts →

    I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

    • About
      • Internet Profile
      • Music Buying
      • Presenting Activities
    • Books & Publications
      • Logging in Action with Fluentd, Kubernetes and More
      • Logs and Telemetry using Fluent Bit
      • Oracle Integration
      • API & API Platform
        • API Useful Resources
        • Useful Reading Sources
    • Mindmaps Index
    • Monster On Music
      • Music Listening
      • Music Reading
    • Oracle Resources
    • Useful Tech Resources
      • Fluentd & Fluent Bit Additional stuff
        • Logging Frameworks and Fluent Bit and Fluentd connectivity
        • REGEX for BIC and IBAN processing
      • Formatting etc
      • Java and Graal Useful Links
      • Official Sources for Product Logos
      • Python Setup & related tips
      • Recommended Tech Podcasts

    Oracle Ace Director Alumni

    TOGAF 9

    Logs and Telemetry using Fluent Bit


    Logging in Action — Fluentd

    Logging in Action with Fluentd


    Oracle Cloud Integration Book


    API Platform Book


    Oracle Dev Meetup London

    Blog Categories

    • App Ideas
    • Books
      • Book Reviews
      • manning
      • Oracle Press
      • Packt
    • Enterprise architecture
    • General
      • economy
      • ExternalWebPublications
      • LinkedIn
      • Website
    • Music
      • Music Resources
      • Music Reviews
    • Photography
    • Podcasts
    • Technology
      • AI
      • APIs & microservices
      • chatbots
      • Cloud
      • Cloud Native
      • Dev Meetup
      • development
        • languages
          • java
          • node.js
          • python
      • drone
      • Fluent Observability
        • Fluentbit
        • Fluentd
        • OpAMP
      • logsimulator
      • mindmap
      • OMESA
      • Oracle
        • API Platform CS
          • tools
        • Helidon
        • ITSO & OEAF
        • Java Cloud
        • NodeJS Cloud
        • OIC – ICS
        • Oracle Cloud Native
        • OUG
      • railroad diagrams
      • TOGAF
    • xxRetired
    • AI
    • API Platform CS
    • APIs & microservices
    • App Ideas
    • Book Reviews
    • Books
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
    • drone
    • economy
    • Enterprise architecture
    • ExternalWebPublications
    • Fluent Observability
    • Fluentbit
    • Fluentd
    • General
    • Helidon
    • ITSO & OEAF
    • java
    • Java Cloud
    • languages
    • LinkedIn
    • logsimulator
    • manning
    • mindmap
    • Music
    • Music Resources
    • Music Reviews
    • node.js
    • NodeJS Cloud
    • OIC – ICS
    • OMESA
    • OpAMP
    • Oracle
    • Oracle Cloud Native
    • Oracle Press
    • OUG
    • Packt
    • Photography
    • Podcasts
    • python
    • railroad diagrams
    • Technology
    • TOGAF
    • tools
    • Website
    • xxRetired

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,616 other subscribers

    RSS

    RSS Feed RSS - Posts

    RSS Feed RSS - Comments

    April 2026
    M T W T F S S
     12345
    6789101112
    13141516171819
    20212223242526
    27282930  
    « Mar    

    Twitter

    Tweets by mp3monster

    History

    Speaker Recognition

    Open Source Summit Speaker

    Flickr Pics

    Boxer Rebellion @ Brixton ElectricBoxer Rebellion @ Brixton ElectricBoxer Rebellion @ Brixton ElectricBoxer Rebellion @ Brixton Electric
    More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View mp3muncher’s profile on WordPress.org
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Subscribe Subscribed
      • Phil (aka MP3Monster)'s Blog
      • Join 228 other subscribers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Subscribe Subscribed
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d