• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
    • Python Setup & related stuff
  • Music
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: GitHub

Demo Fluentd using Ubuntu with optional inclusion of OpenSearch and OCI Log Analytics

17 Wednesday Aug 2022

Posted by mp3monster in Cloud, development, Fluentd, General, Oracle, Technology

≈ Leave a comment

Tags

Cloud, demo, Fluentd, GitHub, Log Analytics, log simulator, OCI, OpenSearch, Oracle, Ubuntu

One of the areas I present publicly is the use of Fluentd. including the use of distributed and multiple nodes. As many events have been virtual it has been easy to demo everything from my desktop – everything is set up so I can demo things very easily. While doing this all on one machine does point to how compact and efficient Fluentd is as I can run multiple instances concurrently it does undermine distributed capabilities somewhat.

Add to that I now work for Oracle it makes sense to use OCI resources. With that, I have been developing the scripts to configure Ubuntu VMs to set up the demo environments installing Ruby, Fluentd, and various gems needed and pulling the relevant configurations in. All the assets can be found in the GitHub repository https://github.com/mp3monster/logging-demos. The repository readme includes plenty of information as well.

While I’ve been putting this together using OCI, the fact that everything is based on Ubuntu should mean it can be run locally on VMs, WSL2, and adaptable for MacOS as well. The environment has been configured means you can still run on Ubuntu with a single node if desired.

Additional Log Destinations

As the demo will typically be run on OCI we can not only run the demo with a multinode setup, we have extended the setup with several inclusion files so we can utilize OCI services OpenSearch and OCI Log Analytics. If you don’t want to use these services simply replace the contents of several inclusion files including files with the contents of the dummy_inclusion.conf file provided.

Representation of the Demo setup

The configuration works by each destination having one or two inclusion files. The files with the postfix of label-inclusion.conf contains the configuration to direct traffic to the respective service with a configuration that will push log events at a very high frequency to the destination. The second inclusion file injects the duplication of log events to each service. The inclusion declarations in the main node Fluentd config file references an environment variable that should provide the path to the inclusion file to use. As a result, by changing the environment variable to point to a dummy file it becomes possible o configure out the use of one of the services. The two inclusions mean we can keep the store declarations compact and show multiple labels being used. With the OpenSearch setup, we have a variant of the inclusion file model where the route inclusion can reference the logic that we would use in the label directly within the sore declaration.

The best way to see the use of the inclusions is to experiment with setting the different environment variables to reference the different files and then using the Fluentd dry-run feature (more on this in the book).

Setup script

The setup script performs a number of tasks including:

  • Pulling from Git all the resources needed in terms of configuration files and folders
  • Retrieving the necessary plugins against the possibility of their use.
  • Setting up the various environment variables for:
    • Slack token
    • environment variables to reference inclusion files
    • shortcut environment variables and aliases
    • network (IP) address for external services such as OpenSearch
  • Setting up a folder for OCI tokens needed.
  • Setting up temp folders to be used by OCI Plugins as a file-based cache.

Using OpenSearch

OpenSearch setup is documented in a tutorial here, and a Reference Architecture at the time of writing there isn’t a one-click deploy Terraform available in the Oracle Reference Architecture library on GitHub.

Currently, the setup for OpenSearch means manually adding the node1 index into the configuration.

Useful Links:

  • https://opensearch.org/
  • https://docs.oracle.com/en/solutions/oci-opensearch-application-search/#GUID-C968ACCC-2E79-4C88-A466-F9DF2503E920
  • https://www.opensearch.org/blog/technical/2022/02/getting-started-with-fluentd-and-opensearch/?utm_source=pocket_mylist

Log Analytics

Feeding the log analytics service is a more complex process to set up as the feeds need to have metadata about the events being ingested. The downside is the configuration effort is greater, but the payback is that it becomes easier to extract meaningful information quickly because the service has a greater understanding of the content. For example, attributing the logs to a type of source means the predefined or default log formats are immediately understood, and maximum meaning can be retrieved from the log event.

Going to OCI Log Analytics does cut out the need for the Connections hub, which would allow rules and routing to be defined to different OCI services which functionally can help such as directing log events to PagerDuty.

Useful Links

  • https://docs.oracle.com/en/solutions/oci-opensearch-log-analytics/index.html#GUID-9A3E3E7A-C899-4D43-8DA0-4BA7FA3E44ED
  • https://docs.oracle.com/en/cloud/paas/management-cloud/logcs/install-output-plug.html
  • https://docs.oracle.com/en/learn/oci_logging_analytics_fluentd/index.html

Demo Enhancements to come

There are a few things we’re planning to do with the demo:

  • Create a terraform script to perform all the environment setup
  • Integrate the configuration script into the terraform
  • Provide some simple dashboard insights for OpenSearch – currently, this is all manual
  • Basic setup for OCI Log Analytics

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

GitHub Actions on OCI

29 Monday Nov 2021

Posted by mp3monster in development, General, Oracle, Technology

≈ Leave a comment

Tags

actions, Cloud, free, GitHub, OCI, Presentation, runners, slide

Today was the first run of some new presentation material looking at the use of GitHub Actions using Runners deployed on OCI Free Tier. The presentation was actually physical rather than virtual which was after 2 years of virtual presenting, rather refreshing. Not to mention the UKOUG hosted the event at the Oval Cricket ground, which made for an interesting venue. The example configuration is included in our GitHub OCI Utilities repository (as we use this solution to help validate and test our development work).

The presentation itself (which includes screenshots of the setup of a simple Action and runner) is here, note I have disconnected my Runners, but you will be able to see the Action configuration but if you try to trigger activity through my repository then nothing will happen.

GitHub Actions – using Free Oracle Cloud Infrastructure (OCI) from Phil Wilkins
Implementing GitHub Action Handlers (aka Runners) on OCI
Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Oracle Cloud + GitHub Actions

05 Monday Jul 2021

Posted by mp3monster in Cloud, development, General, Oracle, Technology

≈ Leave a comment

Tags

@oracledevs, @soacommunity, actions, always free, arm, CICD, Cloud, flake8, free, GitHub, OCI, Oracle, pytest, python, unit test

While there is a deserved amount of publicity around the introduction of ARM compute onto OCI with the ARM Ampere CPU offering, and the amazing level of always free compute being provided (24GB of memory and 4 cores which can be used in any combination of servers). There have been some interesting announcements that perhaps haven’t drawn as much attention that they deserve. This includes OCI support for GitHub Actions, plus several new DevOps services and an Artifact Registry. We’ll comeback to the new services in another post. Today, let’s look at GitHub Actions.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Development Standards for API Policies?

16 Monday Dec 2019

Posted by mp3monster in API Platform CS, General, tools

≈ Leave a comment

Tags

API, Azure, code, GitHub, Oracle, quality, regex, utility

When it comes to development, we have had coding standards for almost as long as we have been coding. We tend to look at coding standards for purposes of helping to promote good quality code and reduce the likelihood of bugs and so on. But they also help with readability, making it easy to navigate a code base and so on. This is sufficiently important that there is a vast choice of tools to help us ensure we align with good practices.

That readability etc, when it comes to code interfaces lends to making it easier to use an interface as it promotes consistency and as Don Norman would say avoids ‘cognitive load‘, in other words, the effort involved in performing actions with the interface. Any Java Developer will tell you, want to print out an object (any object) you get a string representation using the .toString() method and direct it using the io packages.

That consistency and predictability are important not just for code if you look at any API best practises documents you’ll encounter directly or indirectly the need to use conventions that drive consistency – use of singular or plural for the name of entities, application of case – camel case, snake case etc. Good naming etc and we’ll see related things appear together in the documentation. Products such as Apiary and SwaggerHub include tooling to help police this in our API design work.

But what about policies that we use to define how an API Gateway handles the receipt and routing of API invocations? Well yes, we should have standards here as well. Some might say, governance gone mad. But gateways are often shared services, so making it easy to see and logically group APIs together at very least by using a good naming convention will help as a minimum. If API management is being administered in a more DevOps fashion, then information security professionals will probably want assurance that developers are applying policies in a recommended manner.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Making Scripts Work with IDCS Deployed PaaS

09 Saturday Feb 2019

Posted by mp3monster in API Platform CS, APIs & microservices, General, Oracle, Technology, tools

≈ 5 Comments

Tags

API Platform, GitHub, Groovy, IDCS, OAuth, Script, tokens

A while back I made some utilities I developed to help with managing the API Platform. At the time we didn’t have access to an IDCS based environment, so credentials worked using basic auth (I.e. user name and password). But with environments managed by IDCS tokens are used.

As a developer with a Java background I have to admit to liking Groovy over Python for scripting, not to mention for the API Platform groovy is part of the gateway deployment and SDK. Meaning it is readily available in its 2.x form (3.x is relatively recent and aligning to the latest Java idioms). We haven’t tested against Groovy 3.0

Thank you to Andy Knight for sharing with us some Java code which I adapted to be pure Groovy (removing external dependencies for processing JSON). The result is a script that can be taken can be taken and worked into other scripts (which is what we have done for our previously provided scripts – Understanding API Deployment State on API Platform, Managing API Policy Versioning in Oracle API Platform, Documenting APIs on the Oracle API Platform). But this script can be used to get a token and display it on the command line or write it to file. Writing tokens to files is generally not good practise, but as a temporary measure when working on developing scripts arguably a managed risk.

The script can be found at https://github.com/mp3monster/API-Platform-Utils/tree/master/getToken and all the details on using the script can be obtained by passing -h as a parameter. The important thing is to understand how to obtain the Client ID and Client Secret, the details of which are described at https://docs.oracle.com/en/cloud/paas/api-platform-cloud/apfad/find-your-client-id-and-client-secret.html

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Oracle Developer Meetup – London Feb 19

05 Tuesday Feb 2019

Posted by mp3monster in Dev Meetup, development, General, Helidon, Oracle, Technology

≈ 2 Comments

Tags

developer, GitHub, Helidon, meetup, micro profile, Microservices, open tracing, Oracle, signatures

Last night was the first Oracle Developer Meetup in London for 2019.  We were very fortunate to have Tomas Langer fly over to talk about the new micro container/framework being developed as an open-source solution by Oracle.

 

Oracle Developer Meet-up - Tomas Langer presenting on Helidon

Tomas, opened by explaining the evolution of the micro-profile being championed by the Eclipse Foundation who are now the guardians of J2EE also known as Jakarta and how the J2EE and Micro-Profile standards compare (in simplistic terms – micro-profile is J2EE stripped back to be simple and support what is typically needed in a micro-service world).

The presentation then went onto compare Helidon SE and Helidon MP (micro-profile).  What was really pleasing is that with a couple of exceptions everything that Helidon MP can do, can be done in the SE edition, the difference being that for SE you have to implement more code, rather than the auto-magic of annotations, but in return you have a Reactive Java platform with a development paradigm which relates to JavaScript Express.

In addition to talking about what can be done, Tomas described the kinds of features being developed, this includes:

  • Bringing micro-profile support up to the very latest specification,
  • More reactive persistence technologies support,

With the scene set, Tomas then worked through a series of live code scenarios starting with a clean slate and building Hello World in both the SE & MP models illustrating the differences in approach.  This was then built upon to add the following capabilities:

  • Tracing (using Zipkin leveraging the Open Tracing Standard)
  • Dynamic configuration
  • Security (including Signatures)
  • Fault Handling (just MP)

You can get the complete example which uses Helidon in both configurations from Tomas GitHub.

In addition to Helidon itself on GitHub, there are resources provided include rich documentation and examples of each key feature.  Plus a Slack community, that if you contact any of the Helidon team will get you invited allowing you to discuss with the development team how to do things along with other developers using Helidon.

Tomas can be contracted via @Langer_Tomas.  Helidon project also has its own Twitter account – Helidon Project

Helidon itself can be found at:

  • Helidon website
  • GitHub
  • Helidon documentation

I have previously blogged on Helidon at Exploring Helidon – Part 1

 

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Managing API Policy Versioning in Oracle API Platform

27 Tuesday Feb 2018

Posted by mp3monster in API Platform CS, APIs & microservices, Books, General, Oracle, Technology

≈ 6 Comments

Tags

API Platform, API-P, APIs & microservices, CLI, command line, GitHub, Oracle, revert forward, utility, versioning

Updated reflecting changes discussed in blog post:
 Making Scripts Work with IDCS Deployed PaaS

Oracle’s API Platform (API-P) product avoids the use of external configuration management. If you want to better understand why, then checkout our forthcoming book as it goes into detail about why this is the case (it can be pre-release version of the book can be obtained here). In a previous blog I wrote about and illustrated the use of the API-P’s own APIs so that it was possible to see what API iterations had been deployed to API Gateways.

In this blog I want to explore the issues of version management a bit further. API-P provides internal version management through the idea of iterations as previously explored (Understanding API Deployment State on API Platform). In addition to this there are API policy attributes called version, status etc. This information whilst having some impact on behaviour reflects the version of the ‘contract’ that the API represents between the consumer and provider, and requires a manual change.

The API policies themselves are version tracked through the iteration identifiers. Each time a policy is saved the iteration is is incremented. What the API-P doesn’t support is the concept of branching. In relatively simple API Policy branching is unlikely to ever be an issue.

Why is a reversion capability needed?

Let’s take a more complex scenario.  In our book API Platform we introduced the some APIs that would allow the retrieval of meta data about artists in the record companies’ catalog. It has however come to light that the API has been targeted with malicious calls, firstly through trying to attack using injection attacks and secondly trying to overload the back end by creating data requests that make the back-end work hard in retrieving data.

To defend against this, the API Policy has been enhanced to include some custom groovy policies to inspect the values provided. Strictly speaking following the principles of Semantic Versioning the API version should go from 1.0.0 to 1.0.1. However seeing that the ‘contract’ as presented to the consumer hasn’t really changed – the data models are the same, the URI goes unaltered resulting in the implementation team not changing the version.

During development processes, it is not unusual to be developing existing logic, and decide that approach being used isn’t right or not going to perform as well. So you abandon your changes and revert back to the last approved version. However, this isn’t possible as any save will result in a new iteration. The API-P will be getting some enhanced version management features. But today to be able to undo the changes we need a means to ‘revert forward‘ (hence the tool name) that is to take an older iteration and make it the latest, as illustrated in the next diagram.

Today the API-P doesn’t provide a means to perform this process within the User interface. However when looking at a gateway you can review the API policy deployed. As we established previously that you may have different gateways deployed with different iterations. Given this, as API-P has been built true to the principles of separating the UI from the back-end through the use of APIs we can deduce there should be a means to get the details of an API with a specific iteration.

To this end we have built on the pattern previously illustrated to provide the means to ‘Revert Forward’ by creating a Groovy script that will use the APIs provided by API-P to retrieve an iteration and push it back at the latest version. When the policy is pushed back it also modifies the description to show which iteration has been pushed.

You may ask, why not use the conventional developer approach of branching as suggested in the following diagram. However API-P’s iteration framework doesn’t extend to support this.

The next with this is pretty predictable – how do I know which iteration to revert to. You have two options here, firstly either revert in order so you can see the prior version in turn – which whilst visually good, is not necessarily the most practical option. So in the tool we have a parameter that will allow you to display on the console the configuration of each iteration. This does mean you are going to see the policies in a JSON presentation. To make life easier we would recommend good practise and recording in the policy description information that helps determine the policy’s characteristics – and this can be used to better determine iteration behaviour.

If we are able to take an earlier iteration and make it the latest one by pushing it back then it is a short step to actually target a different management cloud in effect migrating the policies. Whilst possible it comes with some serious cautions …

  • You risk undermining your version management, which management cloud has the master, and the iteration numbers will NOT migrate so it’s not like this info can be used to distinguish the laster version
  • The logic included doesn’t accommodate handling differences in policy versions – so if trying move between instances of the API Platform they need to be the same version otherwise your configuration could make a mess of thing
  • This issue is further compounded if you are deploying custom Java policies.
  • Environment specific policies simply won’t work for example gateway based routing.

Oracle does not recommend that the policies be stored anywhere outside of the platform, whilst it this utility makes that a possibility, it deliberately avoids writing any of the information to the file, the policies only reside outside of the platform for the duration of the process execution.

The Tools Commands

All the parameters assume the values will not contain any space characters. Each command is preceded by a dash eg. revertForward.groovy -inpassword mypass -inSvr https://a.b.com

  • -h or -help – provides this information
  • -inName – user name to access the source management cloud
  • -inPass – password for the source management cloud
  • -inSvr – The server address without any attributes e.g. https://1.2.3.4
  • -policy – numeric identifier for the policy of interest
  • -iter – iteration number of interest for the policy – optional
  • -outName – optional, the target management cloud username, only needed for migrations
  • -outPass – optional, the target management cloud password, only needed for migrations
  • -outSvr – optional, the target management cloud server address – same formatting as inSvr, only needed for migrations
  • -override – optional, if migrating to another management, tells the script to replace the existing policy of the same name if found
  • -view – optional, separate command to allow viewing of the policy – requires one of the following values:
    • display – displays all the details of the policy, if no iteration is provided this will be the latest iteration
    • summary – provides the headline information of the policy including name, change date etc
    • summary-all – summarizes all the iterations from the current one back to the 1st
  • -debug – optional, will get script to report more information about what is happening

The code can be obtained from my GitHub repository here.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,574 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

April 2023
M T W T F S S
 12
3456789
10111213141516
17181920212223
24252627282930
« Mar    

Twitter

  • Get all the details about the new enhancements to @Oracle Container Engine for Kubernetes, including Serverless… twitter.com/i/web/status/1…Next Tweet: 3 days ago
  • RT @TechWeekRO: With over 25 years of experience in the software industry, Phil Wilkins, Cloud Developer Evangelist at @Oracle, is coming t…Next Tweet: 3 days ago
  • SSH Key File Permissions blog.mp3monster.org/2023/03/28/ssh…Next Tweet: 4 days ago
  • Oracle's Assurance Service gives customers the proactive guidance they need to move their organization forward whil… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Fraud affects many businesses and can be costly. But there’s a way to fight it. Scalable Machine Learning algorithm… twitter.com/i/web/status/1…Next Tweet: 4 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 218 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d bloggers like this: