Tag Archives: AWS

JMESPath is represented using Railroad diagrams

31 Monday Oct 2022

Posted by in development, General, railroad diagrams, Technology

≈ Leave a comment

Tags

, , , , , ,

JMESPath is a mature syntax for traversing and manipulating JSON objects. The syntax is also supported with multiple language implementations available through GitHub (and other implementations exist). As a result, it has been very widely adopted; just a few examples include:

  • Azure CLI
  • AWS CLI and Lambda
  • Oracle Cloud WAF
  • Splunk

As the syntax is very flexible and recursive in its use following the documented notation can be a little tricky to start with. So following the syntax can be rather tricky. The complete definition runs to 97 lines, of which 32 lines focus on the syntactical structure. The others describe the base types such as numbers, characters, accepted escaped characters, and so on. Nothing wrong with this, as the exhaustive definition is necessary to build parsers. But for the majority of the time it is those 32 lines that we need to understand.

As the expression goes – ‘a picture says a thousand words’, there might not be a thousand words, but there is enough to suggest a visual representation will help. Even if the visual only helps us traverse the use of the detailed syntax. So we’ve use our favoured visual representation – the railroad diagram and the tool produced by Tab Akins to create the representation. We’ve put the code and created images for the syntax in my GitHub repository here, continuing the pattern previously adopted.

Here is the resulting diagram …

To make it easy to trace back to the original syntax document we’ve included groupings on the diagram that have names from the original speciofication.

Parts of the diagram make the expressions look rather simple, but you’ll note that it is possible for the sections to be iterative which allows for the expression to traverse a JSON object of undefined depth. But what can be really challenging is that an in many areas it is possible to nest expressions within expressions. Visually there is no simple way to represent the expression possibilities of this in a linear manner. Other than be clear about where the nesting can take place.

Costs in Multi-Cloud

28 Wednesday Aug 2019

Posted by in General, Oracle, Technology

≈ Leave a comment

Tags

, , , , , , , ,

Over the last couple of years, we have seen growing references to multi-cloud. That is to say, people are recognizing that organisations, particularly larger ones are ending up with cloud services for many different vendors. This at least in part has come from where departments within an organization can buy meaningful resources within their local budgets.

Whilst there is a competitive benefit of the recent partnership agreement between Microsoft and Oracle given the market margin AWS has in comparison to everyone else. Irrespective of the positioning with AWS, this agreement has arisen because of the adoption of multi-cloud. It also provides a solution to the problem of running highly resilient Oracle database setups using RAC, DataGuard etc can be made available to Azure without risk to security and the all-important network performances that are essentially to DB operation. Likewise, Oracle’s SaaS offerings are sector leaders if not best in place, something Microsoft can’t compete with. But at the other end, regardless of Oracle’s offerings, often organisations will prefer Microsoft development ecosystem because of the alignment to office tooling, the ease of building solutions quickly.

Multi-cloud even with the agreements like the Microsoft and Oracle one (See here), doesn’t mean there won’t be higher costs in crossing clouds. Let’s see where the costs reside …

  • Data egress (and in some cases ingress as well) from clouds costs. Whilst the ingress costs have been eliminated because it can be seen as a barrier to selling services, particularly big data. Data egress can, however, be an issue. Oracle has made this cost very low to be almost negligible, but not necessarily others as the following comparison shows …

  • Establishing the high-performance connections That the agreement supports needed between Azure and Oracle cloud is the same tech for the cloud to the ground do incur a cost. In Oracle’s case, there is a fee for the connection (not a large cost, but one that exists none the less) plus any traffic fees the provider of the network connection spanning the data centre locations. This is because you’re leasing capacity on someone’s dedicated fibre or MPLS services. Now, this should prove to be small as part of the enabler of this offering is that both Oracle and Microsoft cloud DCs are often actually physically provided by the same provider or at-least the centres a physically pretty close, as a result of both companies gravitating to locations close together because of the optimal highly available infrastructure (power, telecommunications) legal and commercial factors along with the specialist skills needed.

If data egress is the key challenge to costs, what drives the data egress beyond the obvious content for user interfaces? …

  • Obviously, you have the business data flows, some of these flows will be understood by the business community. But not all, this is down to the way data from the cloud can be exposed to another. For example inefficient services with APIs that require frequent polling and not using expressing the request efficiently, rather than perhaps express the request using HTTP header attributes and other efficiencies or even utilize frameworks such as webhooks so data can be pushed.
  • High-speed data access, often drives data replication having databases in multiple clouds with mirror image data in each location even if the majority of the data is not necessarily needed. This can happen with technologies such as Kafka which for non compacted topics will mean every event can be replicated even if that event has a short lifetime.
  • One of the hidden costs is the operational tasks of gathering logs to a combined view so end to end insights can be obtained. A detailed log can actually yield more ‘data’ by volume than the business flows themselves because it is semi-structured, and intended to be very readable and at the most granular level there to help debug and test.

In addition to the data flows, you need to consider how other linkages in addition to the Oracle-Azure connection are involved. In the detailed documentation, it is not possible to get your on-premises location connected to one of the clouds (e.g. Oracle FastConnect, and then assume your traffic can hop to Azure via the bridge using FastConnect and Azure’s ExpressRoute.  To add performance to your solution parts in both Azure and Oracle Cloud, you still need to have FastConnect and ExpressRoute configured to your on-premises location. This, of course, may impact how bulk data for lift and shift app use cases such as EBS may be applied. For example, if you choose to regularly bulk data transfer between on-premise and EBS via the app/middleware tier rather than via direct DB, and that mid-tier is running in Azure – you will need both routes established.

Conclusion

There is no doubt that the Oracle-Azure cloud to cloud linkage is a step forward, but ‘the devil is in the details‘ as the saying goes. To optimize the benefits and savings we’d suggest that you;

  • you’ll need to think through your use cases – understand data flow and volume (someone bulk syncing application data with a data warehouse?),
  • define a cloud data strategy – to layout principles, approaches and identify compliance needs, this is particularly helpful for custom solution development, so the right level of log data is consolidated with the important details, data retention addresses compliance requirements and doesn’t ratchet up unnecessary costs (there is a tendency to horde data just in case – if this is really wanted, think about how its stored),
  • based on business common usage models define a simple forecasting formula – being able to quantify data costs will always make it easier to challenge back data hoarding tendency,
  • confirm the inter-cloud network vendor charges when working with multi-cloud.

Message Push Listener – Article Update

30 Saturday Dec 2017

Posted by in Cloud, development, General, NodeJS Cloud, Technology

≈ Leave a comment

Tags

, , , , , , , , , , , , , ,

When I first wrote about Oracle Messaging Cloud we used a service called WebScript.io to make it easy to demonstrate the Message Push Listener. WebScript was essentially what we better know as a Serverless or Functions oriented offering (that is we wrote pieces of code and deployed them without any consideration servers etc). Well as I prepared my demos for Messaging Cloud for the UK Oracle User Group Tech 17 Conference I discovered that WebScript is being shutdown in December 2017.

In the light of this news, I needto provide an alternate implementation for my Message Push Listener demo Google’s Cloud Functions.  Before I go into the Google implementation I thought it worth sharing how I landed on Google’s offering.

The Google Cloud Functions is a new service that has been launched with an interesting future. I had hoped to try using project Fn (Oracle’s open source serverless offering) but the cloud offering is not yet publicly available – although you can run Fn on any platform today if you’re prepared to invest in setting up the environment (defeating the point of serverless). I know some of Oracle’s Developer Champions have had a preview so it cant be too far away now. I’m sure when we get a chance to access the new Cloud Native Service announced which will include Fn we will revisit it. Before settling on Google we looked at several other offerings in the serverless space. Whilst this is not an exhaustive analysis it should help give a sense of the challenges and ease of adoption. If you search today on Serverless you’ll most commonly come across Auth0’s WebTask.io, AWS Lambda and IBM OpenWhisk (based on Apache OpenWhisk).

WebTask.io

I started with WebTask.io and it was very nearly a done deal, with a nice easy to work with Cloud Development Platform, integrated testing. Extensive support for Node.js and a number of standard frameworks to use with it such as Express available without doing anything.

Other languages are supported as well by WebTask.io. But as I’m trying to create a demo that warrants very little explanation of the Serverless platform we didn’t dig in to this area. Everything went swimmingly until I tried to setup external calls to my function. This became a headache as the security model whilst not overly complex (several ways to provide the REST call with authentication e.g. adding a key in the URI). The process of generating and associating the credentials was far from clear in the documentation.

AWS Lambda

I moved to look at AWS Lambda, this I just found horribly confusing to get started with. I have heard others saying that getting going isn’t straight forward. So I found myself giving up pretty quickly as the setting up wasn’t that clear. Whilst having used AWS with its IaaS capabilities which is both powerful, flexible and pretty easy to get to grips with if you understand basic ideas like virtual machines this didnt hold true fory Lambdas.

OpenWhisk

As for OpenWisk, we started to look at it, but getting a 404 error when trying to access the Editor following the IBM documentation didn’t inspire confidence. The was plenty of supoprting documentation which explains how OpenWhisk works.

openwhisk_flow_of_processing

The Execution framework for OpenWhisk

  1.  Ningx is used for SSL termination and forwarding appropriate HTTP calls to the next component
  2. Controller first disambiguates what the user is trying to do. It does so based on the HTTP method you use in your HTTP request. This is a Scala solution built using Akka & Spray. This includes ..
  3. Verification who you are (Authentication) against a CouchDB based identiy store.
  4. Once approved details of the Action to be executed is retrieved from the whisks database in CouchDB.
  5. With information on what to do, the action of service discovery is formed using Consul. Which tracks the available executors in the system. Those executors are called Invokers
  6. Kafka is then used to mitigate the demand pipeline from a failure by recording the request and the consumer (invoker) identified by Consul.
  7. The invoker is built using Scala and uses a Docker instance to run the Action which could be anything e.g. Node.js. The action is injected into the container to be processed.
  8. As the result is obtained by the Invoker, it is stored into the whisks database as an activation under the ActivationId. The whisks database lives in CouchDB.

In addition to the 404, as you can see we have a two step process to execute an action and return a respoinse. However the Message Push Listener initial challenge needs a call and response in a single step. So trying to massage this into a call and response is going to be challenging and a distraction from what we want to be conveying.

Using Google Functions

This brings us Back to Google, whilst the Cloud IDE is not as elegant or mature as WebTask it was sufficient and the security model wasn’t imposed. I liked the documentation when needed to refer back  to it, but to be honest it is pretty intuitive. You can’t fault the docs, to the point Google gave time over to explaining how to manage or avoid incurring costs.

Setting up, was very simple, and then once you’ve choosen your cloud services you get a dashboard like this:

Google CLoud mgmt

Google provides the idea of projects which allows you to group pieces together – such as related functions. Each project is name space separated. If we then navigate into a Functions project we get a view as follows:

Google cloud functionsAs you can see in the preceeding diagram I created two functions within a project called OMCS. From here you can create more functions in your project or drill into an individual function, as the following view shows:

Google Functiuons performance

An individual function provides you with several tabbed views overing the Gernal information  (as shown above) or Trigger, Source and Testing. We can see the other views in the following screenshots. The following screen shot shows the Functions Editor, as you can see it is fairly simple – but sufficient to do the job.

GoogleCloud-OMCS

Once saved, if valid the code will automatically get deployed, or you can work offline and then upload the code if you want to use a nice editor like Sublime.

with your code edited and saved, then the next step is to invoke it. This can be done with the next tab, or the details such as the URI can be copied and you can test from your preferred test tool such as SOAPUI, Postman and APIFortress.

Google functions Trigger

The testing view allows you 5o define input and output values, along with the outcomes. Personally I worked with SOAPUI.

Google Functiuons Test

The important thing with running tests or diagnosing issues, is to be able to examine execution logs. In this area Google Functions is pretty feature rich with a solution that works in a style somewhat like the searching in Splunk (and I’m sure other log analytics tools) where you can drill into the logs and build log filters on the fly. The log view is shown in the next screenshot.

Gogole Functions Logging 2

as you can see tool looks pretty straight forward and uncomplicated to use, with freedom to adapt how you work to your preferred style. Based on my experience of using Project FN on my desktop – it is this simplicity I think we’ll see with the Cloud Native Platform from Oracle when it becomes available.

Finally, let’s take a look at the code in Google Functions code produced for this example:

code

conclusion

Google Code whilst its UI is a bit basic, it is easy to use and get started, certainly for using as a demo platform or perhaps for creating stubs, test and mock end points. Having been critical of the other offerings for security and it getting in the way of a simple illustration it is possible that the Google Functions may need some work in this area. I didn’t see anything that obviously integrated security features in easily.

Back to my Orginal Articles…

Just to tie back the impacted articles …

Ansible Book Review Part 4

14 Saturday Mar 2015

Posted by in Book Reviews, Books, Packt, Technology

≈ 1 Comment

Tags

, , , , , , , , , , , , , ,

This the final part of the detailed look at Packt book, Learning Ansible. As the book says in the opening to chapter 6 we’re into the back straight, into the final mile. The first of  two final  chapters look at provisioning of platforms on Amazon AWS, DigitalOcean and the use of the very hip and cool Docker plus updating your inventory of systems given that we have dynamically introduced new ones.  The approach is illustrated by not only instantiating servers but delivering a configured Hadoop cluster. As with everything else we’ve seen in Ansible there isn’t a standardised approach to all IaaS platforms as that restricts you the lowest common denominator which is contrary to Ansible goals described early on. But deploying the Hadoop elements on the two cloud IaaS providers is common.  Although the chapter is pretty short, I did have to read through this more carefully, as the book leverages a lot of demonstrated features from previous chapters (configuration arrays etc) which meant seeing the key element of the interaction with AWS was harder. It does mean if you tried diving into this chapter straight away, although not impossible does require a bit more investment from the reader to see all the value points. That said it is great to see through the use of the various features how easy to setup the provisioning in the cloud is, and the inventory update. Perhaps the win would have been to just so the simple provision and then the clever approach.

Chapter 7 focuses on Deployment. When I read this, I was a little nonplussed, hadn’t we been reading about this in the previous 6 chapters. But when you look at the definition provided:

“To position (troops) in readiness for combat, as along a front or line.”
Excerpt From: “Learning Ansible.” Packt Publishing. 

You can start to see the true target of what we’re really thinking about, which is the process of going from software build to production readiness. So having  gone through the software packaging activities you need to orchestrate the deployment across potentially multiple servers across a server farm. This orchestration piece is really just pulling together everything that has been explained before but also share some Ansible best practise. Then finally an examination of the Ansible approach for the nodes to pull deployments and updates.

The final piece of the book is an Appendix which looks at the work to bring Ansible to the Windows platform, Ansible Galaxy and Ansible Tower.  Ansible Galaxy is a repository of roles build by the Ansible community. Ansible Tower provides a web front end to the Ansible server. The Tower product is the commercial side of the Ansible company – and effectively sales here fund the full time Ansible development effort.

So to summarise …

The Learning Ansible book explains from first principles to the very rich capabilities of building packaging software, instantiating cloud servers or containers through to configuring systems and deploying applications into new environments; and then capturing instantiated system details into the Ansible inventory. How Ansible compares with the more established solutions in this space in the form of Puppet and Chef is discussed, and the pros and cons of the different tools. All the way through, the books has been written in an easy engaging manner. You might even say wonderfully written. The examples are very good with the possible exception of 2 cases (just merely good in my opinion), the examples are supported with very clear explanations that demonstrate the power of the Ansible product. Even if you choose not to use Ansible, this book does an excellent job of showing the value of not resorting to the ‘black art’ of system build and configuration and suggesting good ways to realising automation of this kind of activity, in many place undoubtedly thought provoking

Prior Review Parts:

Gaps in Oracle’s Cloud Cover?

20 Monday Jan 2014

Posted by in General, Oracle

≈ 2 Comments

Tags

, , , , , , ,

As an Enterprise Integration Architect I need to get my hands dirty with products such as Oracle’s SOA suite and AIA Foundation Pack.  In the past, I’ve dealt with this by talking with our infrastructure team – obtaining a VM or a laptop with sufficient guts to host SOA Suite (and it doesn’t have a small footprint).  This is all well and fine, but means I have to lug a big old laptop (our current standard laptop spec’s are lovely light machines with SSD’s but just don’t pack the punch for SOA Suite when it comes to memory) or have to leap through a series of security steps to get remote access – again not a problem unless I want to share my skunk works with someone outside the organisation.  Nor, do I really want to invest chunks of time building a SOA Suite environment to work with – I don’t do it enough to be able to throw these things together quickly.  Even Oracle recognise that with the support for a prebuilt VirtualBox with SOA Suite and BPM. The only problem with VirtualBox is I’ve saved on the build time, but still need that heavy laptop or remote access.

Oracle Cloud Java

With the rise of the cloud, particularly Oracle’s big push (announcements at Open World 2013), Amazon offering small footprint dev platforms more or less for free I thought we’d be able to get a PaaS deployment of SOA Suite – after all Oracle offer a range of Fusion Apps in the cloud (built on top of SOA Suite technologies), have launched development of Java and ADF solutions in their cloud and even offer Weblogic on Microsoft’s Azure.  How I wrong could I have been.  So I started looking around, perhaps someone has an AMI ready to go – well sort of if I want 10g.  So I’ve dug around, and found the odd provider who could deliver what was needed (e.g. Titan GS) but we’re talking big bucks – not a low cost dev/skunk works environment.  

This is very surprising really, and sort of ironic, given Oracle’s recent announcement for SaaS Adapters for the likes of SalesForce and WorkDay along with convenience tooling to connect to Oracle Cloud solutions such as HCM.  I say ironic, because to use the cloud adapters you can’t have a SaaS middleware; in fact the whitepaper Oracle published on Simplifying Cloud Integration infers/assumes that you’d be hosting your own middleware.  So if a midsized business has Has HCM, Taleo etc for their staffing management, SalesForce for the Sales/CRM operations and perhaps EBis or JD Edwards to move your business into the cloud you have to either go IaaS and carry the labour of maintaining the middleware platform or self host (one of the things the adoption of SaaS is trying to free you from).

All of this seems to be a really missed opportunity for Oracle.  If the oracle wants to host the world (and I think Larry Ellison would like that) and definitely get into that midmarket sector that JDEwards particularly tries to inhabit they need to make it easy for businesses to cloud all aspects of their IT solution, that includes orchestrating specialist solutions that will be hosted by someone other than Oracle (shock, horror). All of which means SOA Suite (and ideally AIA) need to be in the cloud.

As for my problem, its either the pain of building something on Amazon or setting up several copies of the VirtualBox deployment linked to a common GIT repository, and hope those I would like to collaborate with can also get their hands on the virtualbox and connect to GIT.