24 Tuesday Apr 2018
Posted in General, Dev Meetup
Another month and another Meetup. Last night was the quietest session we’ve had. But then we covered Identity Management and whilst the subject is so important it isn’t too popular.
Tanks to Atul Kumar for taking on the task of presenting on this subject. You can find out more about Atul at:
The content covering use of Messaging Cloud and our drone project can be seen here:
17 Saturday Mar 2018
Posted in General
Last week we attended the Oracle EMEA PaaS Forum in Budapest. I have to say we’re proud to be part of a team that has picked up two conference awards. Firstly our CTO Luis Weir for Contributions to API Solutions and then the wider team have collected an award for overall Outstanding PaaS Contributions.
L-R, me, Amy Grange, Soham Dasgupta (Capgemini Netherlands), Luis Weir (Oracle Delivery Unit CTO), Jurijs (Yury) Fjodorovs
Some photos from the trip can be found a here.
27 Tuesday Feb 2018
Posted in API Platform CS, APIs, Books, General, Oracle, Technology
Tags
API Platform, API-P, APIs, CLI, command line, GitHub, Oracle, revert forward, utility, versioning
Oracle’s API Platform (API-P) product avoids the use of external configuration management. If you want to better understand why, then checkout our forthcoming book as it goes into detail about why this is the case (it can be pre-release version of the book can be obtained here). In a previous blog I wrote about and illustrated the use of the API-P’s own APIs so that it was possible to see what API iterations had been deployed to API Gateways.
In this blog I want to explore the issues of version management a bit further. API-P provides internal version management through the idea of iterations as previously explored (Understanding API Deployment State on API Platform). In addition to this there are API policy attributes called version, status etc. This information whilst having some impact on behavior reflects the version of the ‘contract’ that the API represents between the consumer and provider, and requires a manual change.
The API policies themselves are version tracked through the iteration identifiers. Each time a policy is saved the iteration is is incremented. What the API-P doesn’t support is the concept of branching. In relatively simple API Policy branching is unlikely to ever be an issue.
Let’s take a more complex scenario. In our book API Platform we introduced the some APIs that would allow the retrieval of meta data about artists in the record companies’ catalog. It has however come to light that the API has been targeted with malicious calls, firstly through trying to attack using injection attacks and secondly trying to overload the back end by creating data requests that make the back-end work hard in retrieving data.
To defend against this, the API Policy has been enhanced to include some custom groovy policies to inspect the values provided. Strictly speaking following the principles of Semantic Versioning the API version should go from 1.0.0 to 1.0.1. However seeing that the ‘contract’ as presented to the consumer hasn’t really changed – the data models are the same, the URI goes unaltered resulting in the implementation team not changing the version.
During development processes, it is not unusual to be developing existing logic, and decide that approach being used isn’t right or not going to perform as well. So you abandon your changes and revert back to the last approved version. However, this isn’t possible as any save will result in a new iteration. The API-P will be getting some enhanced version management features. But today to be able to undo the changes we need a means to ‘revert forward‘ (hence the tool name) that is to take an older iteration and make it the latest, as illustrated in the next diagram.
Today the API-P doesn’t provide a means to perform this process within the User interface. However when looking at a gateway you can review the API policy deployed. As we established previously that you may have different gateways deployed with different iterations. Given this, as API-P has been built true to the principles of separating the UI from the backend through the use of APIs we can deduce there should be a means to get the details of an API with a specific iteration.
To this end we have built on the pattern previously illustrated to provide the means to ‘Revert Forward’ by creating a Groovy script that will use the APIs provided by API-P to retreive an iteration and push it back at the latest version. When the policy is pushed back it also modifies the description to show which iteration has been pushed.
You may ask, why not use the conventional developer approach of branching as suggested in the following diagram. However API-P’s iteration framework doesn’t extend to support this.
The next with this is pretty predictable – how do I know which iteration to revert to. You have two options here, firstly either revert in order so you can see the prior version in turn – which whilst visually good, is not necessarily the most practical option. So in the tool we have a parameter that will allow you to display on the console the configuration of each iteration. This does mean you are going to see the policies in a JSON presentation. To make life easier we would recommend good practise and recording in the policy description information that helps determine the policy’s characteristics – and this can be used to better determine iteration behaviour.
If we are able to take an earlier iteration and make it the latest one by pushing it back then it is a short step to actually target a different management cloud in effect migrating the policies. Whilst possible it comes with some serious cautions …
Oracle does not recommend that the policies be stored anywhere outside of the platform, whilst it this utility makes that a possibility, it deliberately avoids writing any of the information to the file, the policies only reside outside of the platform for the duration of the process execution.
All the parameters assume the values will not contain any space characters. Each command is preceded by a dash eg. revertForward.groovy -inpassword mypass -inSvr https://a.b.com
The code can be obtained from my GitHub repository here.
01 Thursday Feb 2018
Posted in API Platform CS, APIs, General, Oracle, Technology
Tags
API, API Platform, code, development, node.js, Oracle, PlatformTest, policies
When configuring API Policies in the the Oracle API Platform it helps if there is a simple back end that can take the received payload and record the sent values (header & body) as well as reflect the call details back as the response, or possibly respond with a test payload (so that response policies, particularly policies that require payload navigation can be exercised correctly). By having this facility it becomes a lot easier to determine whether the policies are executing correctly in terms of routing, transforming, filtering etc. without needing to worry about whether the API implementation is correct. You could say that this is a kind of mock for testing the API Platform.
The added benefit of having a mock back end is that it is easy to ‘smoke test’ a gateway deployment very easily. Particularly if the mock is happy to receive any form of call.
Whilst implementing such a capability can be done in pretty much any language and platform you like. We have in the past for example built a Springboot Java application that can have the dependencies configured to then deploy into WebLogic for example. We have come to refer these test apps/mocks as PlatformTests as that’s exactly what they help do. A Node.js implementation of a PlatformTest such as as the following implementation is particularly appealing as the Node.js footprint is small and simple to deploy and undeploy. A basic Node.js implementation can also consume any URL and operation you choose to use. The nature of JavaScript makes it very quick to adapt the mock if need be. Although in the ideal world, we write the solution once and then use simple configuration to tune behavior.
The following code looks for a local file called testResponse.json if found then returns the content of the file (assumed to be JSON) otherwise it reflects back in the body, the received header and body. This reflection makes it extremely easy to see how the policies have changed the inbound call. The content is also logged to the console – making it easy to also see what came through to the back end.
The implementation also assumes port 8080, but changing the port is exceptionally easy.
There one enhancement planned, and this is to allow the response test payload to be handled as XML. This will need a little tweaking of the code as presently a JSON Object is currently stringified.
The code is also available in my GitHub repository – https://github.com/mp3monster/Utils/blob/master/PlatformTest.js and an example test response file is at https://github.com/mp3monster/Utils/blob/master/testResponse.json
const http = require('http');
const fs = require('fs');
// create a simple HTTP server that will handle the requests
http.createServer((request, response) => {
const { headers, method, url } = request;
console.log("Called at " + new Date().toLocaleDateString());
let body = [];
request.on('error', (err) => {
console.log("Svr Error Handler :" + err.toString);
response.statusCode(400);
response.end();
}).on('data', (chunk) => {
body.push(chunk);
}).on('end', () => {
body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
});
// record in the console what details have been received
console.log ("Received:\nMethod:" + method.toString() +
"\n URL:"+ url.toString + "\nheaders:\n"+headers.toString() +
"\nBody:\n" + body);
// now build the response
response.setHeader('Content-Type', 'application/json');
response.setHeader('PlatformTestTime', new Date().toLocaleDateString());
// initialise our response object so that if we don't load a response
// file then we reflect the content
var responseBody = { headers, method, url, body };
try {
// try reading a response file
fs.readFile('testResponse.json', function(err, data) {
console.log("handling file");
if (err != null) {
if (err.code === 'ENOENT') {
console.log("on return file - will reflect");
} else {
console.log("Read error:" + err.toString());
}
} else {
// a file exists - but is empty?
if ((data != null) && (data.length > 0)) {
// we have a file with content - lets process so it into a JSON
// object
if (Buffer.isBuffer(data)) {
// convert the buffer from hex to an ASCII string
body = data.toString('utf8');
console.log("test response:" + body);
responseBody = JSON.parse(body);
}
}
}
// create an array with our values and then make it
// JSON with stringfy
var output = JSON.stringify(responseBody);
response.write(output);
console.log("Returning:" + output);
response.statusCode = 200;
response.end();
});
} catch (err) {
if (err.code === 'ENOENT') {
console.log("on return file - will reflect");
} else {
console.log(err.toString());
}
var output = JSON.stringify(responseBody);
response.write(output);
console.log("Returning:" + output);
response.statusCode = 200;
response.end();
}
}).listen(8080); // Activates this server, listening on port 8080.
25 Thursday Jan 2018
Posted in APIs, development, General, Technology
Tags
API, API Platform, API-PCS, Groovy, iterations, Oracle, Script, Technology, utility, versions
The new Oracle API Platform makes it possible to deploy different versions of your APIs to different gateway instances. When you you’re managing the Development API Policies through all the different stages of the lifecycle (Design to Production) from a single management tier such a capability is essential. This is further challenged by the fact that each save of you API Definition creates a new iteration (the term used to identify each saved ‘version’ of the API)
However it does lead the challenge from a management perspective of knowing which iterations are running on each Gateway.. you can get the information from the current UI but it requires multiple steps to get the information. The UI also lends itself more to the design processes today than perhaps the more dense information views that a operational report might warrant.
I’m sure that over time these views will come, but today we can solve the problem by taking advantage of the fact that the product lives by its own ‘mission’ by offering a very rich set of APIs. As a result it becomes possible to actually build your own views. To that end I have written a Groovy script which will go through each API that can be seen and retrieves the iteration deployed to each logical gateway.
In terms of running the script you obviously need Groovy installed. It expects 3 parameters which are:
You can hardwire into the script default values which will then be used if no parameters are provided.
Here is a screenshot of some output. I have masked out some information for reasons of security. But there should be enough here to give a sense of what is happening:
The script includes suppressing certificate validation – necessary if you haven’t yet deployed your own specific certificate and still working with the default Oracle certificate.
Feel free to take the script and play with it. I make no claims to it’s elegance etc but I have tried to comment it so you can see what is going on. I have tried to keep the code fairly simple so you can see how it works and processes the JSON responses. The script is available at: https://github.com/mp3monster/Utils/blob/master/getDeployedIterations.groovy
For more about the APIs involved in the script, checkout
30 Saturday Dec 2017
Posted in Cloud, development, General, NodeJS Cloud, Technology
Tags
"Message Push Listener", article, AWS, Cloud, fn, Functions, google, IBM, JMS, Lambda, messaging, OpenWhisk, Oracle, OraWorld, serverless
When I first wrote about Oracle Messaging Cloud we used a service called WebScript.io to make it easy to demonstrate the Message Push Listener. WebScript was essentially what we better know as a Serverless or Functions oriented offering (that is we wrote pieces of code and deployed them without any consideration servers etc). Well as I prepared my demos for Messaging Cloud for the UK Oracle User Group Tech 17 Conference I discovered that WebScript is being shutdown in December 2017.
In the light of this news, I needto provide an alternate implementation for my Message Push Listener demo Google’s Cloud Functions. Before I go into the Google implementation I thought it worth sharing how I landed on Google’s offering.
The Google Cloud Functions is a new service that has been launched with an interesting future. I had hoped to try using project Fn (Oracle’s open source serverless offering) but the cloud offering is not yet publicly available – although you can run Fn on any platform today if you’re prepared to invest in setting up the environment (defeating the point of serverless). I know some of Oracle’s Developer Champions have had a preview so it cant be too far away now. I’m sure when we get a chance to access the new Cloud Native Service announced which will include Fn we will revisit it. Before settling on Google we looked at several other offerings in the serverless space. Whilst this is not an exhaustive analysis it should help give a sense of the challenges and ease of adoption. If you search today on Serverless you’ll most commonly come across Auth0’s WebTask.io, AWS Lambda and IBM OpenWhisk (based on Apache OpenWhisk).
I started with WebTask.io and it was very nearly a done deal, with a nice easy to work with Cloud Development Platform, integrated testing. Extensive support for Node.js and a number of standard frameworks to use with it such as Express available without doing anything.
Other languages are supported as well by WebTask.io. But as I’m trying to create a demo that warrants very little explanation of the Serverless platform we didn’t dig in to this area. Everything went swimmingly until I tried to setup external calls to my function. This became a headache as the security model whilst not overly complex (several ways to provide the REST call with authentication e.g. adding a key in the URI). The process of generating and associating the credentials was far from clear in the documentation.
I moved to look at AWS Lambda, this I just found horribly confusing to get started with. I have heard others saying that getting going isn’t straight forward. So I found myself giving up pretty quickly as the setting up wasn’t that clear. Whilst having used AWS with its IaaS capabilities which is both powerful, flexible and pretty easy to get to grips with if you understand basic ideas like virtual machines this didnt hold true fory Lambdas.
As for OpenWisk, we started to look at it, but getting a 404 error when trying to access the Editor following the IBM documentation didn’t inspire confidence. The was plenty of supoprting documentation which explains how OpenWhisk works.
The Execution framework for OpenWhisk
In addition to the 404, as you can see we have a two step process to execute an action and return a respoinse. However the Message Push Listener initial challenge needs a call and response in a single step. So trying to massage this into a call and response is going to be challenging and a distraction from what we want to be conveying.
This brings us Back to Google, whilst the Cloud IDE is not as elegant or mature as WebTask it was sufficient and the security model wasn’t imposed. I liked the documentation when needed to refer back to it, but to be honest it is pretty intuitive. You can’t fault the docs, to the point Google gave time over to explaining how to manage or avoid incurring costs.
Setting up, was very simple, and then once you’ve choosen your cloud services you get a dashboard like this:
Google provides the idea of projects which allows you to group pieces together – such as related functions. Each project is name space separated. If we then navigate into a Functions project we get a view as follows:
As you can see in the preceeding diagram I created two functions within a project called OMCS. From here you can create more functions in your project or drill into an individual function, as the following view shows:
An individual function provides you with several tabbed views overing the Gernal information (as shown above) or Trigger, Source and Testing. We can see the other views in the following screenshots. The following screen shot shows the Functions Editor, as you can see it is fairly simple – but sufficient to do the job.
Once saved, if valid the code will automatically get deployed, or you can work offline and then upload the code if you want to use a nice editor like Sublime.
with your code edited and saved, then the next step is to invoke it. This can be done with the next tab, or the details such as the URI can be copied and you can test from your preferred test tool such as SOAPUI, Postman and APIFortress.
The testing view allows you 5o define input and output values, along with the outcomes. Personally I worked with SOAPUI.
The important thing with running tests or diagnosing issues, is to be able to examine execution logs. In this area Google Functions is pretty feature rich with a solution that works in a style somewhat like the searching in Splunk (and I’m sure other log analytics tools) where you can drill into the logs and build log filters on the fly. The log view is shown in the next screenshot.
as you can see tool looks pretty straight forward and uncomplicated to use, with freedom to adapt how you work to your preferred style. Based on my experience of using Project FN on my desktop – it is this simplicity I think we’ll see with the Cloud Native Platform from Oracle when it becomes available.
Finally, let’s take a look at the code in Google Functions code produced for this example:
conclusion
Google Code whilst its UI is a bit basic, it is easy to use and get started, certainly for using as a demo platform or perhaps for creating stubs, test and mock end points. Having been critical of the other offerings for security and it getting in the way of a simple illustration it is possible that the Google Functions may need some work in this area. I didn’t see anything that obviously integrated security features in easily.
Just to tie back the impacted articles …
22 Wednesday Nov 2017
Posted in General
My latest contribution to the Oracle Scene journal is available at here. This article looks at the evolution of APIs, and a look at modern API Gateway capabilities. The article uses an analogy to explian the capabilities in a non-techie way.
In addition to my article the team I’m part of get a mention for their wins at this years Partner of the Year awards.
09 Monday Oct 2017
On Friday 29th September, 2 days before the commencement of Oracle’s most important event of the year – OpenWorld whilst attending the Oracle Partner Advisory Council I received word that I had been promoted to a full Oracle Ace.
For those not working in the Oracle ecosystem this is comparable to being confirmed as a Microsoft MVP, a SAP Mentor or Java Champion. These schemes recognize contributions made by non employees to the community and the parent company itself. These contributions range across public speaking, articles for journals, helping through the various community sites and blogging among others. Hoist the accreditation is based on contribution, to be a successful contributor you need to be deeply knowledgable in your specialisms.
The importance of the Ace recognition is important for my employer (Capgemini) and for myself for different reasons. For an employer the association of expertise can be a key value propositions, and some Oracle partners actually use the number of Aces they employee as a key part of their differentiator and market proposition. Secondly, being out communicating with the community raises brand awareness increasing the chances of both sales but also make the company more attractive as a potential employer. Finally, through participating with in events you get to know product managers and other scenario Oracle people. As a result, when additional support and engagement is needed you have the contacts to draw on. But is not just help, the opportunity to contribute to product development exists. In many respects this can become a virtuous circle – the more you do the more opportunities open up, the more you can do.
For me personally the Ace programme is a very friendly embracing community that whilst can be commercially competitive is very mutually supportive. This combined with the fact that the culture of sharing knowledge is actively encouraged, supported and acknowledgement of those efforts is always satisfying.
In the middleware space there are less than 50 active Oracle Aces of all grades globally. Four of those are in the UK Luis Weir (Ace Director – Capgemini), Simon Haslam (Ace Director – eProseed), Mark Simpson (Ace Director – Griffiths Waite) and myself. I am also fortunate enough to count all three as friends.
05 Thursday Oct 2017
Posted in API Platform CS, General, ICS, Oracle
Tags
API, API Platform, APIP, conference, ICS, OOW, OOW17, Oracle, presentations
With Oracle Open World 2017 over the ICS presentation is available at – Oracle integration cloud service (ICS) best practices learned from the field (OOW17)
We saw a lot of exciting new features and capabilities coming from Oracle in the ICS space. So keep an eye on the site as we publish new articles.
The API Platform presentation that was co-presented with Luis Weir is here…
12 Wednesday Jul 2017
Posted in chatbots
I have been fortunate to be supporting and occasionally contributing to a series of blogs on Chatbots being written by Leon Smiers, Capgemini Oracle Chatbot SME (and also Oracle Ace). The blog posts are:
I’ve been talking with Leon about what next in the blog series, and we can expect to see some more exciting blog posts in the series.
Chatbot Maturity Model
