Tag Archives: API Platform

API Security

01 Saturday Jun 2019

Posted by in API Platform CS, APIs & microservices, General, Oracle, Technology

≈ Leave a comment

Tags

, , , , , ,

I’ve started to subscribe to the APISecurity.io newsletter. The newsletter includes the analysis of recent API based security breaches along with other useful API related news. Some of the details of the breaches make for interesting reading and provide some good examples of what not to do. It is rather surprising how regularly the lack of the application of good practises is, including:

  • Checking the payload is valid to the definition,
  • Checking the payload size to ensure it is in the expected bounds,
  • Use strong typing on the content received it will help validate the content and limit the chances of poisonous content like injected SQL,
  • owaspEnsuring the API has mitigation’s against the classic OWASP Top 10 – SQL Injection, poor authentication implementation.

More broadly, we see that people will recognise the need for applying penetration testing, and look to external organisations to perform the testing, when such work is commissioned the understanding of what the pen tester does is not understood by those logocommissioning the tests (SANS paper of security scoping), therefore know whether all the risks are checked. When you add to that, the temptation to keep such costs down resulting in the service provider not necessarily probing your APIs to the fullest extent. Not all penetration test services are equal, so simply working to a budget isn’t wise, yes there is a need for pragmatism, but only when you understand the cost/risk trade-off.

But also remember application logic and API definitions and the security controls in place change over time as do the discovery of new vulnerabilities on the stack you’re using, along with evolving compliance requirements. All meaning that a penetration test at the initial go-live is not enough and should be an inherent part of an APIs lifecycle.

cloudgs_apimgrWhen it comes to payload checks etc, products like Oracle’s API Platform make it easy to realise or provide out of the box checks for factors such as size limits, implementing payload checks, so better to use them.

If you ever need to be reminded that of why best practises are needed and should be implemented; a mindset of when not if a breach will happen will ensure you’re prepared and the teams are motivated to put the good practises in.

Making Scripts Work with IDCS Deployed PaaS

09 Saturday Feb 2019

Posted by in API Platform CS, APIs & microservices, General, Oracle, Technology, tools

≈ 5 Comments

Tags

, , , , , ,

A while back I made some utilities I developed to help with managing the API Platform. At the time we didn’t have access to an IDCS based environment, so credentials worked using basic auth (I.e. user name and password). But with environments managed by IDCS tokens are used.

As a developer with a Java background I have to admit to liking Groovy over Python for scripting, not to mention for the API Platform groovy is part of the gateway deployment and SDK. Meaning it is readily available in its 2.x form (3.x is relatively recent and aligning to the latest Java idioms). We haven’t tested against Groovy 3.0

Thank you to Andy Knight for sharing with us some Java code which I adapted to be pure Groovy (removing external dependencies for processing JSON). The result is a script that can be taken can be taken and worked into other scripts (which is what we have done for our previously provided scripts – Understanding API Deployment State on API Platform, Managing API Policy Versioning in Oracle API Platform, Documenting APIs on the Oracle API Platform). But this script can be used to get a token and display it on the command line or write it to file. Writing tokens to files is generally not good practise, but as a temporary measure when working on developing scripts arguably a managed risk.

The script can be found at https://github.com/mp3monster/API-Platform-Utils/tree/master/getToken and all the details on using the script can be obtained by passing -h as a parameter. The important thing is to understand how to obtain the Client ID and Client Secret, the details of which are described at https://docs.oracle.com/en/cloud/paas/api-platform-cloud/apfad/find-your-client-id-and-client-secret.html

Defining Boundaries for Logical Gateways on the API Platform a multi cloud / multi region context

31 Wednesday Oct 2018

Posted by in API Platform CS, General, Oracle, Technology

≈ 3 Comments

Tags

, , , ,

The Oracle API Platform takes a different licensing model to many platforms, rather than on CPU it works by the use of Logical Gateways and blocks of 25 million successful API calls per month. This means you can have as many actual gateway nodes as you like within a logical group to ensure resilience as you like, essentially how widely you deploy the gateways is more of a maintenance consideration (i.e. more nodes means more gateways to take through a maintenance process from the OS through to the gateway itself).

In our book (here) we described the use of logical gateways (groups of gateway nodes operating together) based on the classic development model, which provides a solid foundation and can leverage the gateway based routing policy very effectively.

logical partitions

But, things get a little trickier if you move into the cloud and elect to distribute the back end services geographically rather than perhaps have a single global instance for the back-end implementation and leverage technologies such as Content Delivery Networks to cache data at the cloud edge and their rapid routing capabilities to offset performance factors.

map1

Classic Global split of geographies

Some of the typical reasons for geographically distributing solutions are …

  • The low hit rate on data meaning caching solutions like CDNs are unlikely to yield performance benefits wanted and considerable additional work is needed to ‘warm’ the cache,
  • Different regions require different back end implementations ordering of products in one part of the world may be fulfilled using a partner, but in another, it is directly satisfied,
  • Data is subject to residency/sovereignty rules – consider China for example. But Germany and India also have special considerations as well.

So our Global splits start to look like:

map2

Global Split now adding extra divisions for India, China, Russia etc

The challenge that comes, is that the regional routing which may be resolved on the Internet side of things through Geo Routing such as the facilities provided by AWS Route53 and Oracle’s Dyn DNS as a result finding nearest local gateway. However Geo DNS may not be achievable internally (certainly not for AWS), as a result, routing to the nearest local back-end needs to be handled by the gateway. Gateway based routing can solve the problem based on logical gateways – so if we logically group gateways regionally then that works. But, this then conflicts with the use of gateway based routing for separation of Development, Test etc.

Routing Options

So, what are the options? Here are a few …

  • Make you Logical divisions both by the environment and by region – this is fine if you’re processing very high volumes i.e. hundreds of millions or more so the cost of additional Logical gateways is relatively small it the total budget.
map3

Taking the geo split and applying the traditional layers as well has increased the number of Logical gateways

This problem can be further exacerbated, if you consider many larger organisations are likely to end up with different cloud vendors in the same part of the world, for example, AWS and Azure, or Oracle and Google. So continuing the segmentation can become an expensive challenge as the following view helps show:

map4

It is possible to contract things slightly by only have development and test cloud services where ever your core development centre is based. Note that in the previous and next diagrams we’ve removed the region/country-specific gateway drivers.

map5

  • Don’t segment based on environment, but only on the region – but then how do you control changes in the API configuration so they don’t propagate immediately into production?
  • Keep the existing model but clone APIs for each region – certainly the tooling we’ve shared (Managing API Policy Versioning in Oracle API Platform) makes this possible, but it’s pretty inelegant and error-prone as it be easy to forget to clone a change, and the cloning logic needs to be extended to take into account the bits that must be region-specific.
  • Assuming you have a DNS address for the target, you could effectively rewrite the resolution of the address by changing its meaning in each gateway node’s host file. Inelegant, but effective if you have automated deployment and configuration of your gateway servers.
  • Header based routing with the region and environment as header attributes. This does require either the client to set the values (not good as you’re revealing to your API consumer traits of the implementation), or you apply custom policies before the header-based routing that insert those attributes based on the gateway’s location etc.
  • Build a new type of gateway based routing which allows both the environment (dev, test etc) and location (region) to inform the routing,

Or, and the point of this blog, use gateway based routing and leverage some intelligent DNS naming and how the API Platform works with a little bit of Groovy or a custom Java policy.

Continue reading →

Documenting APIs on the Oracle API Platform

21 Monday May 2018

Posted by in API Platform CS, General, Oracle, Technology, tools

≈ 2 Comments

Tags

, , , , , 

Updated reflecting changes discussed in blog post:
Making Scripts Work with IDCS Deployed PaaS

The last week or two I have been working on a new API Platform utility to add to my existing tools (see here). This tool addresses the question of generating documentation.  Much as been said about API documentation and the quality of it, check out these articles :

If you look at these articles and others, there are some common themes, which are:

  • Document the URI / payload
  • Describe error handling
  • Describe contracts such as how many API calls
  • How the API is authenticated

Apiary covers the first theme to a first class standard,  and you will see Apiary called out for its ability to document APIs in a lot of articles. Well written API Blueprints will cover the bulk of the second bullet. But the other points tend to fall outside of a Blueprint and fit more the API Policies and their use.

Not everyone is so commited or enjoys writing documentation. The other driver for going beyond the use of Apiary is that some organizations feel the need to have a traditional word style document to capture/define an API’s contract in detail. With the API Platform the management portal enables an API to be published into the developer portal with the Apiary definition and a markdown file for further documentation.

Continue reading →

Managing API Policy Versioning in Oracle API Platform

27 Tuesday Feb 2018

Posted by in API Platform CS, APIs & microservices, Books, General, Oracle, Technology

≈ 6 Comments

Tags

, , , , , , , , , 

Updated reflecting changes discussed in blog post:
 Making Scripts Work with IDCS Deployed PaaS

Oracle’s API Platform (API-P) product avoids the use of external configuration management. If you want to better understand why, then checkout our forthcoming book as it goes into detail about why this is the case (it can be pre-release version of the book can be obtained here). In a previous blog I wrote about and illustrated the use of the API-P’s own APIs so that it was possible to see what API iterations had been deployed to API Gateways.

In this blog I want to explore the issues of version management a bit further. API-P provides internal version management through the idea of iterations as previously explored (Understanding API Deployment State on API Platform). In addition to this there are API policy attributes called version, status etc. This information whilst having some impact on behaviour reflects the version of the ‘contract’ that the API represents between the consumer and provider, and requires a manual change.

The API policies themselves are version tracked through the iteration identifiers. Each time a policy is saved the iteration is is incremented. What the API-P doesn’t support is the concept of branching. In relatively simple API Policy branching is unlikely to ever be an issue.

Why is a reversion capability needed?

Let’s take a more complex scenario.  In our book API Platform we introduced the some APIs that would allow the retrieval of meta data about artists in the record companies’ catalog. It has however come to light that the API has been targeted with malicious calls, firstly through trying to attack using injection attacks and secondly trying to overload the back end by creating data requests that make the back-end work hard in retrieving data.

To defend against this, the API Policy has been enhanced to include some custom groovy policies to inspect the values provided. Strictly speaking following the principles of Semantic Versioning the API version should go from 1.0.0 to 1.0.1. However seeing that the ‘contract’ as presented to the consumer hasn’t really changed – the data models are the same, the URI goes unaltered resulting in the implementation team not changing the version.

During development processes, it is not unusual to be developing existing logic, and decide that approach being used isn’t right or not going to perform as well. So you abandon your changes and revert back to the last approved version. However, this isn’t possible as any save will result in a new iteration. The API-P will be getting some enhanced version management features. But today to be able to undo the changes we need a means to ‘revert forward‘ (hence the tool name) that is to take an older iteration and make it the latest, as illustrated in the next diagram.

Today the API-P doesn’t provide a means to perform this process within the User interface. However when looking at a gateway you can review the API policy deployed. As we established previously that you may have different gateways deployed with different iterations. Given this, as API-P has been built true to the principles of separating the UI from the back-end through the use of APIs we can deduce there should be a means to get the details of an API with a specific iteration.

To this end we have built on the pattern previously illustrated to provide the means to ‘Revert Forward’ by creating a Groovy script that will use the APIs provided by API-P to retrieve an iteration and push it back at the latest version. When the policy is pushed back it also modifies the description to show which iteration has been pushed.

You may ask, why not use the conventional developer approach of branching as suggested in the following diagram. However API-P’s iteration framework doesn’t extend to support this.

The next with this is pretty predictable – how do I know which iteration to revert to. You have two options here, firstly either revert in order so you can see the prior version in turn – which whilst visually good, is not necessarily the most practical option. So in the tool we have a parameter that will allow you to display on the console the configuration of each iteration. This does mean you are going to see the policies in a JSON presentation. To make life easier we would recommend good practise and recording in the policy description information that helps determine the policy’s characteristics – and this can be used to better determine iteration behaviour.

If we are able to take an earlier iteration and make it the latest one by pushing it back then it is a short step to actually target a different management cloud in effect migrating the policies. Whilst possible it comes with some serious cautions …

  • You risk undermining your version management, which management cloud has the master, and the iteration numbers will NOT migrate so it’s not like this info can be used to distinguish the laster version
  • The logic included doesn’t accommodate handling differences in policy versions – so if trying move between instances of the API Platform they need to be the same version otherwise your configuration could make a mess of thing
  • This issue is further compounded if you are deploying custom Java policies.
  • Environment specific policies simply won’t work for example gateway based routing.

Oracle does not recommend that the policies be stored anywhere outside of the platform, whilst it this utility makes that a possibility, it deliberately avoids writing any of the information to the file, the policies only reside outside of the platform for the duration of the process execution.

The Tools Commands

All the parameters assume the values will not contain any space characters. Each command is preceded by a dash eg. revertForward.groovy -inpassword mypass -inSvr https://a.b.com

  • -h or -help – provides this information
  • -inName – user name to access the source management cloud
  • -inPass – password for the source management cloud
  • -inSvr – The server address without any attributes e.g. https://1.2.3.4
  • -policy – numeric identifier for the policy of interest
  • -iter – iteration number of interest for the policy – optional
  • -outName – optional, the target management cloud username, only needed for migrations
  • -outPass – optional, the target management cloud password, only needed for migrations
  • -outSvr – optional, the target management cloud server address – same formatting as inSvr, only needed for migrations
  • -override – optional, if migrating to another management, tells the script to replace the existing policy of the same name if found
  • -view – optional, separate command to allow viewing of the policy – requires one of the following values:
    • display – displays all the details of the policy, if no iteration is provided this will be the latest iteration
    • summary – provides the headline information of the policy including name, change date etc
    • summary-all – summarizes all the iterations from the current one back to the 1st
  • -debug – optional, will get script to report more information about what is happening

The code can be obtained from my GitHub repository here.

Tracing Executions in an API Environment

07 Wednesday Feb 2018

Posted by in API Platform CS, APIs & microservices, General, Technology

≈ 2 Comments

Tags

, , , , , , ,

As APIs become more pervasive within our solutions we see the arrival of not just design and cataloging tools such as Apiary, Apigee and others but also the arrival of gateways. The gateways provide execution of operations including validation, accounting (moneytization), routing, and other controls such as throttled checks that would often not occur until the first contact with a service bus. For example initial routing based on the API call, fine grained authentication and authorisation (differing from your firewalls who will just perhaps authorise).

In the more traditional integration middleware of Oracle Service Bus and SOA (regardless of cloud, on-premises) you can trace the execution through the middleware end to end. This tracing can be achieved because the platform creates and assigns the a UUID (aka eCID) and ensures that it is carried through the middleware. It is this very behaviour that allows Oracle to provide Business Activity Monitoring without any need to be invasive. Burt not only that in a highly distyributed environment you can track the processing of a transaction from end to end.

The challenge now is that the first point of middleware style behaviour can be at the gateway. So we actually need to move forward the UUID or equivalent forward the the first point of contact. Not only that we need to think about the fact we will see non Oracle integration middleware involved.  Within Spring, Kabana and other established frameworks and tools which are getting signficiant traction with the rise of Microservices, the Ids being used are not the same as the UUIDs used by Oracle. For example Spring Cloud Slueth uses the same HTTP header Ids that Zipkin and Kabana support:

  • X-B3-TraceId
  • X-B3-SpanId
  • X-B3-ParentSpanId

More information can be found here and here.

For the new Oracle API Platform Cloud Service we can check for the existance of the header attributes as a policy and apply actions such as:

  • Apply a header with a TraceId or spanId,
  • If a SpanId exists then we may wish to nest our call as a child span by moving the SpandId to the ParentSpanId and creating a new SpanId.

Ultimately it would be more attractive to apply the logic using the API Platform’s SDK but to get things rolling applying the IDs with the API Groovy policy is sufficient (more here).

Next question that begs, is where to put the ID and want to call it? Put the value in the body, and you’re invading the business aspect of an API with execution specific details, not to mention potentially  changing the API definition. Whilst stuffing HTTP(s) headers with custom attributes is often discouraged as the values aren’t to immediately visible. In my opinion at least the answer largely has been set by president for us. If you weren’t using HTTPS but JMS then you would use the header, but also a number of frameworks already exist that do make use of this strategy such as those mentioned above.

Using the header defined by Kabana etc means that the very process will mean that a number of Support tools out of the box will understand and be able to help visualise your logs with no additional effort.

The following is a Groovy script that could ber used for the purposes of applying an Id appropriately into the HTTP header in the API Platform:

if (!context.getApiRequest().getHeaders().containsKey ("baggage-UUID "))
{
  // use current time to seed random generator
  def now = java.time.Instant.now()
  def random = new java.util.Random(now.getEpochSecond())

  // create an array of 16 bytes to hold the random value
  byte[] uid = new byte[16]
  random.nextBytes(uid)

  // convert the random string from Hex to an ASCII string
  Writable uidInHex= uid.encodeHex()
  String uidStr= uidInHex.toString() 

  // set the outbound header
  context.getServiceRequest().setHeader("baggage-UUID", uidStr)
}

Validating API Platform Policies & Gateway Deployments

01 Thursday Feb 2018

Posted by in API Platform CS, APIs & microservices, General, Oracle, Technology

≈ 3 Comments

Tags

, , , , , , ,

When configuring API Policies in the the Oracle API Platform it helps if there is a simple back end that can take the received payload and record the sent values (header & body) as well as reflect the call details back as the response, or possibly respond with a test payload (so that response policies, particularly policies that require payload navigation  can be exercised correctly).  By having this facility it becomes a lot easier to determine whether the policies are executing correctly in terms of routing, transforming, filtering etc. without needing to worry about whether the API implementation is correct. You could say that this is a kind of mock for testing the API Platform.

The added benefit of having a mock back end is that it is easy to ‘smoke test’ a gateway deployment very easily.  Particularly if the mock is happy to receive any form of call.

Whilst implementing such a capability can be done in pretty much any language and platform you like.  We have in the past for example built a Springboot Java application that can have the dependencies configured to then deploy into WebLogic for example.  We have come to refer these test apps/mocks as PlatformTests as that’s exactly what they help do. A Node.js implementation of a PlatformTest such as as the following implementation is particularly appealing as the Node.js footprint is small and simple to deploy and undeploy. A basic Node.js implementation can also consume any URL and operation you choose to use. The nature of JavaScript makes it very quick to adapt the mock if need be. Although in the ideal world, we write the solution once and then use simple configuration to tune behavior.

The following code looks for a local file called testResponse.json if found then returns the content of the file (assumed to be JSON) otherwise it reflects back in the body, the received header and body.  This reflection makes it extremely easy to see how the policies have changed the inbound call.  The content is also logged to the console – making it easy to also see what came through to the back end.

The implementation also assumes port 8080, but changing the port is exceptionally easy.

There one enhancement planned, and this is to allow the response test payload to be handled as XML.  This will need a little tweaking of the code as presently a JSON Object is currently stringified.

JavaScriptThe code is also available in my GitHub repository – https://github.com/mp3monster/Utils/blob/master/PlatformTest.js and an example test response file is at https://github.com/mp3monster/Utils/blob/master/testResponse.json

const http = require('http');
const fs = require('fs');

// create a simple HTTP server that will handle the requests
http.createServer((request, response) => {
const { headers, method, url } = request;
console.log("Called at " + new Date().toLocaleDateString());
let body = [];
request.on('error', (err) => {
console.log("Svr Error Handler :" + err.toString);
response.statusCode(400);
response.end();
}).on('data', (chunk) => {
body.push(chunk);
}).on('end', () => {
body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.

});

// record in the console what details have been received
console.log ("Received:\nMethod:" + method.toString() +
"\n URL:"+ url.toString + "\nheaders:\n"+headers.toString() +
"\nBody:\n" + body);
// now build the response
response.setHeader('Content-Type', 'application/json');
response.setHeader('PlatformTestTime', new Date().toLocaleDateString());

// initialise our response object so that if we don't load a response
// file then we reflect the content
var responseBody = { headers, method, url, body };

try {
// try reading a response file
fs.readFile('testResponse.json', function(err, data) {
console.log("handling file");
if (err != null) {
if (err.code === 'ENOENT') {
console.log("on return file - will reflect");
} else {
console.log("Read error:" + err.toString());
}
} else {
// a file exists - but is empty?
if ((data != null) && (data.length > 0)) {
// we have a file with content - lets process so it into a JSON
// object
if (Buffer.isBuffer(data)) {
// convert the buffer from hex to an ASCII string
body = data.toString('utf8');
console.log("test response:" + body);
responseBody = JSON.parse(body);
}
}
}

// create an array with our values and then make it

// JSON with stringfy

var output = JSON.stringify(responseBody);
response.write(output);
console.log("Returning:" + output);
response.statusCode = 200;
response.end();

});

} catch (err) {

if (err.code === 'ENOENT') {
console.log("on return file - will reflect");
} else {
console.log(err.toString());
}
var output = JSON.stringify(responseBody);
response.write(output);
console.log("Returning:" + output);
response.statusCode = 200;
response.end();
}
}).listen(8080); // Activates this server, listening on port 8080.

Understanding API Deployment State on API Platform

25 Thursday Jan 2018

Posted by in API Platform CS, APIs & microservices, development, General, Oracle, Technology, tools

≈ 6 Comments

Tags

, , , , , , , , ,

The new Oracle API Platform makes it possible to deploy different versions of your APIs to different gateway instances. When you you’re managing the Development API Policies through all the different stages of the lifecycle (Design to Production) from a single management tier such a capability is essential. This is further challenged by the fact that each save of you API Definition creates a new iteration (the term used to identify each saved ‘version’ of the API)

However it does lead the challenge from a management perspective of knowing which iterations are running on each Gateway.. you can get the information from the current UI but it requires multiple steps to get the information. The UI also lends itself more to the design processes today than perhaps the more dense information views that a operational report might warrant.

I’m sure that over time these views will come, but today we can solve the problem by taking advantage of the fact that the product lives by its own ‘mission’ by offering a very rich set of APIs. As a result it becomes possible to actually build your own views. To that end I have written a Groovy script which will go through each API that can be seen and retrieves the iteration deployed to each logical gateway.

In terms of running the script you obviously need Groovy installed. It expects 3 parameters which are:

  • Server address e.g. https://1.2.3.4
  • Username e.g. weblogic
  • Password e.g. Welcome1

You can hardwire into the script default values which will then be used if no parameters are provided.

Here is a screenshot of some output.  I have masked out some information for reasons of security. But there should be enough here to give a sense of what is happening:

APIPlatformScript

The script includes suppressing certificate validation – necessary if you haven’t yet deployed your own specific certificate and still working with the default Oracle certificate.

Feel free to take the script and play with it. I make no claims to it’s elegance etc but I have tried to comment it so you can see what is going on. I have tried to keep the code fairly simple so you can see how it works and processes the JSON responses. The script is available at: https://github.com/mp3monster/Utils/blob/master/getDeployedIterations.groovy

For more about the APIs involved in the script, checkout

Presentations from Oracle Open World

05 Thursday Oct 2017

Posted by in API Platform CS, General, OIC - ICS, Oracle

≈ Leave a comment

Tags

, , , , , , , ,

With Oracle Open World 2017 over the ICS presentation is available at – Oracle integration cloud service (ICS) best practices learned from the field (OOW17)

We saw a lot of exciting new features and capabilities coming from Oracle in the ICS space. So keep an eye on the site as we publish new articles.

The API Platform presentation that was co-presented with Luis Weir is here…