Monthly Archives: August 2019

Handling Socket connectivity with API Gateway

30 Friday Aug 2019

Posted by in API Platform CS, General, Oracle, Technology

≈ Leave a comment

Tags

, , , , , ,

At the time of writing the Oracle API Platform doesn’t support the use of Socket connections for handling API data flows. Whilst the API Platform does provide an SDK as we’ve described in other blogs and our book it doesn’t allow the extension of how connectivity is managed.

The use of API Gateways and socket-based connectivity is something that can engender a fair bit of debate – on the one hand, when a client is handling a large volume of data, or expects data updates, but doesn’t want to poll or utilize webhooks then a socket strategy will make sense. Think of an app wanting to listen to a Kafka topic. Conversely, API gateways are meant to be relatively lightweight components and not intended to deal with a single call to result in massive latency as the back-end produces or waits to forward on data as this is very resource-intensive and inefficient. However, a socket-based data transmission should be subject to the same kinds of security controls, and home brewing security solutions from scratch are generally not the best idea as you become responsible for the continual re-verification of the code being secure and handling dependency patching and mitigating vulnerabilities in other areas.

So how can we solve this?

As a general rule of thumb, web sockets are our least preferred way of driving connectivity, aside from the resource demand, it is a fairly fragile approach as connections are subject to the vagaries of network connections, which can drop etc. It can be difficult to manage state (i.e. knowing what data has or hasn’t reached the socket consumer). But sometimes, it just is the right answer. Therefore we have developed the following pattern as the following diagram illustrates.

API Protected Sockets

How it works …

The client initiates things by contacting the gateway to request a socket, with the details of the data wanted to flow through the socket. This can then be validated as both a legitimate request or (API Tokens, OAuth etc) and that the requester can have the data wanted via analyzing the request metadata.

The gateway works in conjunction with a service component and will if approved acquire a URI from the socket manager component. This component will provide a URL for the client to use for the socket request. The URL is a randomly generated string. This means that port scans of the exposed web service are going to be difficult. These URLs are handled in a cache, which ideally has a TTL (Time To Live). By using Something like Redis with its native TTL capabilities means that we can expire the URL if not used.

With the provided URL we could further harden the security by associating with it a second token.

Having received the response by the client, it can then establish the socket-based connection which gets routed around the API Gateway to the Socket component. This then takes the randomly-generated part of the URL and looks up the value in the cache, if it exists in the cache and the secondary token matches then the request for the socket is legitimate. With the socket connection having been accepted the logic that will feed the socket can commence execution.

If the request is some form of malicious intent such as a scan, probe or brute force attempt to call the URL then the attempt should fail because …

  • If the socket URL has never existed in or has been expired from the Cache and the request is rejected.
  • If a genuine URL is obtained, then the secondary key must correctly verify. If incorrect again the request is rejected.
  • Ironically, any malicious attack seeking to overload components is most likely to affect the cache and if this fails, then a brute access tempt gets harder as the persistence of all keys will be lost i.e. nothing to try brute force locate.

You could of course craft in more security checks such as IP whitelisting etc, but every-time this is done the socket service gets ever more complex, and we take on more of the capabilities expected from the API Gateway and aside from deploying a cache, we’ve not built much more than a simple service that creates some random strings and caches them, combined with a cache query and a comparison. All the hard security work is delegated to the gateway during the handshake request.

Thanks to James Neate and Adrian Lowe for kicking around the requirement and arriving at this approach with us.

 

Costs in Multi-Cloud

28 Wednesday Aug 2019

Posted by in General, Oracle, Technology

≈ Leave a comment

Tags

, , , , , , , ,

Over the last couple of years, we have seen growing references to multi-cloud. That is to say, people are recognizing that organisations, particularly larger ones are ending up with cloud services for many different vendors. This at least in part has come from where departments within an organization can buy meaningful resources within their local budgets.

Whilst there is a competitive benefit of the recent partnership agreement between Microsoft and Oracle given the market margin AWS has in comparison to everyone else. Irrespective of the positioning with AWS, this agreement has arisen because of the adoption of multi-cloud. It also provides a solution to the problem of running highly resilient Oracle database setups using RAC, DataGuard etc can be made available to Azure without risk to security and the all-important network performances that are essentially to DB operation. Likewise, Oracle’s SaaS offerings are sector leaders if not best in place, something Microsoft can’t compete with. But at the other end, regardless of Oracle’s offerings, often organisations will prefer Microsoft development ecosystem because of the alignment to office tooling, the ease of building solutions quickly.

Multi-cloud even with the agreements like the Microsoft and Oracle one (See here), doesn’t mean there won’t be higher costs in crossing clouds. Let’s see where the costs reside …

  • Data egress (and in some cases ingress as well) from clouds costs. Whilst the ingress costs have been eliminated because it can be seen as a barrier to selling services, particularly big data. Data egress can, however, be an issue. Oracle has made this cost very low to be almost negligible, but not necessarily others as the following comparison shows …
  • Establishing the high-performance connections That the agreement supports needed between Azure and Oracle cloud is the same tech for the cloud to the ground do incur a cost. In Oracle’s case, there is a fee for the connection (not a large cost, but one that exists none the less) plus any traffic fees the provider of the network connection spanning the data centre locations. This is because you’re leasing capacity on someone’s dedicated fibre or MPLS services. Now, this should prove to be small as part of the enabler of this offering is that both Oracle and Microsoft cloud DCs are often actually physically provided by the same provider or at-least the centres a physically pretty close, as a result of both companies gravitating to locations close together because of the optimal highly available infrastructure (power, telecommunications) legal and commercial factors along with the specialist skills needed.

If data egress is the key challenge to costs, what drives the data egress beyond the obvious content for user interfaces? …

  • Obviously, you have the business data flows, some of these flows will be understood by the business community. But not all, this is down to the way data from the cloud can be exposed to another. For example inefficient services with APIs that require frequent polling and not using expressing the request efficiently, rather than perhaps express the request using HTTP header attributes and other efficiencies or even utilize frameworks such as webhooks so data can be pushed.
  • High-speed data access, often drives data replication having databases in multiple clouds with mirror image data in each location even if the majority of the data is not necessarily needed. This can happen with technologies such as Kafka which for non compacted topics will mean every event can be replicated even if that event has a short lifetime.
  • One of the hidden costs is the operational tasks of gathering logs to a combined view so end to end insights can be obtained. A detailed log can actually yield more ‘data’ by volume than the business flows themselves because it is semi-structured, and intended to be very readable and at the most granular level there to help debug and test.

In addition to the data flows, you need to consider how other linkages in addition to the Oracle-Azure connection are involved. In the detailed documentation, it is not possible to get your on-premises location connected to one of the clouds (e.g. Oracle FastConnect, and then assume your traffic can hop to Azure via the bridge using FastConnect and Azure’s ExpressRoute.  To add performance to your solution parts in both Azure and Oracle Cloud, you still need to have FastConnect and ExpressRoute configured to your on-premises location. This, of course, may impact how bulk data for lift and shift app use cases such as EBS may be applied. For example, if you choose to regularly bulk data transfer between on-premise and EBS via the app/middleware tier rather than via direct DB, and that mid-tier is running in Azure – you will need both routes established.

Conclusion

There is no doubt that the Oracle-Azure cloud to cloud linkage is a step forward, but ‘the devil is in the details‘ as the saying goes. To optimize the benefits and savings we’d suggest that you;

  • you’ll need to think through your use cases – understand data flow and volume (someone bulk syncing application data with a data warehouse?),
  • define a cloud data strategy – to layout principles, approaches and identify compliance needs, this is particularly helpful for custom solution development, so the right level of log data is consolidated with the important details, data retention addresses compliance requirements and doesn’t ratchet up unnecessary costs (there is a tendency to horde data just in case – if this is really wanted, think about how its stored),
  • based on business common usage models define a simple forecasting formula – being able to quantify data costs will always make it easier to challenge back data hoarding tendency,
  • confirm the inter-cloud network vendor charges when working with multi-cloud.

Why do I have an Ace logo on my website?

19 Monday Aug 2019

Posted by in General, Oracle

≈ Leave a comment

Tags

, , , ,

For the observant, you’ll have noticed that I have a logo on the left side of my site saying Oracle Ace. Periodically I get asked what is it, what does it mean, and for those who are less involved in the Oracle community probably don’t know what it means.

What

Most developers will probably have encountered the idea of Java Champions or perhaps Microsoft MVPs (Most Valued Professional). All of these badges and other large vendors such as SAP have comparable ones are a recognition of individuals outside of the organisation (in this case Oracle) who do a lot to support the community and wider technology ecosystem.

These contributions vary but typically take the form activities such as writing blogs/articles/books, answering questions on StackOverflow and other community sites where questions are raised and answered by experts. Organising and/or presenting at conferences.

This is the content helps bridge the gap between the standard guidance, documentation, white papers that the vendors will produce and real-world practical experience.

Whilst you, in theory, you don’t have to be an expert to be part of these advocacy programmes, the reality is to communicate the meaningful value you need to have a level of experience and understanding that is more than the majority. I know a number of people in the Ace community who would deny being experts, and the only thing that differentiates them from everyone else is being willing to stand up and share what they have learnt. I would say that inevitably they are experts, as the processes and resource (at least for Aces inevitably enable that development of expertise as I will try to illustrate shortly).

But before we progress, let me quickly summarise the advocacy communities that Oracle support …

  • 9928f94d6fb7bc0024781fa68e0bc571_400x400Java Champions – these are people working in the pure Java ecosystem
  • Oracle Ace’s – within this community we have three tiers of Ace and which tier reflects the amount of time actively involved in the Ace programme and how much you contribute. So you start out as …
    • Ace Associate progressing to
    • ace-logoAce then a at the top are a smaller community of
    • Ace Directors

Ace’s generally focus on Oracle’s mainstream products from the database to Middleware like WebLogic and Apps

  • obga_badgeThe final group are Groundbreaker Ambassadors – this group are comparable to Ace Directors and actually progress through the Associate and Ace accreditation. But rather than focus on more traditional Oracle offerings this group tend to work with what could be described as modern app dev tech from Microservices and APIs to Blockchain.

Why?

Why get involved in such a community? Whilst I can only speak for myself, I suspect some of my motivators hold true for others, for me, it’s about…

  • The is a strong sense of community amongst the Aces and obviously an inbuilt common interest. Given we often encounter each other at conferences etc, it makes it a lot easier socially when attending conferences. Stuck for a coffee conversation? Go say hello to someone you already know.
  • The value in knowledge and experience is in the sharing of that information, and you can’t beat the sense of validation when someone says – thank you, that really helped me.
  • Talking to other Aces means you may pick up useful insights. Certainly, the Ace community are encouraged to develop relationships with Oracle product management (to be nominated for Ace Director / Groundbreaker Ambassador you need the sponsorship of a product manager).
  • These insights will further your knowledge which makes the day job easier. It becomes easier to influence Oracle when it comes to having features or priorities set that are of interest.
  • Some employers and customers put a value on the Ace recognition as
    • there is the implicit expertise
    • gives indirect channels to product management
    • track record of sharing and enabling others
  • …so it creates some extra career opportunities or a foot up. If you look at eProseed’s website you’ll see that they are very proud to employ a lot of Ace Directors.

Expertise

Coming back to the point of expertise, as you develop within the community, the chances of learning from others increases, but developing relationships with product management means getting to hear about what’s next etc as well as getting to hear the product managers and their thinking. In fact, Ace Directors and Groundbreakers have dedicated briefing sessions and additional access that provides further insight into the product, strategy and direction. These relationships can start to create a virtuous circle of knowledge accumulation.

Biased?

Carrying the badge of a vendor, and obviously contributing to a vendor’s community carries the risk of being perceived as not being independent/impartial or perhaps understanding the wider landscape. But having been part of the community, this is deeply inaccurate. The community members I know take pride in being professional which usually means being clearly impartial and appreciating the wider IT landscape in which they specialize. Being an Ace doesn’t mean you only know Oracle products, many Ace’s in the integration and development space are often also certified on the Azure or AWS platforms for example. What you won’t find is an Ace publicly calling Oracle out, but then with the access afforded/acquired into the organisation means where there are concerns/challenges/issues they are communicated through the relationships developed and this input appears to be taken very seriously.

Benefits

When it comes to benefits, there are some, but I wouldn’t want people to think that it will ‘pay’ for the level of effort put in. The benefits are very much in the realm of acknowledgement for the contributions made. So yes, we get a few goodies – nice polo shirt with the community logo and the alike. Engraved glassware acknowledging your progress to Ace. The real reward for me is the community and having opportunities to share insights and a bit of acknowledgement of the effort invested, everything else is a bonus.