• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Mindmaps Index
    • Oracle Integration Site
    • Useful Tech Resources …
      • Oracle Tech Resources inc Open Source
      • Useful Tech Resources
      • Python Setup & related stuff
  • Music

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: API

API Gateway for data egress

11 Wednesday Jan 2023

Posted by mp3monster in Cloud, General, Technology

≈ Leave a comment

Tags

API, architecture, gateway, Security

Most larger organizations route their outbound web traffic through a web proxy. The primary motivation for this is to measure where traffic is going. Log traffic for analysis to try and detect activities trying to egress data that should remain within the organization and prevent access to websites that are considered harmful in one form or another.

So why consider an API Gateway as part of an outbound traffic flow? After all, isn’t a Gateway there to protect us? Several very good reasons. Let’s look at them:

  • Managing the use of an external paid service. You may have multiple solutions using a third-party service – for example, an SMS service. Rather than expecting all these different calls to the external API, each having a copy of the 3rd party credentials to manage, we could use the gateway as a single point to attach the credentials.
  • When it comes to being charged for a service, being able to identify the requests at the API level makes it very easy to track your own consumption and forecast forward before being billed. This is really helpful if you have an agreement that provides a good price for pre-booked capacity and a higher charge for overage/capacity not pre-booked.
  • Economies of scale for using 3rd party services can be very powerful. But it can also present two problems.
    • Switching providers quickly can be difficult as multiple points of possible change
    • How to partition the cost of the external service across different departments if everyone is using a common account.

The first of these issues can be easily overcome using the anti-corruption layer pattern where the gateway represents the correct route so it can reformat the requests in one place to work with a different provider.

At the same time, we can more intelligently use Gateway’s metering mechanisms rather than having to implement functionality to mine the proxy’s logs.

Of course you can achieve same effect without a gateway, but you don’t get the benefits that a gateway will offer out of the box. In addition the chances are that you have already got an API Gateway running for your current North-South traffic.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Podcast with Anatolii Ulitovskyi of UNmiss

28 Wednesday Dec 2022

Posted by mp3monster in Books, General, Podcasts, Technology

≈ Leave a comment

Tags

API, book, integration, manning, Oracle, podcast

Just before the Christmas break, I got to record an excellent podcast with Anatolii of UNmiss. It was a great conversation about Cloud Integration, APIs, and approaches to Cloud-based integration. While I am not in a consulting role in the conventional sense, a lot of an Evangelist’s task is still to listen, understand, and, when necessary, challenge assumptions and help people understand how technologies can help address problems. This might include sketching out a journey of evolution and improvement. During the podcast, we discussed some of these ideas.

You can listen to the podcast audio or the live video stream of the conversation here.

In addition to some of the practices, we’ve used. The conversation touched upon books. My books are on the sidebar, including links to Manning, who, as a publisher, I’d recommend. I’ve previously blogged some reading recommendations and previously written some book reviews which may be of interest to anyone following up.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Published content

22 Thursday Dec 2022

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

API, book, Cloud, DevRel, Notification, OCI, Oracle, Queue, SEDaily, Software Engineering Daily

We haven’t blogged too much recently as we have been busy helping get and producing content for my employer Oracle, working with Software Engineering Daily, and developing a collaborative book. So, I thought I’d pull together some links to these new resources.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

API payload design getting the semantics right

14 Wednesday Dec 2022

Posted by mp3monster in APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, Design, OAGI, OASIS, OMG, payload, semantics, TMForum

One area of API design that doesn’t get discussed much is the semantics of the payload. That is, the names we give our attributes and elements for the values being communicated. When developing single-use APIs (usually for client applications), this is unlikely to be an issue as the team(s) involved are likely to know each other and are able to interact and resolve clarity issues easily enough (although getting the semantics right makes this easier particularly in the long term). But when it comes to providing reusable endpoints, we may know the early adopters but are unlikely to interact with consumers beyond that unless there is a problem.

This makes getting the semantics right somewhat harder. How do we know if our early adopters represent the wider customer base (internally and externally)? Conversely, if we simply use our own company terminology, how do we know that it is representative of the wider user base? It isn’t unusual for organizations to develop their own variations of a term or apply assumed meaning. Even simple things, a ‘post code’ element of an address, other parts of the world use ‘zip codes’ or PINS are they the same? Perhaps if we said ‘postal code,’ we break the direct specific country associations with ‘post code.’ We can overcome these issues by providing a dictionary of meanings and lengthy explanations. Using the right term goes beyond simply understanding the data value; it will infer specific formatting and potential application behaviors. Taking our postcode/zip code example. In the UK data is published, which means it is possible to easily validate a postcode against the address line and vice versa. In fact, in the UK to get something delivered, you only need the property number and the postcode. A US 5-digit zipcode can’t do that. For that precision, the ZIP+4 needs to be used.

If we can address these issues, then life becomes easier for us in maintaining the information and for consumers in not needing to look up the details. The question is how can we be sure of using semantics that is consistent across our APIs and widely understood and, when necessary, already documented, so we don’t have to document the information again?

Read more: API payload design getting the semantics right

Public Data Models

There is a shortcut to some of these problems. Many industries have agreed on data models for different industries. The bodies such as OASIS, OMG, and others are developed and maintained by multiple organizations. As a result, there is a commonality in the meaning achieved. So if you align with that meaning, then use that semantic. Not only can the naming of attributes become easier, but any documentation can be simplified to reference the published definitions. in most cases, these standards are publicly available as it promotes the widest adoption – one of the goals of developing such models. But there are some pitfalls to be mindful of using this approach:

  • Sometimes rather than arrive at a universal definition, the models will accommodate structural variations or aliased names – as a result, they may not necessarily be helpful to you.
  • The more well-known models are internationalized. If you have no intent to support international needs and not expecting to have international consumers, then the naming may not align with localized conventions.
  • If you use the semantics provided, ensure your data abides by the meaning. For example, don’t use ‘shipping address’ if you’re not shipping anything.
  • Don’t slavishly copy the data models provided – the model may not be intended for API use cases. At the same time, it doesn’t stop you from asking why the data in the model is there and whether your users may want such data (and whether it makes sense for you to provide that information).

Predefined APIs

Some organizations, such as TMForum have taken the public data model to the next step and provided predefined API specifications. This is ideal where you’re following industry standards and providing standardized/common services that aren’t a differentiator but need to be offered as part of doing business.

Data Catalogs

Larger, data-mature organizations will keep some form of Data Catalog. These catalogs are often held to help understand compliance needs, such as where personal data is held, how data issues can impact data accuracy and integrity, etc. It is possible that metadata may also be kept to address the semantic meaning of data or reference the definitions. Such information is used to help inform any data cleansing that may be needed. This offers a potentially good source of information for internal API use cases.

Vendor Led

If your business is delivery/service focussed so that your unique value isn’t in IT processes but perhaps something that the company manufactures or a specialist service such as consulting in a specific industry, then it is possible that the majority of your systems are SaaS or COTs based. If your business has opted to focus on a particular vendor, e.g., Oracle or SAP, for most services, then vendor-led data models are a possibility. These vendors are often involved with public data model development, so they won’t be too divergent in most situations – but awareness of differences is necessary, but as both models should be internally consistent, the differences will also be consistent. This approach will give you better alignment and reduce the chances of needing to address any divergence. The downside of this is a change of direction on strategic vendors can create additional work going forward as the alignment is disrupted. More work will be needed to map from your naming and semantics to the new core, and attempts to move away from the selected model to try to realign semantics with a new core will potentially create breaking changes for API consumers.

Don’t Forget

Regardless of the approach taken, there are some very simple but critical rules that will keep you in a good place:

  • Don’t use your underlying storage data models – this is a well-documented API anti-pattern.
  • Consistency of language across your APIs, regardless of whether they are internal or external, is important.

Information Sources

Regardless of approach – be careful not to lock your API semantics and data model to that of the storage layer – these can change and even create breaking changes that you shouldn’t expose to your users. Some sources to consider.

  • OAGIS – covers a broad variety of business data domains. Some ERP suppliers have used this as a foundation for their application data models.
  • OASIS – covers many industries
  • TMForum APIs
  • ARTS (formally hosted by NRF now with the OMG). The full OMG standards catalog.
  • GS1 – lots here on shipping, supply chain, and product tracking

Some more reading on the subject:

  • http://mlwiki.org/index.php/Semantic_Domains
  • https://www.w3.org/TR/vocab-dcat-2/

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

API more than a payload – Cloud Lunch Learn

10 Thursday Nov 2022

Posted by mp3monster in Books, General, Technology

≈ 1 Comment

Tags

API, books, manning, Packt, Presentation

Today I was fortunate enough to present at one of the Cloud Lunch and learn events (you can register for any of the events here and see previous sessions here). One of the questions asked at the end of the session was recommended reading on APIs. So I’ve gathered up some links to books I’d suggest worthwhile reading I’d suggest:

  • Enterprise API Management: Design and deliver valuable business APIs by Luis Weir (Amazon.co.uk)
  • The Design of Web APIs by Arnaud Lauret (direct from Manning) (Amazon.co.uk)
  • Designing APIs with Swagger and OpenAPI by Joshua S. Ponelat and Lukas L. Rosenstock (direct from Manning) (Amazon.co.uk)
  • API Security in Action by Neil Madden (direct from Manning) (Amazon.co.uk)
  • OAuth2 in Action by Justin Richer and Antonio Sanso (direct from Manning) (Amazon.co.uk)

I should also mention an API book I’ve co-authored. While it focuses on an Oracle product, there is a lot of content that is relevant to any API development using an API Gateway (Amazon.co.uk). I’ve not looked at all the books at API-University, but from I have seen the content is worth examining.

The slides for my presentation can be found on slideshare, and here:

API Design – More than just a Payload Definition from Phil Wilkins

The Presentation recording can be found here:

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Stop polling, Let’s go Streaming the podcast

15 Thursday Sep 2022

Posted by mp3monster in General, Podcasts

≈ Leave a comment

Tags

API, logrocket, podcast, podrocket, streaming

I was fortunate enough to be invited to join LogRocket‘s Podcast (PodRocket) to discuss some of the insights and considerations relating to API Streaming that I presented at a reference conference. To hear more go checkout :

If you’d like to see more from the presentation, go here.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Streaming APIs

05 Friday Aug 2022

Posted by mp3monster in APIs & microservices, development, General, Technology

≈ 1 Comment

Tags

API, architecture, code, GraphQL, gRPC, Oracle, streaming, subscriptions

Yesterday I was fortunate enough to participate in the Dev Innovation summit part of the World Festival virtual conference.

The presentation took a look at how Streaming APIs offer an alternative to API polling and the considerations needed when adopting streaming.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Apollo GraphQL – some pointers

16 Thursday Jun 2022

Posted by mp3monster in development, General, languages, node.js, Technology

≈ 2 Comments

Tags

API, code, development, GraphQL, javascript, node.js, Technology

I’ve designed a variety of GraphQL schemas and developed microservice backends. But not done much with configuring the Apollo implementation of a GraphQL server until recently. This may reflect the fact my understanding of JavaScript doesn’t extend into the world of Node.JS as much as I’d like (the problem with being a multi-language developer is you’re likely to find your way around many languages but never be a master of one). Anyway, the following content is about the implementation within a GraphQL server part of a solution. It may be these pointers are just for my benefit you might find them helpful as well.

Read more: Apollo GraphQL – some pointers

To make it easy to reference the code, we’ve added entries (n) into the code, where n is a number. This is not part of the code. But there to make the different lines referenceable. Where code should go but is not relevant to the point being made I’ve added ellipsis (…)

Dynamic loading and server configuration

import { ApolloServer } from 'apollo-server';
import { loadFilesSync } from '@graphql-tools/load-files';
import { resolvers } from './resolvers.js';   (1)
import ProviderInternalAPI from './ProviderInternalAPI.js'; (1)
import EventsInternalAPI from './EventsInternalAPI.js';  (1)
const server = new ApolloServer({
  debug : true,    (2)
  typeDefs: loadFilesSync('./schema.graphql'),   (3)
  resolvers,
  dataSources: () => {
    return {
      eventsInternalAPI: new EventsInternalAPI(),    (4)
      providerInternalAPI: new ProviderInternalAPI() (4)
      pro
    };
  }});

There is the potential to dynamically load the resolvers rather than importing each JavaScript file as we see on lines (1). The mechanics to do this is documented here. It would be cool if an opinionated implementation was provided. As shown by (3) we can take a independent schema file being loaded. The Apollo example approach for this didn’t seem to work for us, although both approaches make use of graphql-tools in a synchronous manner.

We can switch on debugging (2) for the GraphQL server, although the level of information published doesn’t appear to be significant. Ideally this setting is changed for production.

Defining the resolvers

The prefix for each resolver (1) must correlate to the name in the schema of the mutator or query (not the type as you would expect with Java). Often we don’t need all the parameters for the resolver. The documentation describes replacing each unused parameter with one or more underscores (i.e _, __ ). The underscore denoting the field not in use. However we can satisfy the indication of not being used, but keep the meaning of each position by using the underscore then a name (i.e. _parent, _args ) as shown in (2).

By taking the response into a variable (3) we can optionally log it. Trying to return using invocation line would result in the handler object rather than the payload itself. By taking the result into a variable we can log the content if desired and return the content.

The use of the backward quote is a node feature. It allows us to incorporate variables into a string by referencing it within ${} (4).

We need to supply the GraphQL server with instances with a layer of code that will interact with the resolvers. We can instantiate the instances in the declaration. The naming of the object is important (4) to the resolver.js (declarations).

import { useLogger } from "@graphql-yoga/node";
...
latestEvent (1): async (_parent, _args, { dataSources }, _info) (2)   => {
      if (log) { console.log("resolvers - get latest event"); }
      let responseValue = await dataSources.eventsInternalAPI.getLatestEvent(); (3)
      if (log) { console.log(`(4)  Resolver response for latest event:\n ${responseValue}`); }
      return responseValue;
    },

Resolver declarations

 Query: {  ...
 },
  
Mutation: {...
},
  Event: {  (1)
    providers: (event, args, { dataSources }, info) => {
      if (log) { console.log(`going to locate ${event.sources}`) }
      let responseValue = await (2) dataSources.providerInternalAPI.getProviders(event.sources);
      return responseValue;
    }

To handle the use of resolvers within a larger resolver we need to declare the resolution outside of the Query and Mutator blocks (but inside the whole declaration block)(1). The name provided needs to match the parent entity that the query resolver contributes to.

To then provide values from the outer resolution we need to prover to the chained resolution use the naming as represented in the GraphQL schema as shown by (2). The GraphQL engine will resolve the mapping values.

Web resolver URL

  // GET
  async getProvider(code) {
    console.log("getProvider (%s) directing to %s",code,this.baseURL);
    return this.get(`provider?code=${code} (1)`);
  }

The URL parameters need to be appended to the base URL path for the parent class to use in the invocation as shown by (1). The Apollo examples showed a setter option but we didn’t see the URI being addressed properly. This approach produces the relevant requirement.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

gRPC, GraphQL and more …

01 Tuesday Mar 2022

Posted by mp3monster in APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, GraphQL, gRPC, REST

I got into a discussion with several people about the use of GraphQL and related API technologies and discovered that a presentation I’ve been using and evolving for a while now, didn’t appear in my blog. So here is a version of it used at an API Conference …

gRPC, GraphQL, REST – Which API Tech to use – API Conference Berlin oct 20 from Phil Wilkins

The presentation may appear again in the future as the perspective of API technologies evolves the presentation will need to evolve. For example, AsyncAPI is starting to make an impression now. Other variants to API technologies such as DRPC are showing up.

If you’re new to GraphQL you might find a couple of other posts on the subject helpful:

  • GraphQL Mindmap
  • GraphQL
  • Useful Tech Resources

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Busy week with webinars & virtual conferences …

19 Friday Nov 2021

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

API, conference, DOAG, Flexagon, Flexdeploy, webinar

It has been a busy week with the day job of delivering to customers, with a webinar with Oracle, Flexagon and a customer – Heathrow Airport. Plus two technical sessions for DOAG.

Webinar

The webinar is the latest step in the partnership between Capgemini and Flexagon. If you’d like to see it, the recording can be accessed here.

The webinar development was very much a team effort with Oracle providing the hosting and promo logistics (particularly from Jurgen Kress and input from Daryl Eicher). Presentation development effort with Flexagon (Dan Goerdt and Tanya O’Connor). Of course the great input from our Heathrow customers – Mark South and Nick Byng-Maddick.

We have another blog post related to the webinar sharing some more detail. This will be published in a week or so on blog.oracle.com.

DOAG

The Deutsche Oracle Anwendergruppe (DOAG) conference took place virtually this week. DOAG and the UKOUG events are probably the biggest of the European user group events. With UKOUG happening at the end of the month.

My presentations at DOAG have been an update on which GraphQL, gRPC or REST API technology to use (see the presentation material here), along with a brand new presentation, again relating to APIs. But this time we’re looking at the application and capabilities of the Oracle Cloud’s API which can allow us to manage a wide range of services (see here). While I don’t have the demo piece recorded, you can see the slide aspect here.

Coming up

As mentioned I will be presenting at UKOUG Together. I have another brand new presentation for this as well, which will look at GitHub Actions combined with Oracle Cloud. We also have our first 2022 conference confirmed, in the form of API World.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...
← Older posts

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,540 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

January 2023
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  
« Dec    

Twitter

  • Learn how migrating your apps to #OCI can align with your overall application modernization strategy. Register for… twitter.com/i/web/status/1…Next Tweet: 7 hours ago
  • Join this demonstration and learn how banks can accelerate the account opening and onboarding process, improve aban… twitter.com/i/web/status/1…Next Tweet: 12 hours ago
  • IDC named @Oracle a leader in the MarketScape Worldwide #Hospitality Property Management Systems Vendor Assessment… twitter.com/i/web/status/1…Next Tweet: 1 day ago
  • Thank you for naming @Oracle the 'Best Enterprise Software Vendor' for 2022, @constellationr! social.ora.cl/60193bEs5Next Tweet: 1 day ago
  • Phoenix project blog.mp3monster.org/2023/01/21/pho…Next Tweet: 2 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 216 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d bloggers like this: