• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Mindmaps Index
    • Oracle Integration Site
    • Useful Tech Resources …
      • Oracle Tech Resources inc Open Source
      • Useful Tech Resources
      • Python Setup & related stuff
  • Music

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Category Archives: APIs & microservices

Different aspects of APIs and microservices from API Design to utilities to work with API Platform and microservices from mindmasps on design to paradigm concepts

API payload design getting the semantics right

14 Wednesday Dec 2022

Posted by mp3monster in APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, Design, OAGI, OASIS, OMG, payload, semantics, TMForum

One area of API design that doesn’t get discussed much is the semantics of the payload. That is, the names we give our attributes and elements for the values being communicated. When developing single-use APIs (usually for client applications), this is unlikely to be an issue as the team(s) involved are likely to know each other and are able to interact and resolve clarity issues easily enough (although getting the semantics right makes this easier particularly in the long term). But when it comes to providing reusable endpoints, we may know the early adopters but are unlikely to interact with consumers beyond that unless there is a problem.

This makes getting the semantics right somewhat harder. How do we know if our early adopters represent the wider customer base (internally and externally)? Conversely, if we simply use our own company terminology, how do we know that it is representative of the wider user base? It isn’t unusual for organizations to develop their own variations of a term or apply assumed meaning. Even simple things, a ‘post code’ element of an address, other parts of the world use ‘zip codes’ or PINS are they the same? Perhaps if we said ‘postal code,’ we break the direct specific country associations with ‘post code.’ We can overcome these issues by providing a dictionary of meanings and lengthy explanations. Using the right term goes beyond simply understanding the data value; it will infer specific formatting and potential application behaviors. Taking our postcode/zip code example. In the UK data is published, which means it is possible to easily validate a postcode against the address line and vice versa. In fact, in the UK to get something delivered, you only need the property number and the postcode. A US 5-digit zipcode can’t do that. For that precision, the ZIP+4 needs to be used.

If we can address these issues, then life becomes easier for us in maintaining the information and for consumers in not needing to look up the details. The question is how can we be sure of using semantics that is consistent across our APIs and widely understood and, when necessary, already documented, so we don’t have to document the information again?

Read more: API payload design getting the semantics right

Public Data Models

There is a shortcut to some of these problems. Many industries have agreed on data models for different industries. The bodies such as OASIS, OMG, and others are developed and maintained by multiple organizations. As a result, there is a commonality in the meaning achieved. So if you align with that meaning, then use that semantic. Not only can the naming of attributes become easier, but any documentation can be simplified to reference the published definitions. in most cases, these standards are publicly available as it promotes the widest adoption – one of the goals of developing such models. But there are some pitfalls to be mindful of using this approach:

  • Sometimes rather than arrive at a universal definition, the models will accommodate structural variations or aliased names – as a result, they may not necessarily be helpful to you.
  • The more well-known models are internationalized. If you have no intent to support international needs and not expecting to have international consumers, then the naming may not align with localized conventions.
  • If you use the semantics provided, ensure your data abides by the meaning. For example, don’t use ‘shipping address’ if you’re not shipping anything.
  • Don’t slavishly copy the data models provided – the model may not be intended for API use cases. At the same time, it doesn’t stop you from asking why the data in the model is there and whether your users may want such data (and whether it makes sense for you to provide that information).

Predefined APIs

Some organizations, such as TMForum have taken the public data model to the next step and provided predefined API specifications. This is ideal where you’re following industry standards and providing standardized/common services that aren’t a differentiator but need to be offered as part of doing business.

Data Catalogs

Larger, data-mature organizations will keep some form of Data Catalog. These catalogs are often held to help understand compliance needs, such as where personal data is held, how data issues can impact data accuracy and integrity, etc. It is possible that metadata may also be kept to address the semantic meaning of data or reference the definitions. Such information is used to help inform any data cleansing that may be needed. This offers a potentially good source of information for internal API use cases.

Vendor Led

If your business is delivery/service focussed so that your unique value isn’t in IT processes but perhaps something that the company manufactures or a specialist service such as consulting in a specific industry, then it is possible that the majority of your systems are SaaS or COTs based. If your business has opted to focus on a particular vendor, e.g., Oracle or SAP, for most services, then vendor-led data models are a possibility. These vendors are often involved with public data model development, so they won’t be too divergent in most situations – but awareness of differences is necessary, but as both models should be internally consistent, the differences will also be consistent. This approach will give you better alignment and reduce the chances of needing to address any divergence. The downside of this is a change of direction on strategic vendors can create additional work going forward as the alignment is disrupted. More work will be needed to map from your naming and semantics to the new core, and attempts to move away from the selected model to try to realign semantics with a new core will potentially create breaking changes for API consumers.

Don’t Forget

Regardless of the approach taken, there are some very simple but critical rules that will keep you in a good place:

  • Don’t use your underlying storage data models – this is a well-documented API anti-pattern.
  • Consistency of language across your APIs, regardless of whether they are internal or external, is important.

Information Sources

Regardless of approach – be careful not to lock your API semantics and data model to that of the storage layer – these can change and even create breaking changes that you shouldn’t expose to your users. Some sources to consider.

  • OAGIS – covers a broad variety of business data domains. Some ERP suppliers have used this as a foundation for their application data models.
  • OASIS – covers many industries
  • TMForum APIs
  • ARTS (formally hosted by NRF now with the OMG). The full OMG standards catalog.
  • GS1 – lots here on shipping, supply chain, and product tracking

Some more reading on the subject:

  • http://mlwiki.org/index.php/Semantic_Domains
  • https://www.w3.org/TR/vocab-dcat-2/

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Streaming APIs

05 Friday Aug 2022

Posted by mp3monster in APIs & microservices, development, General, Technology

≈ 1 Comment

Tags

API, architecture, code, GraphQL, gRPC, Oracle, streaming, subscriptions

Yesterday I was fortunate enough to participate in the Dev Innovation summit part of the World Festival virtual conference.

The presentation took a look at how Streaming APIs offer an alternative to API polling and the considerations needed when adopting streaming.

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Kubernetes Deployment – pulling from OCI Registry (OCIR)

26 Thursday May 2022

Posted by mp3monster in APIs & microservices, Cloud Native, General, Oracle, Oracle Cloud Native, Technology

≈ Leave a comment

Tags

Containers, deploy, K8s, Kuberrnetes, OCIR, registry, secret, token

The following isn’t unique to OCIR, as it will hold true for any K8s Deployment YAML configuration that works with an Open Container Initiative compliant registry. To define the containers part of the YAML file we need to provide an attribute that can be used to confirm the legitimacy of the request. To do this we need to supply a token. However, we don’t want this token to be visible in plain sight in our YAML. The solution to this is to set up a secret within Kubernetes.

In the following YAML extract, we can see the secret is named.

kind: Deployment
metadata:
  name: graph-svr-deploy
  labels:
    app: arch-oke-graphql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: arch-oke-graphql
  template:
    metadata:
      name: graph-svr-deploy
      labels:
        app: arch-oke-graphql
    spec:
      containers:
      - name: graphql-svr
        image: iad.ocir.io/ociobenablement/graphql-svr:latest 
        ports:
        - containerPort: 4000    
          name: graph-svr-web
      imagePullSecrets:
      - name: ocirsecret     

This does mean we need to create the secret. As this is a one-off task the easiest step is to create the secret by hand. To do that we use the command:

@kubectl create secret docker-registry ocirsecret --docker-server=iad.ocir.io --docker-username=ociobenablement/identitycloudservice/philip.wilkins@oracle.com --docker-password='xxxxxxxx' --docker-email=philip.wilkins@oracle.com

This naturally leads to the next question where do we get the secret?

This step is straightforward. Navigating using the user icon top right (highlighted in the screenshot below), select the User Settings option to get to the screen shown below. Then use the right-hand menu option highlight (Auth Tokens). This displays a section of the UI showing your current auth tokens and provides a button that will popup a window to guide you through creating a new auth token.

Related Posts

  • Image building across machine architectures for Oracle Container Registry (OCIR)
  • Container Registry – pushing and storing containers

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

gRPC, GraphQL and more …

01 Tuesday Mar 2022

Posted by mp3monster in APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, GraphQL, gRPC, REST

I got into a discussion with several people about the use of GraphQL and related API technologies and discovered that a presentation I’ve been using and evolving for a while now, didn’t appear in my blog. So here is a version of it used at an API Conference …

gRPC, GraphQL, REST – Which API Tech to use – API Conference Berlin oct 20 from Phil Wilkins

The presentation may appear again in the future as the perspective of API technologies evolves the presentation will need to evolve. For example, AsyncAPI is starting to make an impression now. Other variants to API technologies such as DRPC are showing up.

If you’re new to GraphQL you might find a couple of other posts on the subject helpful:

  • GraphQL Mindmap
  • GraphQL
  • Useful Tech Resources

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Kubernetes Best Practises – Review & Mindmaps

13 Thursday Jan 2022

Posted by mp3monster in APIs & microservices, Book Reviews, Books, Cloud, Cloud Native, development, General, Technology

≈ 2 Comments

Tags

"best practise", book, Istio, Kubernetes, mindmap, review

I’ve had some time to catch up on books I’d like to read, including Kubernetes Best Practises in the last few weeks. While I think I have a fair handle on Kubernetes, the development of my understanding has been a bit ad-hoc as I’ve dug into different areas as I’ve needed to know more. This meant reading a Dummies/Introduction to entry style guide would, to an extent, likely prove to be a frustrating read. Given this, I went for the best practises book because if I don’t understand the practises, then there are gaps in my understanding still, and I can look at more foundation resources.

As it goes, this book was perfect. It quickly covered the basics of the different aspects of Kubernetes helping to give context to the more advanced aspects, and the best practices become almost a formulated summary in each section. The depth of coverage and detail is certainly very comprehensive, explaining the background of CNI (Container Network Interface) to network-level security within Kubernetes.

The book touched upon Service Meshes such as Istio and Linkerd2 but didn’t go into great depth, but again this is probably down to the fact that Service Mesh ideas are still maturing, and you have initiatives like SMI (Service Mesh Interface still in the CNCF’s sandbox).

In terms of best practices, that really stood out for me:

  • Use of Taints and Tolerations for refined control of pod deployment (Allowing affinity to be controlled to optimise resilience, or direct types of pod deployment to nodes with specialist capabilities such as GPU).
  • There are a lot more differences and options then you might realize in terms of ingress controller capabilities, so take time to identify what you may need from an ingress controller.
  • Don’t forget pods can be scaled vertically with the VPA (Vertical Pod Autoscaler)as well as horizontally through the HPA.
  • While using a managed persistence service will make statement storage a lot easier, stateful sets will give you a very portable solution.

As with a lot of technical books I read. As I go through the book I build up a mind map of what I think are the key points. Doing so leaves me with a resource I can use as a quick reference, but creating the mind map helps reinforce the learning. So here is the mind map …


  • mindmap in iThoughts format
  • mindmap in FreeMind format
  • mindmap as an expanded png

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

APIs more than a payload spec – Examples of Good APIs

31 Sunday Oct 2021

Posted by mp3monster in APIs & microservices, General, Technology

≈ 1 Comment

Tags

API, apiary, APIWorld, car, conference, Hospitality, Mercedes, Nordic API, OHIP, Oracle, photoss, Postman, presenting, REST, Walgreens

I recently presented at APIWorld about how API definitions go beyond the payload specification into providing details of terms and conditions and so on. You can see the presentation here (more about my presentations here).

One of the questions during the presentation did I have other examples of good APIs, reflecting the points I’d made. A very valid question, to which I didn’t have more examples to hand, hence this post.

So the easy answer would be to point to an excellent article on Nordic APIs (here) that address the question and explain why they rate the APIs. But that’s a little bit of a lazy answer and in all fairness, the examples provided are from organisations where APIs are recognised as a primary or important contributor to business revenue. So I’ve looked at areas where the API may not necessarily be seen by the business as the primary source of revenue. With the examples provided, we’ve described what we think is good, or not so good about them. Hopefully, through these examples, you’ll see why points are made in the presentation. So here are my reviews…

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

OraWorld – With Great APIs …

21 Monday Jun 2021

Posted by mp3monster in APIs & microservices, General, Oracle, Technology

≈ Leave a comment

Tags

API, OraWorld, Security

The latest edition of OraWorld is out which includes the second part of my part part articles relating to GraphQL and API Security. You can check it out at on page 22, along with lots of other great content here.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

DeveloperWeek Europe 2021 – APIs more than just a Payload definition

27 Tuesday Apr 2021

Posted by mp3monster in APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, DeveloperWeek, OAS, payload

DeveloperWeek Europe 2021

So this week the big Developer Week Europe conference is running online at the moment. I got to present today. It was a relatively short session, with an unfortunate brief interruption of a smoke alarm. My presentations is here …

API more than payload from Phil Wilkins

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

OraWorld Magazine – Latest Edition

09 Tuesday Mar 2021

Posted by mp3monster in APIs & microservices, General, Oracle, Technology

≈ Leave a comment

Tags

API, article, GraphQL, journal, magazine, OraWorld

The latest edition of OraWorld has become available to today. With its blend of insight into the Oracle community, and Oracle technologies from database to modern apps. I have to own up and say, I mention the magazine not only because of the beautifully crafted independent insights, but also it includes an article from myself. Taking a look at GraphQL what it is and how recent new Oracle product features could make a big difference to the GraphQL adoption opportunities.

The next edition should include a follow up article to this focussing on API security considerations.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Oracle API CS vs OCI API approach to securing gateway configuration

02 Tuesday Jun 2020

Posted by mp3monster in API Platform CS, APIs & microservices, General, Technology

≈ Leave a comment

Tags

API, Cloud Native, IaaS, Linux Foundation, OCI, Oracle, Owasp, PaaS, SaaS, Security, Terraform

A couple of years ago I got to discuss some of the design ideas behind API Platform Cloud Service. One of the points we discussed was how API Platform CS kept the configuration of APIs entirely within the platform, which meant some version management tasks couldn’t be applied like any other code. Whilst we’ve solved that problem (and you can see the various tools for this here API Platform CS tools). The argument made that your API policies are pretty important, if they get into the public domain then people can better understand to go about attacking your APIs and possibly infer more.

Move on a couple of years, Oracle’s 2nd generation cloud is established an maturing rapidly (OCI) and the organisational changes within Oracle mean PaaS was aligned to SaaS (Oracle Integration Cloud, Visual Builder CS as examples) or more cloud native IaaS. The gateway which had a strong foot in both camps eventually became aligned to IaaS (note that this doesn’t mean that the latest evolution of the API platform (Oracle Infrastructure API) will lose its cloud agnostic capabilities, as this is one of unique values of the solution, but over time the underpinnings can be expected to evolve).

Any service that has elements of infrastructure associated with it has been mandated to use Terraform as the foundation for definition and configuration. The Terraform mandate is good, we have some consistency across products with something that is becoming a defacto standard. However, by adopting the Terraform approach does mean all of our API configurations are held outside the product, raising the security risk of policy configuration is not hidden away, but conversely configuration management is a lot easier.

This has had me wondering for a long time, with the use of Terraform how do we mitigate the risks that API CS’s approach was trying to secure? But ultimately the fundamental question of security vs standardisation.

Mitigation’s

Any security expert will tell you the best security is layered, so if one layer is found to be vulnerable, then as long as the next layer is different then you’re not immediately compromised.

What this tells us is, we should look for ways to mitigate or create additional layers of security to protect the security of the API configuration. These principles probably need to extend to all Terraform files, after all it not only identifies security of not just OCI API, but also WAF, networks that are public and how they connect to private subnets (this isn’t an issue unique to Oracle, its equally true for AWS and Azure). Some mitigation actions worth considering:

  • Consider using a repository that can’t be accidentally exposed to the net – configuration errors is the OWASP Top 10. So let’s avoid the mistake if possible. If this isn’t an option, then consider how to mitigate, for example …
    • Strong restrictions on who can set or change visibility/access to the repo
    • Configure a simple regular check that looks to see if your repos have been accidentally made publicly visible. The more frequent the the check the smaller the potential exposure window
  • Make sure the Terraform configurations doesn’t contain any hard coded credentials, there are tools that can help spot this kind of error, so use them. Tools exist to allow for the scanning of such errors.
  • Think about access control to the repository. It is well known that a lot of security breaches start within an organisation.
  • Terraform supports the ability to segment up and inject configuration elements, using this will allow you to reuse configuration pieces, but could also be used to minimize the impact of a breach.
  • Of course he odds are you’re going to integrate the Terraform into a CI/CD pipeline at some stage, so make sure credentials into the Terraform repo are also secure, otherwise you’ve undone your previous security steps.
  • Minimize breach windows through credentials tokens and certificate hanging. If you use Let’s Encrypt (automated certificate issuing solution supported by the Linux Foundation). Then 90 day certificates isn’t new.

Paranoid?

This may sound a touch paranoid, but as the joke goes….

Just because I’m paranoid, it doesn’t mean they’re not out to get me

Fundamental Security vs Standardisation?

As it goes the standardisation is actually a dimension of security. (This article illustrates the point and you can find many more). The premise is, what can be ensured as the most secure environment, one that is consistent using standards (defacto or formal) or one that is non standard and hard to understand?

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...
← Older posts

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,540 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

January 2023
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  
« Dec    

Twitter

  • Learn how migrating your apps to #OCI can align with your overall application modernization strategy. Register for… twitter.com/i/web/status/1…Next Tweet: 6 hours ago
  • Join this demonstration and learn how banks can accelerate the account opening and onboarding process, improve aban… twitter.com/i/web/status/1…Next Tweet: 11 hours ago
  • IDC named @Oracle a leader in the MarketScape Worldwide #Hospitality Property Management Systems Vendor Assessment… twitter.com/i/web/status/1…Next Tweet: 1 day ago
  • Thank you for naming @Oracle the 'Best Enterprise Software Vendor' for 2022, @constellationr! social.ora.cl/60193bEs5Next Tweet: 1 day ago
  • Phoenix project blog.mp3monster.org/2023/01/21/pho…Next Tweet: 2 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 216 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d bloggers like this: