• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
    • Python Setup & related stuff
  • Music
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: CloudEvents

API Gateways to manage dynamic scaling

17 Friday Feb 2023

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

Cloud, CloudEvents, CNCF, dynamic, Plans, scaling

One of the benefits of using cloud providers is the potential to scale solutions to meet demand by scaling out with additional compute nodes without concern for physical capacity limits (cost considerations are smaller, but still apply). Dynamic scaling is typically driven by monitoring defined groups of nodes for CPU use and when demand hits a threshold an additional node is spun up. Kubernetes can create some more nuanced scaling configurations but this is tends to be used for microservice style solutions.

This is well and fine, but if we need to finesse the configuration to ensure multiple container instances (such as microservices) can be allowed on a single node without compromising the deployment of containers across nodes of a cluster to provide resilience things can get a lot more challenging. We also want to manage the number of active containers – we don’t want to have unnecessary volumes of containers effectively idling. So how do we manage this?

In addition to this, what if we’re using services that demand tens of seconds or even minutes to be spun up to a point of readiness – such as instantiating database servers, adding new nodes to data caches such as Coherence which will need time to clone data, and adjust the demand balancing algorithms? Likewise, for more traditional application servers. Some service calls will impact upstream configuration changes, such as altering load-balancing configurations. Cloud-native Load Balancers typically can understand node groups, but what if your configuration is more nuanced? If you’re running services such as ActiveMQ you need to update all the impacted nodes in a cluster to be aware of the new node. Bottomline is some solutions or solution parts need lead time to handle increased demand?

This points to more advanced, nuanced metrics than simply current CPU load. And possibly the scaling algorithm needs to be aware of lead times of the different types of functionality involved. So how can we advance a more nuanced, and informed scaling logic? You could look at the inbound traffic on firewalls or load balancers. But this will require a fair bit of additional effort to apply application context. For example is the traffic just getting directed to static page content? We do need to derive some context so that the right rules are impacted. Even in simple K8s only hosted solutions a cluster may host services that aren’t related to the changes in demand and need to be scaled as a result.

A use case perspective

Let’s look at this from a hypothetical use case. We’re a music streaming service. Artists can control when their music is made available, but the industry norm is 12:01am on a Friday morning. There is always a demand spike at this time, but the size of the demand spike can vary, with some artists triggering larger spikes (this isn’t directly correlating to an artist’s popularity, some smaller groups can have very enthusiastic fans) – so dynamic scaling is needed, and reactive to demand. Reporting, analytics, and payment services see cyclical bumps around the monthly financial periods. These monthly cyclic activities can also be done during quieter operating hours. So it is clear we may wish to apply logical partitioning, but for maximum cost efficiency keep everything in the same cluster. There are correlations between different service demands. Certain data services see increased demand during the demand spikes and reporting period, but ‘The bottom line is we need to not only address dynamic scaling, but tailor the scaling to different services at different times.

Back to our question – how do we manage the scaling? We could monitor the firewall and load balancer logs and analyze the kinds of requests being received. That would need additional processing logic to determine which services are receiving the traffic. But our API gateway is likely to have that intelligence implicitly in place as we have different API policies for the different types of endpoints. So we can monitor specific endpoints or groups of endpoints very easily – meaning we can infer the types of traffic demand and how to respond. Not only that, we may have API plans in place so we can control priority and prevent the free versions of our service from using APIs to initiate high-quality media streams. So we already have business and specific process meaning. So linking scaling controls such as KEDA (Kubernetes Event-driven Autoscaling) directly to measurements such as plans and specific APIs creates a relatively easy way to control scaling. Further, we may also use the gateways to provide a rate throttle so it’s not possible to crash our backend with an instantaneous spike. This strategy isn’t that different from an approach we’ve demonstrated for scaling message processing backends (see here and here).

Representation of how we can use the metrics from a gateway to  not only support the scaling of a K8s cluster but also other cloud services
Representation of how we can use the metrics from a gateway to not only support the scaling of a K8s cluster but also other cloud services

We can also use the data KEDA can see in terms of node readiness to adjust the API Gateway rate limiting dynamically as well. Either as a direct trigger from the number of active container instances or by triggering a more advanced check because we still have to address our services that need more lead time before relaxing the rate limiting.

Handling slow scaling services

So we’ve got a way of targeting the scaling of services. But what do we do to address the slower scaling services we described? With the data dimensioned, we can do a couple of things. Firstly we can use the rate of change to determine how many nodes to add. A very sudden increase in demand and adding a node at a time will create the effect of the service performance stuttering as it scales, and all additional capacity is suddenly consumed and then scales again. Factoring in the rate of change to the workload threshold and the context of which services could generate the increased workload can be used to not only determine which databases may need scaling but also be used to adjust the threshold of workload that actually triggers the node introduction. So a sudden increase in demand on services that are known to create a lot of DB activity is then met with dropping the threshold that triggers new nodes from, say, 75% to 25% so we effectively start the nodes on a lower threshold, means the process is effectively started sooner.

Illustrating how the rate of growth in utilization can mean the need to change the threshold to trigger scaling up
With different rates of utilization growth, we can see that we need to alter the trigger threshold for launching new resources

For this to be fully effective we do need the Gateway tracking traffic both internally (East-West) and inbound (North-South). Using the gateway with East-West traffic means that we can establish an anti corruption layer.

Conclusion

Not all parts of a solution will be instantaneous in scaling – regardless of how fast the code startup cycle might be, some services have to address the need to move large chunks of data before becoming ready, e.g., in-memory databases. Some services may not have been built for the latest business demands and the ability to exploit cloud scale and dynamic scaling. Some services need time to adjust configurations.

We may also need to adjust our protection mechanisms if we’re protecting against service overloading.

Scaling capabilities that have response latency to the scaling process can be addressed by achieving earlier, more intelligent recognition of need than warning simply by CPU loads hitting a threshold. The intelligence that can be derived from implicit service context simplifies the effort in creating such intelligence and makes it easier to recognize the change. API Gateways, message queues, message streams, and shared data storage are all means that, by their nature, have implicit context.

Pushing the recognition towards the front of the process creates milliseconds or possibly seconds more warning of the demand than waiting for the impacted nodes to see compute spikes.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

More Free Oracle Cloud than you might know

04 Monday Nov 2019

Posted by mp3monster in Cloud, General, Oracle, Technology

≈ Leave a comment

Tags

CloudEvents, CNCF, events, free, Functions, Notifications, Oracle

free-750x536The news about Oracle offering some free cloud services ‘for life’ is making an impact.  But, the free services don’t end there. The pricing of some other native cloud services includes some free bands. So it’s worth keeping an eye on the fine print. I wouldn’t be surprised if we see limited capacity access in other areas.

Oracle Functions – whilst the core of this service is built on the open-source Fn Project (also largely driven by Oracle) the managed service has a free tier allowing up to 2 million invocations that can consume 400, 000 gigabytes of memory per second use (details can be seen here). Plenty enough to experiment with the concepts behind Serverless aka FaaS capabilities.

Oracle Notifications whilst focussed on the technical side of gathering key event data from OCI and its services, as the document states “sending notifications to numerous interested parties, or even synchronizing the moving parts of a distributed application” – this obviously means a service with characteristics a bit like AWS’ SNS. Like SNS it can be hooked up to email and other HTTPS services using Oracle Events which also has free use. Events is particularly interesting as it is bases the event structure on the CNCF CloudEvents spec. There is an excellent illustration of such a use case in the Oracle blogs here.

It will be interesting to see if we a similar trend with other Oracle cloud-native services. A new take on the now-defunct Application Container Cloud Service (ACCS) would be an ideal vehicle – whether there is sufficient demand for such a capability is not clear (it would in effect be an always live service like a Kubernetes solution, but the simpler, smaller footprint more like Functions in a multi-tenant environment. At the same time, it doesn’t have potential latency of a Function being activated).

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,573 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

March 2023
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Feb    

Twitter

  • .@Oracle #CloudWorld Tour #London is just around the corner! ⏰ 🇬🇧 Don’t forget to register for the one-day #cloud… twitter.com/i/web/status/1…Next Tweet: 4 hours ago
  • Learn how @OracleCloud can help you improve the efficiency of your business operations at the upcoming Level Up eve… twitter.com/i/web/status/1…Next Tweet: 2 days ago
  • Join Juan Loaiza for the Data Strategies Day keynote at Level Up to learn how to eliminate complexity by leveraging… twitter.com/i/web/status/1…Next Tweet: 3 days ago
  • RT @WunderlichRd: Great post by @mp3monster around how APIs are relevant in so many industries! lnkd.in/eshagCDKNext Tweet: 3 days ago
  • King’s College Hospital London in Dubai announces a strategic collaboration with Oracle Cerner to help accelerate i… twitter.com/i/web/status/1…Next Tweet: 3 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 218 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
     

    Loading Comments...
     

    You must be logged in to post a comment.

      Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
      To find out more, including how to control cookies, see here: Our Cookie Policy
      %d bloggers like this: