• Home
  • Site Aliases
    • www.cloud-native.info
    • oracle.cloud-native.info
    • Phil-Wilkins.uk
  • About
    • Background
    • Presenting Activities
    • Internet Profile
      • LinkedIn
    • About
  • Books & Publications
    • Logging in Action with Fluentd, Kubernetes and More
      • Logging in Action with Fluentd – Book
      • Fluentd Book Resources
      • Log Generator
    • API & API Platform
      • API Useful Resources
    • Oracle Integration
      • Book Website
      • Useful Reading Sources
    • Publication Contributions
  • Resources
    • GitHub
    • Oracle Integration Site
    • Oracle Resources
    • Mindmaps Index
    • Useful Tech Resources
    • Python Setup & related stuff
  • Music
    • Music Reading

Phil (aka MP3Monster)'s Blog

~ from Technology to Music

Phil (aka MP3Monster)'s Blog

Tag Archives: Splunk

Is The 12 Factor App right about Logging?

05 Wednesday Oct 2022

Posted by mp3monster in development, Fluentd, General, Technology

≈ Leave a comment

Tags

12 Factor, 12 Factor App, conference, development, Grafana, JAX, logging, London, OpenSearch, Prometheus, Splunk, stdout

The 12 Factor App definition is now ten years old.  In the world of software that is a long time. So perhaps it’s time to revisit and review what it says.  As I have spent a lot of time around Logging – I’ve focussed on Factor 11 – Logging.

I have been fortunate enough to present at the hybrid JAX London conference on this subject. It was great to get out and see people at a conference rather than just with a screen and a chat console of online-only events.

You can see my presentation here:

Continue reading →

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Mitigating Risks of Cloud Services

08 Monday Dec 2014

Posted by mp3monster in General, Technology

≈ Leave a comment

Tags

Big Data, data, Data Warehouse, RPO, RTO, SaaS, service, Splunk, XaaS

As previously blogged there are risks with using cloud that differ from self hosting solutions. SaaS, PaaS and all the other XaaS offerings aren’t a panacea. Hopefully you won’t become the next Sony as the provider keeps you patched etc. But if you’re using a SaaS provider that goes bust or you get into litigation with your provider, as result losing access to your data. It could be potentially be months whilst the lawyers sort things out. A horrible situation that no one wants to find themselves in. But how to mitigate such risks?

Any half decent SaaS provider should give directly the means to get a view of all your data through a generic or custom report (s), or will should make available the means for providing an export of your data. The later approach may well come with a cost. If your SaaS solution has a lot of data in place – for example a multinational’s HR solution you may want to just target the extract of deltas. This means extra donkey work and someone to ensure it is happening. How frequently that should depend upon your business needs through an agreed Recovery Point Objective and the tolerance to potential data loss as you can assume you’ll lose everything from the last snapshot. If you have middleware in front of your SaaS service you can have a wiretap to reduce the risk here.

Your net position is in the event of a loss or possibly a prolonged service outage (remember even Amazon have had multi-day failures & not all SaaS solutions follow good cloud practise of being able to fail to secondary centres) is that you have your data and can atleast cobble something together to bridge the gap. Unless you SaaS vendor is offering you something very unique then they’re probably going to have competitors that are more than likely to be glade to help you import the data into their solution for you.

All this for a case of paranoia? Well actually you can have harvest a raft of other benefits from taking full data extracts – for example reconciliation with a view to managing data quality – statistics from Experian show the value of resolving discrepancies. This is to say – that you might find data errors between systems as a result things like edge scenarios such as handling errors in the integration layer. To illustrate the point, let’s assume that your web sales channel is via a SaaS provider and you’re receiving the sales into your on premise ERP for fulfilment and accounting. By taking every week all transactions in the SaaS solution you can identify and discrepancies and reconcile any issues between the sales solution, your finance and fulfilment capabilities to ensure what you have sold is what you have accounted for.  If we’re talking about solutions that impact your financial accounting, then for atleast US declarations it maybe necessary to perform such reconciliation in support of Sarbanes Oxley (SOX) requirements.

Add to this a richer data set can be added to your Big Data or Data Warehouse environments allowing you to gain potentially further insights into your activities.

When you are running a hybrid of on premise and cloud solutions or event just cloud but a mix of vendors don’t just think about you application data, but consider whether audit and web traffic information can be retrieved from the vendor – there maybe value in feeding that data into a solution such as Splunk which may then find a pattern of misuse or attack that may not show up with just the monitoring data from your on premise solutions.

The final point I should make, is don’t assume your service provider will let you at the data as described – look at your contracts before any payment or act of agreement. Ideally such checks should be part of your service due diligence activities (along with ESCROW) etc. There are SaaS providers who will consider the data as their property not yours even when the data might be about your employees.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Mastering Puppet Review

09 Tuesday Sep 2014

Posted by mp3monster in Books, General, Technology

≈ Leave a comment

Tags

book, EasticSearch, Foreman, Kibana, Logstash, mCollective, Packt, Pulp, Puppet, review, Ruby, Splunk, Thomas Uphill, YAML

Packt’s Mastering Puppet kicks off with substantial first chapter on how to setup Puppet in a manner that can then scale. The core of this is driven by an explanation of the constituent parts of a Puppet solution and where the workload is. In terms of execution this is as much about understanding the configuration of things like Apache, Passenger and Ningx as it is Puppet. As part of the explanation there are indicative numbers in terms of supportable scale which reflects the knowledge of the product.

Looking at configuration distribution for headless deployments with Git is a solid well considered piece and the writing suggests considers all the needs of a solid deployment of a production quality solution such as access control, whilst supporting collaborative working etc. it would be interesting to have seen how that would have stacked against capabilities such as Zookeeper.

As we move through the chapters the books continues with more advanced themes such as using Hiera as a object hierarchical framework for managing configuration and on into leveraging Puppet forge and various Git repositories (and the challenges when linking to git repositories of the latest code vs a release). With the repositories we can draw in additional tooling and how to incorporate these capabilities into a deployment. This includes looking at several modules that practical experience from the author would recommend.

By chapter 6 we’re into writing our own custom modules and facts and deploying them. So you can do things such as create modules to manage your custom solutions.

The next natural step is to look at the reporting aspects of Puppet, orchestration through marionette collective (mCollective). Obviously to report you need to gather the activity information, so the book touches on the out of the box (OOTB) approach and moves onto the idea of using IRC; presentation via Foreman and Puppet Dashboard. Finally then with a reporting view, the next step is to dynamically query the nodes in Puppet environment which uses mcollective to communicate back & forth with the nodes.

So now we have a dynamically configurable set of Nodes, which can report and have dynamic querying against the nodes.  Final chapters cover the use of things like PuppetDB, roles & profiles and developing and debugging your puppet environment.

Reading the book, I get the feeling that a fair grasp of Linux system administration would help (i.e. a bit more than the average developer). There are a few useful touches that I think could have been included, such as external references such as man pages for RPM or site for the Pulp tool mentioned. But, as criticisms go, this as much me being too lazy to Google. The only other refinement would be inclusion of some diagrams to support the words. As they say a picture can tell a 1000 words, even if this was to just show the hierarchy or directory structures involved.

Compared to the recently reviewed Puppet Reporting book, this book isn’t for someone starting out with Puppet (but the Packt site says as much). You atleast need to have got some basic understanding or practical exposure to Puppet,  and exposure to a development environment is an added bonus.  So if you’re setting out with Puppet you might consider starting with the Puppet 3 Beginner’s Guide (Amazon) or Instant Puppet 3 Starter (Amazon).  Having got those under your belt, try this book to to really develop the use of Puppet configuration and deployment.  When it comes to reporting I’d look at this book along with reporting book (reviewed here).  This book feels like more options are on offer, but Puppet Reporting is a lot richer (but you’d expect that given the different book emphasis).

In summary – good solid book, full of practical experience and ideas.  But don’t try to use this as a jumpstart to Puppet.

Below are a few links I thought might be helpful as they aren’t in the book:

  • YAML – human readable serialization format
  • Pulp – software repository management app
  • Ruby – Open Source OO programming language
  • Foreman – tool capable of extending puppet to deliver PXE capabilities along with capabilities such as reporting
  • Splunk – BigData style analytics on log files etc
  • Elasticsearch / Logstash / Kibana (ELK) – set of tools to provide analytics against log files
  • ActiveMQ – Apache implementation of a JMS compliant messaging solution used my mcollective

Mastering Puppet at Amazon.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Print
  • Pocket
  • Email
  • Tumblr
  • Reddit
  • Pinterest
  • WhatsApp
  • Skype

Like this:

Like Loading...

Aliases

  • phil-wilkins.uk
  • cloud-native.info
  • oracle.cloud-native.info

I work for Oracle, all opinions here are my own & do not necessarily reflect the views of Oracle

Oracle Ace Director Alumni

TOGAF 9

Logging in Action

Oracle Cloud Integration Book

API Platform Book


Oracle Dev Meetup London

Categories

  • App Ideas
  • Books
    • Book Reviews
    • manning
    • Oracle Press
    • Packt
  • Enterprise architecture
  • General
    • economy
    • LinkedIn
    • Website
  • Music
    • Music Resources
    • Music Reviews
  • Photography
  • Podcasts
  • Technology
    • APIs & microservices
    • chatbots
    • Cloud
    • Cloud Native
    • Dev Meetup
    • development
      • languages
        • node.js
    • drone
    • Fluentd
    • logsimulator
    • mindmap
    • OMESA
    • Oracle
      • API Platform CS
        • tools
      • Helidon
      • ITSO & OEAF
      • Java Cloud
      • NodeJS Cloud
      • OIC – ICS
      • Oracle Cloud Native
      • OUG
    • railroad diagrams
    • TOGAF
  • xxRetired

My Other Web Content & Contributions

  • Amazon Author entry
  • API Platform
  • Dev Meetup (co-managed)
  • Fluentd Book
  • ICS Book Website
  • OMESA
  • Ora World
  • Oracle Community Directory
  • Packt Author Bio
  • Phil on Blogs.Oracle.com
  • Sessionize Profile

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 2,574 other subscribers

RSS

RSS Feed RSS - Posts

RSS Feed RSS - Comments

April 2023
M T W T F S S
 12
3456789
10111213141516
17181920212223
24252627282930
« Mar    

Twitter

  • Get all the details about the new enhancements to @Oracle Container Engine for Kubernetes, including Serverless… twitter.com/i/web/status/1…Next Tweet: 3 days ago
  • RT @TechWeekRO: With over 25 years of experience in the software industry, Phil Wilkins, Cloud Developer Evangelist at @Oracle, is coming t…Next Tweet: 3 days ago
  • SSH Key File Permissions blog.mp3monster.org/2023/03/28/ssh…Next Tweet: 4 days ago
  • Oracle's Assurance Service gives customers the proactive guidance they need to move their organization forward whil… twitter.com/i/web/status/1…Next Tweet: 4 days ago
  • Fraud affects many businesses and can be costly. But there’s a way to fight it. Scalable Machine Learning algorithm… twitter.com/i/web/status/1…Next Tweet: 4 days ago
Follow @mp3monster

History

Speaker Recognition

Open Source Summit Speaker

Flickr Pics

Pembroke CastleSeven Bridge Crossing
More Photos

    Social

    • View @mp3monster’s profile on Twitter
    • View philwilkins’s profile on LinkedIn
    • View mp3monster’s profile on GitHub
    • View mp3monster’s profile on Flickr
    • View philmp3monster’s profile on Twitch
    Follow Phil (aka MP3Monster)'s Blog on WordPress.com

    Blog at WordPress.com.

    • Follow Following
      • Phil (aka MP3Monster)'s Blog
      • Join 218 other followers
      • Already have a WordPress.com account? Log in now.
      • Phil (aka MP3Monster)'s Blog
      • Customize
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar
    Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
    To find out more, including how to control cookies, see here: Our Cookie Policy
    %d bloggers like this: