When configuring Fluentd we often need to provide credentials to access event sources, targets, and associated services such as notification tools like Slack and PagerDuty. The challenge is that we don’t want the credentials to be in clear text in the Fluentd configuration.
Using Env Vars
In the Logging In Action with Fluentd book, we illustrated how we can take the sensitive values from environment variables so the values don’t show up in the configuration file. But, we’ve seen regularly the question of how secure is this, can’t the environment variable be seen by everyone on that machine?
The answer to this question comes down to having a deeper understanding of how environment variables work. There is a really good explanation here. The long and short of it is that environment variables can only be seen by the process that creates the variable and any child process will receive a copy of the parent’s variables.
This means that if we create the variable in a shell, only that shell and any processes launched by that shell can see the environment variable. So as long as we don’t set variables up as part of a system-level configuration then we already have a level of security. So we could wrap the start of Fluentd with a script that sets the environment variables needed. Then everything launches that script.
In a previous blog (here) I wrote about the structure and naming of assets to be applied to OCIR. What I didn’t address is the interesting challenge of what if my development machine has a different architecture to my target environment. For example, as a developer, I have a nice shiny Mac Book Pro with the M1 chipset which uses an ARM architecture. However, my target cloud environment has been built and runs with an AMD64 chipset? As we’re creating binary images it does raise some interesting questions.
As we’re creating our containers with Docker, this addresses how to solve the problem with Docker. Other OCI Compliant containers will address the problem differently.
Buildx
Buildx is a development feature in Docker which makes use of a cross-platform build capability. When using buildx we can specify one or more build platform types. These are specified using the –platform parameter. In the code below we use it to define the Linux AMD64 architecture mentioned (linux/amd64). But we can make the parameter a comma-separated list targeting different platform types. When that is done, multiple images will be built. By default, the build will happen in sequence, but it is possible to switch on additional process threads for the Docker build process to get the build process running concurrently.
Unlike the following example (which is only intended for one platform, if you are building for multiple platforms then it would be recommended that the name include the platform type the image will work for. For production builds we would promote that idea regardless, just as we see with installer and package manager-related artifacts.
If you compare this version of the code to the previous blog (here) there are some additional differences. Now I’ve switched to setting the target tag as part of the build. As we’re not interested in hanging onto any images built we’ve included the target repository in the build statement. Immediately push it to OCIR, after all the images won’t work on our machine.
Oracle’s product portfolio is significant, from databases (obviously) to GraalVM to a cloud platform capable of competing with GCP, AWS, and Azure. This means locating the Oracle-provided plugins, or community ones can get messy. Depending on your perspective Oracle Developer Plugins could relate to Java and GraalVM or Oracle Database.
As broad as the portfolio, is the Oracle details regarding the plugins. So the following two tables represent what we’ve identified as Oracle-provided tooling, and the second table of plugins we’ve used when working on Oracle-based solutions from the community.
Name / Plugin Search
Description / Additional Details
Related resource links
search:Oracle Labs This will return all the Oracle plugins related to GraalVM
SuiteCloud Extension for Visual Studio Code is part of the SuiteCloud Software Development Kit (SuiteCloud SDK), a set of tools to customize your NetSuite accounts.
Github actions is a means by which actions like commits to github trigger external infrastructure to perform actions such as creating application binaries.
My eldest son is studying computer science at school. As part of that course, like most educational settings they’re using Python to teach programming skills. Having sat with my son to help him get to grips with the coding the different ways of looping, implementing conditions, variable scopes and so on it has been interesting to see that school student level programming books take developers through very very simple steps with the rewards of success immediate but of limited value.
The challenge and concern, with this, is the step from this to delivering something meaningful, that continues to provide a sense of reward, while also something that is closer to real-world work.
This is where a small project I had comes into play. I have for many years been using an app called Orange CD catalog to record all the details of my music collection. The only problem is that it is a thick desktop app and the web solution called Racks and Tags hasn’t seen much development from what I can see for a long time and by current user experience looks archaic. As you may have guessed our pet project is to provide a web interface for the exported data from the Orange CD catalog. You can see the code in my GitHub repo here). I’ve carefully structured the code so the logic of processing the data (i.e. loops and conditions using data structures – using Python’s XMLTree library, which behaves a lot like lists – helping with basic learning). The web serving tier is implemented with Flask using simple annotations. This provides a nice opportunity to talk about the use of HTTP. The Jinja templates are also kept separate, so we don’t get bogged down with HTML, which can be a bit messy (IMHO).
Orange CD Catalogue
We’ll deploy the solution to our little Raspberry Pi, so it can be used anywhere in our network. Ultimately we could use it to stream the albums I’ve ripped to MP3. If my son wants/needs to learn a little bit about databases – then migrating the data into something like SQLLite becomes a simple development opportunity.
This does mean some prep work and including part of the solution, so something works very quickly. For me that’s fine, my Python web skills could benefit from some work and Jinja2 is new to me. But I could take the project embed suitable security and deploy it to my free Oracle cloud instance as a container into K8s. Those bits I won’t be sharing (minimize the risk of someone wanting to test how well I patch code etc).
Feel free to take the code and use it for yourself, extend, etc. But you don’t have to use my idea. The important thing is the idea is going to interest your child. But keep it simple. for example, if your child is a gamer, then perhaps you generate a simple data extract from the gaming servers to capture player performance, then your app does simple things like searching the content, sorting it, and providing a vanilla UI. The secret I think is each bit of the project is simple steps and not over well with heaps of code. Provide results quickly for example just search for an album by its title, and get the results into a webpage quickly.
I’ve had some time to catch up on books I’d like to read, including Kubernetes Best Practises in the last few weeks. While I think I have a fair handle on Kubernetes, the development of my understanding has been a bit ad-hoc as I’ve dug into different areas as I’ve needed to know more. This meant reading a Dummies/Introduction to entry style guide would, to an extent, likely prove to be a frustrating read. Given this, I went for the best practises book because if I don’t understand the practises, then there are gaps in my understanding still, and I can look at more foundation resources.
As it goes, this book was perfect. It quickly covered the basics of the different aspects of Kubernetes helping to give context to the more advanced aspects, and the best practices become almost a formulated summary in each section. The depth of coverage and detail is certainly very comprehensive, explaining the background of CNI (Container Network Interface) to network-level security within Kubernetes.
The book touched upon Service Meshes such as Istio and Linkerd2 but didn’t go into great depth, but again this is probably down to the fact that Service Mesh ideas are still maturing, and you have initiatives like SMI (Service Mesh Interface still in the CNCF’s sandbox).
In terms of best practices, that really stood out for me:
Use of Taints and Tolerations for refined control of pod deployment (Allowing affinity to be controlled to optimise resilience, or direct types of pod deployment to nodes with specialist capabilities such as GPU).
There are a lot more differences and options then you might realize in terms of ingress controller capabilities, so take time to identify what you may need from an ingress controller.
Don’t forget pods can be scaled vertically with the VPA (Vertical Pod Autoscaler)as well as horizontally through the HPA.
While using a managed persistence service will make statement storage a lot easier, stateful sets will give you a very portable solution.
As with a lot of technical books I read. As I go through the book I build up a mind map of what I think are the key points. Doing so leaves me with a resource I can use as a quick reference, but creating the mind map helps reinforce the learning. So here is the mind map …
Low level aka detailed design documentation has been an interesting point of debate. We range from the agile manifesto which states focus should be on working code over documentation and people using this as an argument for not producing documentation. On the other end of the spectrum as someone working for a large SI, documentation is more often than not contractually binding and accurate docs are key when taking work from a different supplier organization.
It is clear that documentation is an essential element. But I do agree with the agile manifesto, a business operates on its software, not documents, although the docs help us keep the software maintained and running.
How do we balance the age-old conflicts of …
Documents get out of date because they are kept separate from the implementation
Documents, particularly when rushed don’t provide the information necessary
Document templates having sections used a tickboxes rather than guide rails
Making sure we’re working with the most upto date document
Possibilities
One of the key issues for documents getting out of date is a compound issue of accessibility, visibility and ease of maintenance. These compound to separate the documentation from the reality of code and configuration.
This can be eased by bringing documentation to being ‘physically’ closer to the code as we often see with readme markdown files on GitHub for example. But we can get closer with quality code commenting, particularly for each package and module. Just about every code or notation format had its own document generator from well-proven Javadoc to Terradoc for Terraform. To illustrate my point here are a couple of examples:
If done well the documentation can be generated and deliver the right information. It would mean in structured change management the change task for the code includes the documentation. The ideas behind combining code and documentation can be seen with good API Blueprints.
When you still need to produce publishable documents, you have the opportunity to stitch multiple class and package generated docs together using tools such as pandoc. Arguably it would be the developer’s job to establish the pandoc configuration file (Documentation as Code).
You can add to this if done carefully, by adding diagrams such as UML representations. Importantly this process can generate representations that include lots of detail that would be noise to the key representation (time for tools like Sparx to support annotations that can give hints as to what to show in a generated model).
Pitfalls
The biggest risks of this approach are:
People paying lip service to documenting code, or using the argument that agile means no documentation (an age old misrepresentation).
Comments reflect the code correctly
Assuming the documentation will be clear because it is writte6x n
These pitfalls could be in theory be addressed through some smarts such as comparing the volume documentation generated against the number of lines of code and code complexity metrics.
But like many things, good culture and good application of principles are essential.
Exploring further
There are growing dedicated resources in this space, check out:
Today was the first run of some new presentation material looking at the use of GitHub Actions using Runners deployed on OCI Free Tier. The presentation was actually physical rather than virtual which was after 2 years of virtual presenting, rather refreshing. Not to mention the UKOUG hosted the event at the Oval Cricket ground, which made for an interesting venue. The example configuration is included in our GitHub OCI Utilities repository (as we use this solution to help validate and test our development work).
The presentation itself (which includes screenshots of the setup of a simple Action and runner) is here, note I have disconnected my Runners, but you will be able to see the Action configuration but if you try to trigger activity through my repository then nothing will happen.
Through my day job at Capgemini UK I have regular dealings with Flexagon as we use their FlexDeploy product with a number of customers (as we do recommend the product). Recently I was invited to contribute to a paper being developed by the Flexagon team. The fruits of the collaboration are now available at https://info.flexagon.com/ebook/devops-for-developers. The Flexagon team have done a nice job with it, and we’d recommend taking a look if you’re wanting to know more about DevOps and CI/CD.
The paper is completely free of sales pitch – completely devoid of product references which makes for a good read.
This isn’t the first time I’ve written about the Oracle Cloud SDK (check here), but it seems rather fitting, as some of the utilities I’ve been working on are open to the community, and #JoelKallmanDay is all about community. If you’d like to know more about #JoelKallmanDay then checkout Tim Hall’s blog here.
Oracle have provided a very rich API and then overlaid it with a number of SDKs in Python, Java etc. The SDKs immediately remove the work of creating connections and correct payloads. Taking the Python SDK for example, all I need to do is create a standard configuration file with all the necessary connection properties to my OCI instance. Then it’s simply a case of creating the correct Python object for the correct group of services wanted. Then it’s down to populating the object attributes. This is the illustration of exactly what a good SDK does. I can lean on my IDE to use the correct set and get operators. The code for establishing a connection is done for me.
What I’ve found most striking is the level of consistency in the methods provided by the SDK regardless of the service. This makes it very easy to develop functionality without needing to check every API before I can write any code. it would be easy to say, so what. But when you look at the breadth of the OCI services it becomes more impressive.
The convenience doesn’t end there. Rather than having to run your utilities from a local command line (Python means we’re pretty much OS agnostic), the Oracle Cloud shell is preconfigured with Python, OCI SDK, GitHub and FTP server and basic Linux text editors. The all amounts to the fact that you can use your scripts/tools from within the web UI of OCI. Edit your credentials file locally, push and pull any changes to the scripts from the shell and any Git repo such as GitHub.
With this insight, we just need to build that catalogue of accelerator tools to make those repetitive processes just a little easier. For example ensuring that when you tear down your manually created services all interlinked entities are deleted first (which can be troublesome with policies, groups, compartments and so on).
The log simulator we’ve built and written about in the past has had a release made that lines up with the Logging In Action book (v0.1). I am now continuing to add improvements on the main line. Not best Git branching practice, but as I’m working on this solo it doesn’t represent a problem.
If you expect multi line events all you need to do, is add to the properties file a name value pair, with the name FIRSTOFMULTILINEREGEX and the value is a Java/Groovy regular expression which can be used to determine if a line in a log entry is the 1st line in a new log. Then all subsequent log lines are appended to the previous line until a line identifies as a new log entry. The log entry will be written with newline characters in the same place as the read.
In addition to this if the synthetic log events need to be set to be new line then using the ALLOWNL property to be set to true will result in any new line escape sequences (\n) to be made into proper new lines in the output.
The details are all included in the documentation in GitHub.
You must be logged in to post a comment.