10 Monday Oct 2022
Posted in General, Technology
Tags
For a long time, I’ve tracked and read articles on Software Engineering Daily. We’ll day represents what is hopefully the first of many articles that we will write for them. The article is about the kind of people that make technical book authors, and the perception we have of authors – so if you’re interested check it out here.
Some more content on the subject of books …
05 Wednesday Oct 2022
Posted in development, Fluentd, General, Technology
Tags
12 Factor, 12 Factor App, conference, development, Grafana, JAX, logging, London, OpenSearch, Prometheus, Splunk, stdout
The 12 Factor App definition is now ten years old. In the world of software that is a long time. So perhaps it’s time to revisit and review what it says. As I have spent a lot of time around Logging – I’ve focussed on Factor 11 – Logging.
I have been fortunate enough to present at the hybrid JAX London conference on this subject. It was great to get out and see people at a conference rather than just with a screen and a chat console of online-only events.
You can see my presentation here:
Continue reading16 Friday Sep 2022
Posted in General, railroad diagrams, Technology
Tags
Having previously blogged about being a fan of Railroad diagrams (here) as a means to communicate language syntax, I have been asked about some of the ways details should be represented. I’ve looked around and not actually found an easy-to-read for ‘newbies’ guide to reading railroad diagrams. A lot of content either focuses on the generator tools or how the representation compares to BNF (Backus Naur Form) – all very distracting. So, I thought as an advocate, I should help address the gap (the documentation with TabAkins tool does a good job of explaining things, but its focus is on the features provided rather than understanding the notation).
The following table provides all the information to help you interpret the Railroad Diagrams, and create syntax representations.
Note: Clicking on the diagrams will result in a bigger version of the image being displayed.
| How to Interpret the Railroad | Railroad Representation | Code using tabakins tool |
|---|---|---|
| The start and end of an expression are shown with vertical bars. The expression is read from left to right following a path defined by the line(s). |  | Diagram( |
| Larger diagrams may need to be read both across and down the page to make it sensibly fit a page like this. |  | Diagram( Stack( |
| We can differentiate between literal values and ‘variables’ by shape. A square shape should denote a variable, and a lozenge represents a literal. I have to admit to sometimes getting these the wrong way around, but as long as the notation is used consistently in an expression, it isn’t too critical. In this example, we have replaced mp3monster with the name of a variable called domain-name. So if I set the variable domain-name = mp3monster then I’d read the same result. |  | Diagram( Stack( |
| We can make parts of an expression optional. This can be seen by following an alternate path around the optional element. In this case, we’ve made the domain-name optional. Assuming domain-name = mp3monster, we could get either: – This is a mp3monster blog OR – This is a blog |  | NonTerminal(' domain-name')), |
| We can represent optionality by having the flow split to multiple values. Those values could be either literal values or variables. In this case, we now have several possible names in the blog with the choices being mp3monster, phil-wilkins, cloud-native, or the contents of the variable domain-name. So the expression here could resolve to (assuming domain-name = something-else): – This is a mp3monster blog OR – This is a phil-wilkins blog OR – This is a cloud-native blog OR – This is a something-else blog It is typical for the option most likely to be selected to be the value that is directly in line with the choice. Here that would mean the default would be phil-wilkins |  | ' mp3monster', ' phil-wilkins ', ' cloud-native ', NonTerminal('domain-name')), |
| We can have variations on a choice where we can express the choice as being any of the options (one or more e.g. mp3monster and cloud-native) or all of the choices. These scenarios are differentiated by the additional symbols before and after the choice. |   | Diagram( Stack('any',' mp3monster', ' phil-wilkins ', ' cloud-native ', NonTerminal('domain-name')), |
| We can represent the looping with the inclusion of an additional literal(s) or variable(s) by having a second line from the right (exit) of a literal or variable and flowing back into the left (entry) of a literal or variable. Then in the loop of the flow below are the variable(s) or literal(s) that go around between each occurrence of the loop. If our variable was a list now i.e. domain-name = [‘ mp3monster’, ‘cloud-native’] then this would resolve to : This is a mp3monster and cloud-native blog |  | NonTerminal('domain-name'),[' and '])), |
| We can group literals and variables together with a label. These groupings are denoted with a dashed line and usually include some sort of label. In our example, we’ve grouped two literal values together and called that group the blog name. |  | Diagram( Stack(Terminal(' mp3monster '),['blog name']) |
All of these constructs can be combined so you can do things like making choices optional, and iterate over multiple variables and literals.
| Tips | Railroad Representation | Code using tabakins tool |
|---|---|---|
| Sometimes the number of options in a choice can get impractically large to show in a Railroad diagram. One way to overcome this is to show the first and last options and an ellipsis. We can then wrap it with a comment that directs the reader to the complete list of choices. This way the diagram continues to be readable – the most valuable part of the diagram and easy to locate the specific fine details. |  | Diagram( Stack( Terminal(‘This ‘), Terminal(‘ is ‘), Terminal(‘ a ‘), Group(Choice(1, ‘ mp3monster’, ‘ … ‘, ‘ cloud-native ‘), [‘ See Section X for full list’]), Terminal(‘ blog ‘)) ) |
15 Thursday Sep 2022
07 Wednesday Sep 2022
Posted in General
Tags
architecture, CICD, Containers, decision matrix, devops, jenkins, maven, reference architecture, stress test
When it comes to CI/CD deployments, something that doesn’t show very often in documentation is the pros and cons of running your worker nodes as containers in a Kubernetes environment or as (virtual) machines in a cloud environment.
02 Friday Sep 2022
Posted in General
Tags
Decision, decision matrix, factor, matrix, stress test, stressor, tree
We’ve all encountered decision trees at one time or another as a means to determine an answer based on a set of inputs. The problem with decision trees is that they are binary in nature. The answer is always yes or no.
But life is not binary. This is often best underlined by the classic joke of asking an architect what the right thing to do is, and you’ll get the answer ‘it depends‘. The art of being an architect is working out what the right trade-offs are when coming to a decision. This is fine, but to know all the factors (or trade-offs) that need to be considered requires a wealth of experience (a rare and expensive asset) or someone capturing these insights in a consumable manner that enables people who will understand how to use imparted wisdom, rather than try to blindly follow binary decisions.
This is where decision Matrices, or as I was introduced to them ‘Stress Tests’ – so called because the matrix is made up of a factor or ‘stressor’ are tested against one or more options. Each slot in the matrix can either be marked as yes/no or scored on the range (for example 1-5) on alignment to meeting the stressor. If you’re doing a product selection then the options are the different products. But equally the options could be an architectural pattern.
This is probably best illustrated. So the following Matrix looks at different programming languages against a range of different stressors. This isn’t a realistic case, but sufficient to help convey the idea. (The assessments in this matrix are not comprehensively researched, so feel free to argue)
| Java | Python | Ruby | C++ | |
| Single binary for multiple OS and architectures | 5 | 5 | 4 1 | 1 |
| Language evolves quickly to provide standardized libraries for common needs | 5 | 5 | 4 | 3 |
| Suitable for scripting and application development | 42 | 5 | 4 | 1 |
| Readily available skills | 4 | 5 | 33 | 33 |
As you can see to help use the matrix we can add elaboration notes. To use the matrix we simply determine which stressors are most important or not and then score the different options. So if I wanted to use the matrix to determine which language is most suitable for developing a deployment utility then we know from the use case performance is less critical, portability is important, and skills should outweigh language evolution. Based on that the answer should resolve to Python failing that Java would be good enough.
While our question and its outcome is fairly obvious, in more nuanced situations the stress test gives several benefits:
Through the use of the stress test we can translate the ‘it depends‘ to a more concrete but rationalized decision.
19 Friday Aug 2022
Posted in General
My blogging activities have expanded a little bit further. Oracle staff (particularly those involved with Product Management) and the Ace community have been contributing to a Medium publication called Oracle Devs (go here). The content here is 1st rate, technically comprehensive, and from some exceptional technical people. So I’d recommend tracking it if you aren’t already and I feel very honored to have several posts already included, which can be found at:
Railroad diagrams for OCI Policies
Those that follow me here, not o worry, this is the center of the universe for my blogging activity, and I will tie back anything that is unique to Medium o posts here.
In addition to that, I do also have a profile on the official Oracle Blog (here), and a range of posts have also been picked up via the PaaSCommuniy which haven’t been directly linked to my Oracle blog profile but can be found with this link
17 Wednesday Aug 2022
Posted in Cloud, development, Fluentd, General, Oracle, Technology
Tags
Cloud, demo, Fluentd, GitHub, Log Analytics, log simulator, OCI, OpenSearch, Oracle, Ubuntu
One of the areas I present publicly is the use of Fluentd. including the use of distributed and multiple nodes. As many events have been virtual it has been easy to demo everything from my desktop – everything is set up so I can demo things very easily. While doing this all on one machine does point to how compact and efficient Fluentd is as I can run multiple instances concurrently it does undermine distributed capabilities somewhat.
Add to that I now work for Oracle it makes sense to use OCI resources. With that, I have been developing the scripts to configure Ubuntu VMs to set up the demo environments installing Ruby, Fluentd, and various gems needed and pulling the relevant configurations in. All the assets can be found in the GitHub repository https://github.com/mp3monster/logging-demos. The repository readme includes plenty of information as well.
While I’ve been putting this together using OCI, the fact that everything is based on Ubuntu should mean it can be run locally on VMs, WSL2, and adaptable for MacOS as well. The environment has been configured means you can still run on Ubuntu with a single node if desired.
As the demo will typically be run on OCI we can not only run the demo with a multinode setup, we have extended the setup with several inclusion files so we can utilize OCI services OpenSearch and OCI Log Analytics. If you don’t want to use these services simply replace the contents of several inclusion files including files with the contents of the dummy_inclusion.conf file provided.
The configuration works by each destination having one or two inclusion files. The files with the postfix of label-inclusion.conf contains the configuration to direct traffic to the respective service with a configuration that will push log events at a very high frequency to the destination. The second inclusion file injects the duplication of log events to each service. The inclusion declarations in the main node Fluentd config file references an environment variable that should provide the path to the inclusion file to use. As a result, by changing the environment variable to point to a dummy file it becomes possible o configure out the use of one of the services. The two inclusions mean we can keep the store declarations compact and show multiple labels being used. With the OpenSearch setup, we have a variant of the inclusion file model where the route inclusion can reference the logic that we would use in the label directly within the sore declaration.
The best way to see the use of the inclusions is to experiment with setting the different environment variables to reference the different files and then using the Fluentd dry-run feature (more on this in the book).
The setup script performs a number of tasks including:
OpenSearch setup is documented in a tutorial here, and a Reference Architecture at the time of writing there isn’t a one-click deploy Terraform available in the Oracle Reference Architecture library on GitHub.
Currently, the setup for OpenSearch means manually adding the node1 index into the configuration.
Feeding the log analytics service is a more complex process to set up as the feeds need to have metadata about the events being ingested. The downside is the configuration effort is greater, but the payback is that it becomes easier to extract meaningful information quickly because the service has a greater understanding of the content. For example, attributing the logs to a type of source means the predefined or default log formats are immediately understood, and maximum meaning can be retrieved from the log event.
Going to OCI Log Analytics does cut out the need for the Connections hub, which would allow rules and routing to be defined to different OCI services which functionally can help such as directing log events to PagerDuty.
There are a few things we’re planning to do with the demo:
05 Friday Aug 2022
Posted in APIs & microservices, development, General, Technology
Tags
API, architecture, code, GraphQL, gRPC, Oracle, streaming, subscriptions
Yesterday I was fortunate enough to participate in the Dev Innovation summit part of the World Festival virtual conference.
The presentation took a look at how Streaming APIs offer an alternative to API polling and the considerations needed when adopting streaming.
Continue reading26 Tuesday Jul 2022
Posted in Cloud, General, Technology
Infrastructure as Code (IaC) should be treated the same way as any other code. That is to say that we should be considering configuration management, testing, regression, code quality, and coverage. We should be addressing these points for the same reasons we address them with our application code. Such as ensuring that we don’t introduce bugs as things evolve and develop, ensuring that the code is maintainable over a prolonged period etc.
The problem is that the only real way to test IaC is to run it. Particularly with the likes of Terraform where it is largely declarative rather than containing a lot of logic. This point is nicely conveyed by Yevgeniy Brikman’s presentation (below)
The presentation goes on to illustrate Terratest which has the look and feel of JUnit or any other xUnit test framework. Terratest is implemented in Golang, But to be honest, given the nature of Terraform ( largely declarative meaning it enables ideas of composition and not sophisticated logic) it means the writing of the tests isn’t going to demand anything clever like how to achieve polymorphic behavior through Go’s type structures.
While Yevgeniy focussed on testing by invoking an application on the infrastructure deployed something we’ve described though our Platform Test logic (more here). You may wish to test things further by interrogating infrastructure components. For example, do I have the right number of nodes in a dynamic group or are container or server logs going into the cloud monitoring services.
Performing such checks is very easy with OCI as it provides a Golang SDK making it very easy to write tests that can call the OCI APIs and interrogate the setup. Better still when considering whether the Terraform configuration will behave correctly to support dynamic/auto-scaling can be done easily without modifying the Terraform configurations as part of the Terratest logic can include Go API calls to temporarily modify scaling triggers or invoking code that can stimulate OCI dynamic features.
There is an interesting question to be considered. There is a point of separation between when to use Terraform (or Pulumi and others for that matter) and tools better suited to application deployment and configuration like Ansible and Chef. Therefore should we separate the testing of these details? Maybe I am too purist but seeing local and remote execs in Terraform as these actions are very opaque and can be used to conceal things or unwittingly depend on the way Terraform handles its dependency graph.
Of course, Ansible has its test framework ansible_test and has the means to measure test coverage. So one possibility is to treat Ansible as a separate module, independently test it, and then integrate its use in the wider picture of deploying infrastructure.
Phil (aka MP3Monster)'s Blog 
You must be logged in to post a comment.