Category Archives: General

All things general

OCI Notifications through the LogSimulator

04 Friday Nov 2022

Posted by in development, General, logsimulator, Oracle, Technology

≈ Leave a comment

Tags

, , , , , , ,

We’ve been busy putting together a number of Oracle Architecture Center assets over the last week. This has included building LogSimulator extensions that can either be run in a very simple manner using just a single file, but limited in the payloads that can be sent to OCI (if you take the appropriate custom file from the LogSimulator you do need to make one minor tweak. But the code has also been added to the Oracle GitHub repository here in a manner that doesn’t require the full tool. There of course a price to pay for the simplified implementation. This comes in the form of the notifications being sent and received being hardwired into the code rather than driven through the insulator’s configuration options.

The decoupling has been done by implementing the interface for the custom methods in a class without the implements declaration, and then we extend the base class and apply the implements declaration at that level.

While notifications could take log events, it is more suited to JSON payloads. But as the simulator can tailor the content being sent using some formatting, it does not care if the provided events to send are pre-formatted as JSON objects making it an easy tool to test the configuration of OCI Notifications.

Unit testing as well

In addition to the new channel, as previously mentioned we have been making some code improvements. To support that we have started to add unit tests, and double checking code will compile under Java. To keep the dependencies down we’re making use of Java assert statements rather than a pretty JUnit. But the implementation ideas are very similar. As the tests use Java asserts the use of asserts does need to be enabled in the command line; for example:

Groovy test.groovy -enableasserts

JMESPath is represented using Railroad diagrams

31 Monday Oct 2022

Posted by in development, General, railroad diagrams, Technology

≈ Leave a comment

Tags

, , , , , ,

JMESPath is a mature syntax for traversing and manipulating JSON objects. The syntax is also supported with multiple language implementations available through GitHub (and other implementations exist). As a result, it has been very widely adopted; just a few examples include:

  • Azure CLI
  • AWS CLI and Lambda
  • Oracle Cloud WAF
  • Splunk

As the syntax is very flexible and recursive in its use following the documented notation can be a little tricky to start with. So following the syntax can be rather tricky. The complete definition runs to 97 lines, of which 32 lines focus on the syntactical structure. The others describe the base types such as numbers, characters, accepted escaped characters, and so on. Nothing wrong with this, as the exhaustive definition is necessary to build parsers. But for the majority of the time it is those 32 lines that we need to understand.

As the expression goes – ‘a picture says a thousand words’, there might not be a thousand words, but there is enough to suggest a visual representation will help. Even if the visual only helps us traverse the use of the detailed syntax. So we’ve use our favoured visual representation – the railroad diagram and the tool produced by Tab Akins to create the representation. We’ve put the code and created images for the syntax in my GitHub repository here, continuing the pattern previously adopted.

Here is the resulting diagram …

To make it easy to trace back to the original syntax document we’ve included groupings on the diagram that have names from the original speciofication.

Parts of the diagram make the expressions look rather simple, but you’ll note that it is possible for the sections to be iterative which allows for the expression to traverse a JSON object of undefined depth. But what can be really challenging is that an in many areas it is possible to nest expressions within expressions. Visually there is no simple way to represent the expression possibilities of this in a linear manner. Other than be clear about where the nesting can take place.

LogSimulator New Feature – Custom Targets with OCI Logging example

14 Friday Oct 2022

Posted by in Cloud, development, General, logsimulator, Oracle, Technology

≈ Leave a comment

Tags

, , , , ,

Those who have been using my Logging in Action book will know that to help test the configuration of monitoring tools including Fluentd we have built a LogGenerator that can very easily play and replay logging events into a variety of destinations and formats. all written in Groovy to make the utility easy to run as a script and extend without needing to set up a proper Java development environment.

With the number of different destinations built into the script and the logic to load the source log events and format them the utility is getting rather large for a single file. Rather than letting it continue to grow as we add more destinations to pump log events too, I’ve extended the implementation so you can point to a Groovy file that implements the logic to send the log events. It only requires three simple methods to be implemented.

To demonstrate the feature we have created a custom extension and fully documented it. The extension allows you to send log events to the OCI Logging service. This includes an optional crude aggregation mechanism as sending individual log events is a little inefficient over REST. By doing this we can send synthetic or playback logs as if we’re an application in real-life to ensure that any alerting or routing for the logging works properly before we get anywhere production and do not need to run the application and induce error events.

Beyond this, we’re also thinking about creating a plugin to fire log events at Prometheus so we can send events using the Prometheus pushgateway. As a result, we can tune Prometheus’ configuration.

More improvements – refactoring the existing code

We will refactor the existing code to use the same approach which should make the code more maintainable, but the changes won’t stop the utility from working as it always has (so we won’t break out the existing output channels from the core).

We have also started to improve the code commenting – so hopefully it will make the code a bit more navigable.

Practical Steps when it comes to writing a technical book

11 Tuesday Oct 2022

Posted by in Books, General, manning, Oracle Press, Packt, Technology

≈ 1 Comment

Tags

, , ,

Following my article on Software Engineering Daily, here are some practical things that will help you if you’re considering taking on a technical book project.

Identifying a Publisher

While it is easy to self-publish today. The recognition comes from having worked with a traditional publisher as they have processes that ensure a level of quality. Not all publishers are equal, and some publishers are attributed with more prestige than others. In addition to this, some publishers are willing to take a risk on a subject and/or author. Have a look at the titles already published, and whether there are any publishers you can connect to.

When comes to contacting the publishers, most of their websites will have a page for recruiting authors. Some are easier to find than others. Here are a couple:

If, or when you get to talk to a publisher it is worth ensuring you understand how their editorial process works and what is expected from you? Plus what happens if you find yourself in the position of not being able to work to the original schedule. Day-to-day work can get in the way which you hadn’t expected.

Continue reading →

Contributing to Software Engineering Daily

10 Monday Oct 2022

Posted by in General, Technology

≈ Leave a comment

Tags

, , , ,

For a long time, I’ve tracked and read articles on Software Engineering Daily. We’ll day represents what is hopefully the first of many articles that we will write for them. The article is about the kind of people that make technical book authors, and the perception we have of authors – so if you’re interested check it out here.

Some more content on the subject of books …

Is The 12 Factor App right about Logging?

05 Wednesday Oct 2022

Posted by in development, Fluentd, General, Technology

≈ Leave a comment

Tags

, , , , , , , , , , ,

The 12 Factor App definition is now ten years old.  In the world of software that is a long time. So perhaps it’s time to revisit and review what it says.  As I have spent a lot of time around Logging – I’ve focussed on Factor 11 – Logging.

I have been fortunate enough to present at the hybrid JAX London conference on this subject. It was great to get out and see people at a conference rather than just with a screen and a chat console of online-only events.

You can see my presentation here:

Continue reading →

Interpreting Railroad diagrams

16 Friday Sep 2022

Posted by in General, railroad diagrams, Technology

≈ Leave a comment

Tags

, , , ,

Having previously blogged about being a fan of Railroad diagrams (here) as a means to communicate language syntax, I have been asked about some of the ways details should be represented. I’ve looked around and not actually found an easy-to-read for ‘newbies’ guide to reading railroad diagrams. A lot of content either focuses on the generator tools or how the representation compares to BNF (Backus Naur Form) – all very distracting. So, I thought as an advocate, I should help address the gap (the documentation with TabAkins tool does a good job of explaining things, but its focus is on the features provided rather than understanding the notation).

Reading Railroad Diagrams

The following table provides all the information to help you interpret the Railroad Diagrams, and create syntax representations.

  • If you’re only interested in understanding the notation, read just the first two columns.
  • If you want to see examples of how to create the diagram, then the 3rd column will help.

Note: Clicking on the diagrams will result in a bigger version of the image being displayed.

How to Interpret the RailroadRailroad RepresentationCode using tabakins tool
The start and end of an expression are shown with vertical bars.
The expression is read from left to right following a path defined by the line(s).		Diagram(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
Terminal(' mp3monster '),
Terminal(' blog ')
)
Larger diagrams may need to be read both across and down the page to make it sensibly fit a page like this.Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
Terminal(' mp3monster '),
Terminal(' blog '))
)
We can differentiate between literal values and ‘variables’ by shape. A square shape should denote a variable, and a lozenge represents a literal.
I have to admit to sometimes getting these the wrong way around, but as long as the notation is used consistently in an expression, it isn’t too critical. In this example, we have replaced mp3monster with the name of a variable called domain-name. So if I set the variable domain-name = mp3monster then I’d read the same result.		Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
NonTerminal(' domain-name'),
Terminal(' blog '))
)
We can make parts of an expression optional. This can be seen by following an alternate path around the optional element.
In this case, we’ve made the domain-name optional. Assuming domain-name = mp3monster, we could get either:
– This is a mp3monster blog OR
– This is a blog		Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
Optional(
NonTerminal(' domain-name')),
Terminal(' blog '))
)
We can represent optionality by having the flow split to multiple values. Those values could be either literal values or variables. In this case, we now have several possible names in the blog with the choices being mp3monster, phil-wilkins, cloud-native, or the contents of the variable domain-name. So the expression here could resolve to (assuming domain-name = something-else):
– This is a mp3monster blog OR
– This is a phil-wilkins blog OR
– This is a cloud-native blog OR
– This is a something-else blog
It is typical for the option most likely to be selected to be the value that is directly in line with the choice. Here that would mean the default would be phil-wilkins		Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
Choice(1,
' mp3monster',
' phil-wilkins ',
' cloud-native ',
NonTerminal('domain-name')),
Terminal(' blog '))
)
We can have variations on a choice where we can express the choice as being any of the options (one or more e.g. mp3monster and cloud-native) or all of the choices. These scenarios are differentiated by the additional symbols before and after the choice.
Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
MultipleChoice(1,
'any',' mp3monster',
' phil-wilkins ',
' cloud-native ',
NonTerminal('domain-name')),
Terminal(' blog '))
)
We can represent the looping with the inclusion of an additional literal(s) or variable(s) by having a second line from the right (exit) of a literal or variable and flowing back into the left (entry) of a literal or variable. Then in the loop of the flow below are the variable(s) or literal(s) that go around between each occurrence of the loop. If our variable was a list now i.e. domain-name = [‘ mp3monster’, ‘cloud-native’] then this would resolve to :
This is a mp3monster and cloud-native blog		Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),
OneOrMore(
NonTerminal('domain-name'),
[' and '])),
Terminal(' blog ')
)
We can group literals and variables together with a label. These groupings are denoted with a dashed line and usually include some sort of label. In our example, we’ve grouped two literal values together and called that group the blog name.Diagram( Stack(
Terminal('This '),
Terminal(' is '),
Terminal(' a '),),
Group(Sequence(
Terminal(' mp3monster '),
Terminal(' blog ')),
['blog name'])
)

All of these constructs can be combined so you can do things like making choices optional, and iterate over multiple variables and literals.

Tips & Tricks to Consider

TipsRailroad RepresentationCode using tabakins tool
Sometimes the number of options in a choice can get impractically large to show in a Railroad diagram. One way to overcome this is to show the first and last options and an ellipsis.
We can then wrap it with a comment that directs the reader to the complete list of choices.
This way the diagram continues to be readable – the most valuable part of the diagram and easy to locate the specific fine details.		Diagram( Stack(
Terminal(‘This ‘),
Terminal(‘ is ‘),
Terminal(‘ a ‘),
Group(Choice(1,
‘ mp3monster’,
‘ … ‘,
‘ cloud-native ‘), [‘ See Section X for full list’]),
Terminal(‘ blog ‘))
)

Stop polling, Let’s go Streaming the podcast

15 Thursday Sep 2022

Posted by in General, Podcasts

≈ Leave a comment

Tags

, , , ,

I was fortunate enough to be invited to join LogRocket‘s Podcast (PodRocket) to discuss some of the insights and considerations relating to API Streaming that I presented at a reference conference. To hear more go checkout :

If you’d like to see more from the presentation, go here.

CI/CD worker nodes as virtual machines or K8s Containers?

07 Wednesday Sep 2022

Posted by in General

≈ Leave a comment

Tags

, , , , , , , ,

When it comes to CI/CD deployments, something that doesn’t show very often in documentation is the pros and cons of running your worker nodes as containers in a Kubernetes environment or as (virtual) machines in a cloud environment.

Containerized CI/CD
Continue reading →

Decision Matrix aka ‘Stress Test’ as a vehicle to make decisions easier

02 Friday Sep 2022

Posted by in General

≈ 2 Comments

Tags

, , , , , ,

We’ve all encountered decision trees at one time or another as a means to determine an answer based on a set of inputs. The problem with decision trees is that they are binary in nature. The answer is always yes or no.

From http://www.stodolkiewicz.com/2020/04/27/decision-trees-introduction/

But life is not binary. This is often best underlined by the classic joke of asking an architect what the right thing to do is, and you’ll get the answer ‘it depends‘. The art of being an architect is working out what the right trade-offs are when coming to a decision. This is fine, but to know all the factors (or trade-offs) that need to be considered requires a wealth of experience (a rare and expensive asset) or someone capturing these insights in a consumable manner that enables people who will understand how to use imparted wisdom, rather than try to blindly follow binary decisions.

This is where decision Matrices, or as I was introduced to them ‘Stress Tests’ – so called because the matrix is made up of a factor or ‘stressor’ are tested against one or more options. Each slot in the matrix can either be marked as yes/no or scored on the range (for example 1-5) on alignment to meeting the stressor. If you’re doing a product selection then the options are the different products. But equally the options could be an architectural pattern.

This is probably best illustrated. So the following Matrix looks at different programming languages against a range of different stressors. This isn’t a realistic case, but sufficient to help convey the idea. (The assessments in this matrix are not comprehensively researched, so feel free to argue)

JavaPythonRubyC++
Single binary for multiple OS and architectures554 11
Language evolves quickly to provide standardized libraries for common needs5543
Suitable for scripting and application development42541
Readily available skills453333
  1. Ruby has several variants such as JRuby, Ruby with a C core – this has the potential to impact
  2. Use of Shebang or overlaying with Groovy makes Java more usable as a scripting language
  3. Not so commonly taught in educational settings which typically favor Python and Java

As you can see to help use the matrix we can add elaboration notes. To use the matrix we simply determine which stressors are most important or not and then score the different options. So if I wanted to use the matrix to determine which language is most suitable for developing a deployment utility then we know from the use case performance is less critical, portability is important, and skills should outweigh language evolution. Based on that the answer should resolve to Python failing that Java would be good enough.

While our question and its outcome is fairly obvious, in more nuanced situations the stress test gives several benefits:

  • The decision process becomes transparent and easily communicated.
  • The hard work is in developing the stressors and researching the different options in terms of how they fit with stressors.
  • The matrix can be re-used, to help achieve other decisions. What if I was looking at the most suited solution for running data analytics that can maximize GPUs?

Through the use of the stress test we can translate the ‘it depends‘ to a more concrete but rationalized decision.