Tag Archives: Git

A Fast (and Dirty) Way to Publish API specs

03 Monday Jun 2024

Posted by in APIs & microservices, General, Technology

≈ Leave a comment

Tags

, , , , , , , , , , ,

The API specs created using Open API Specification (OAS) and ASyncAPI specification aren’t just for public API consumption. In today’s world of modular component services that make up a business solution, we’re more than likely to have APIs of one sort or another. These need documenting, perhaps not as robustly as those public-facing ones, but the material needs to be easily accessible.

Spotify’s contribution to the CNCF—Backstage is a great tool for sharing development content, particularly when your document and code repository is at least git-based if not GitHub (you move away from this or don’t easily have permissions to configure application authentication, you can still work with Backstage, but your workload will grow a lot). There is no doubt that Backstage is a very powerful, information-rich product. But that does come at the cost of needing lots of configuration, the generation of metadata descriptors additional to the APIs to be cataloged, etc. All of these can be a little heavy if you’re using Backstage as a low-cost API documentation portal that might fill the gaps that your corporate wiki/doc management (Confluence/SharePoint) solution can’t support (it is one of the very, very few open-source options that can support both OAS and AsyncAPI reader friendly API rendering tools).

We could, of course, adopt the approach of there are free VS Code plugins that can render the friendly views of APIs, so just perform a git pull (or copy the API specs from a central location) to give the nice visualization. This is fine, but the obligation is now on the developer to ensure they have the latest version of the API spec and that they are using VSCode – while it is very dominant as an IDE – not everyone uses it, particularly if you’re working with low code tooling.

There is a fast and inelegant solution to this if you’re not in need of nice features such as attribute-based search and sorting, etc. Both the Open API Specification and the Async API communities have built command line-based renderers that will read your API specification (even if the schema is spread across multiple files) and generate HTML (an index.html file), CCS, and JavaScript renderings that you see in many tools (hyperlinked, folding, with code and payload examples of the API).

So, we need to grab the YAML/JSON specifications and run them through the tool to get the presentation formatting. You do need to get the specs, but we can easily script that with a bit of shell script that retrieves/finds the relevant files from a repository and then runs the CLI utility on the files.

We want to bring the static content to life across the network for developers. So, on a little server, we can host this logic, plus an instance of Apache, IIS, or Nginx if you’re comfortable with one of the industrial superpower web servers. Or use a spin-off project from the Chrome Server called the Simple Web Server. This tool is incredibly simple and provides you with a UI that allows you to configure quickly and easily and then start a web server that can dish up static content. I would hesitate to suggest such an approach for production use cases, but it’s not to be sniffed at for internal solutions, safely behind firewalls, network security, etc.

Steps in summary:

  • Install NPM
  • Install a Simpler Web Server – Apache, Nginx, or even Simple Web Server
  • Install the CLI tools for OpenAPI and AsyncAPI
  • Script to identify API documents and use the CLIs

Steps …

As all the functionality is dependent on Node, we need both Niode.js and NPM (Node Package Manager). Installing the Node Version Manager (NVM) is the easiest way to do that for Linux, and Mac with the command:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

Windows has a separately produced binary called NVM for Windows (which will eventually be superseded by Runtime), which has an installer that can be downloaded from the GitHub releases part of the repo.

Once nvm is installed (and ideally in the OS’ PATH environment variable) we can complete the process with the command:

nvm install lts

Which will see the latest Long Term Support (LTS) version installed.

Open API

To install Open API using NPM:

npm install @openapitools/openapi-generator-cli -g

The command that we will need to wrap in a script is:

npx @openapitools/openapi-generator-cli generate -i <your-open-api-spec.yaml> -g html2 -o <your-output-folder-for-this-api>

As the output generated is index.html with subfolders for the stylesheet and Javascript needed, we recommend using the name of the API Spec file (without the postfix, e.g., .yaml) as the folder name.

AsyncAPI

Just Like the Open API command line, we need to install the Async version using the command line:

npm install -g @asyncapi/cli

The equivalent command to generate the HTML is pretty similar, but, note over time, the template-referenced version will evolve (i.e. @2.3.5 to be a newer version)

asyncapi generate fromTemplate <your-async-api-spec.yaml> @asyncapi/html-template@2.3.5 -o ./<your-output-folder-for-this-api> --force-write

Scripting the Process

As you can see, we need to tease out the API files from the source folder, which may contain other resources, even if such resources are schemas that get included in the API (as our APIs grow in scope, we’ll want to break the definitions up to keep things manageable. but also re-use common schema definitions.

The easiest way to do this is to have a text file providing the path and name of the API definition. Each type of API has its own file – removing the need to first work out which type of API needs to be run.

This also means we can read all the API list files to determine then if any API spec pages need to be removed.

Final Thoughts

One of the things we saw when adopting this approach is that the generating process did highlight an issue in the API YAML that the VS Code plugin for Open API didn’t flag, which was the accidental duplication of the operationId when defining an API (an error when creating related API definitions using a bit of cut, paste, and edit).

A static documentation generator is also available for GraphQL (https://2fd.github.io/graphdoc/); although we have not tested it, the available examples, while making the schema navigable, it isn’t as elegant in presenting the details as our Async and Open APIs,

Oracle Cloud Shell Tool

15 Sunday Mar 2020

Posted by in General, Oracle, Technology

≈ Leave a comment

Tags

, , , , , ,

A few weeks ago Oracle announced a new tool for all Oracle cloud users including the Always Free tier. Cloud Shell provides a Linux (Oracle v7.7) environment to freely use ( (within your tenancy’s monthly limits) – no paying for VM or using your limited set of VMs (for free-tier users) or anything like that.

As you can see the Shell can be started using a new icon at the top right (highlighted).  When you open the shell for the 1st time, it takes a few moments to instantiate – and you’ll see the message at the top of the console window (also highlighted). The window provides a number of controls which allows you to expand to full screen and back again etc.

The shell comes preconfigured with a number of tools, such as Terraform with the Oracle extensions, OCI CLI, Java and Git, so linking to Developer Cloud or GitHub for example to manage your scripts etc is easy (as long as you know you GIT CLI – cheat sheet here).  The info for these can be seen in the following screenshots.

In addition to the capabilities illustrated, the Shell is set up with:

  • Python (2 and 3)
  • SQL Plus
  • kubectl
  • helm
  • maven
  • Gradle

The benefit of all of this is that you can work from pretty much any device you like. It removes the need to manage and refresh security tokens locally to run scripts.

A few things to keep in mind whilst trying to use the Shell:

  • It is access controlled through IAM, so you can of course grant or block the use of the tool. Even with access to the shell, users will need to obviously have to have access to the other services to use the shell effectively.
  • The capacity of the home folder is limited to 5GB – more than enough for executing scripts and a few CLI based tools and plugins – but that will be all.
  • If the shell goes unused for 6 months then the tenancy admin will be warned, but if not used, then the storage will be released.  You can, of course, re-activate the Shell features at a future date, but of course, it will be a blank canvas again.
  • For reasons of security access to the shell using SSH is blocked.

The shell makes for a great environment to manage and perform infrastructure development from and will be a dream for Linux hard code users.  For those who like to be lazy with a visual IDE, there are ways around it (e.g. edit in GitHub) and sync. But power users will be more than happy with vim or vi.

Oracle’s own documentation can be fiound at https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/cloudshellintro.htm

Learning Ansible Review Part 2

18 Wednesday Feb 2015

Posted by in Book Reviews, Books, General, Packt, Technology

≈ 2 Comments

Tags

, , , , , , ,

Example Ansible playbook

So having had a significant introduction to Ansible and its capabilities, chapter 2 gets into developing Ansible Playbooks. To do this the book confronts development practises of environment creation and management scripts, or the tendency for ops teams not to apply development like rigour. It is refreshing to read things like ….

… seen these practices at close quarters, we firmly believe that these are more or less similar to voodoo practices that need to be done away with.

Excerpt From: “Learning Ansible.” Packt Publishing.

 

The book does there for dig into the basics of using Git, and illustrating it using the configuration files from an earlier example. The next step would be if you follow development practises to use a Continuous Integration (CI). The book is surprisingly brief on this subject, compared to say Git.

In a development lifecycle the next step is to test, which elegantly introduces Ansible and Vagrant for instantiating virtual machines that the Ansible playbook can be tested against.

This creates the environment by which several additional capabilities provided by Ansible can aide testing activities particularly the ability to tag activities and then run the Ansible script targeting the specifically tagged activities. The additional ability that allows Ansible Playbooks to be executed in a way that allows it only tell what would be changed, rather than perform the change.

The book takes you on from there to introducing how Serverspec could be used. Things for me gain a little too much velocity, perhaps it doesn’t help that I am not familiar with Ruby.

The final part of the chapter and we’re back on solid ground with options on configuring SCM solutions such as Git to support deployment and how Ansible can support the same playbook in different environments such production, preproduction where the playbook is the same but you will be working with different server ids and credentials.

Even if you weren’t to use Ansible there are some thought provoking and good principles for config management and system scripting here.

So by having completed the first two chapters all the principle of Ansible are covered along a range of guiding practises that reflect good development practises in the context of producing environment management. There are limits and the next chapters then go into how to build upon the basics for so it is easy to create  more advanced Playbooks for real world environments with capabilities like playbook iteration, including Playbooks within Playbooks and conditionals for example.

The chapter builds upon the simple examples used on the first chapters showing how the Playbooks can be made to be a lot smarter. For example rather than a task to install each individual application required you can build a table of configuration values and get the task to iterate through the set of RPMs and versions for example. The structures to iterate over can be multi dimensional so you can define some advanced configuration and keep the tasks simple. By building on the earlier examples it helps highlight the benefits of the feature being explained which really helps.

Other features explained here include handlers, so you can trigger one or more activities to be performed once the playbook tasks have completed – so restarting processes can be executed only when all tasks are completed for example.  Other development like features like including other Playbooks are introduced (inline with the idea of DRY – Don’t Repeat Yourself). This leads into using roles, where than identifying your target servers by name or IP you can bring everything together by assigning the server roles e.g. DBTier, AppTier etc. The final two elements are the approach to templating using the Python Jinga2 framework and most crucially given that you will be handling configuration data and passwords Security.

Another couple of well written chapters that embody both insight into Ansible but also hung on the good development ideas so even if you choose not to use Ansible, some of the thinking here could be applied or at-least used to formulate questions as to how the ideas might translate to chef or puppet for example.