A Fast (and Dirty) Way to Publish API specs

03 Monday Jun 2024

Posted by in APIs & microservices, General, Technology

≈ Leave a comment

Tags

, , , , , , , , , , ,

The API specs created using Open API Specification (OAS) and ASyncAPI specification aren’t just for public API consumption. In today’s world of modular component services that make up a business solution, we’re more than likely to have APIs of one sort or another. These need documenting, perhaps not as robustly as those public-facing ones, but the material needs to be easily accessible.

Spotify’s contribution to the CNCF—Backstage is a great tool for sharing development content, particularly when your document and code repository is at least git-based if not GitHub (you move away from this or don’t easily have permissions to configure application authentication, you can still work with Backstage, but your workload will grow a lot). There is no doubt that Backstage is a very powerful, information-rich product. But that does come at the cost of needing lots of configuration, the generation of metadata descriptors additional to the APIs to be cataloged, etc. All of these can be a little heavy if you’re using Backstage as a low-cost API documentation portal that might fill the gaps that your corporate wiki/doc management (Confluence/SharePoint) solution can’t support (it is one of the very, very few open-source options that can support both OAS and AsyncAPI reader friendly API rendering tools).

We could, of course, adopt the approach of there are free VS Code plugins that can render the friendly views of APIs, so just perform a git pull (or copy the API specs from a central location) to give the nice visualization. This is fine, but the obligation is now on the developer to ensure they have the latest version of the API spec and that they are using VSCode – while it is very dominant as an IDE – not everyone uses it, particularly if you’re working with low code tooling.

There is a fast and inelegant solution to this if you’re not in need of nice features such as attribute-based search and sorting, etc. Both the Open API Specification and the Async API communities have built command line-based renderers that will read your API specification (even if the schema is spread across multiple files) and generate HTML (an index.html file), CCS, and JavaScript renderings that you see in many tools (hyperlinked, folding, with code and payload examples of the API).

So, we need to grab the YAML/JSON specifications and run them through the tool to get the presentation formatting. You do need to get the specs, but we can easily script that with a bit of shell script that retrieves/finds the relevant files from a repository and then runs the CLI utility on the files.

We want to bring the static content to life across the network for developers. So, on a little server, we can host this logic, plus an instance of Apache, IIS, or Nginx if you’re comfortable with one of the industrial superpower web servers. Or use a spin-off project from the Chrome Server called the Simple Web Server. This tool is incredibly simple and provides you with a UI that allows you to configure quickly and easily and then start a web server that can dish up static content. I would hesitate to suggest such an approach for production use cases, but it’s not to be sniffed at for internal solutions, safely behind firewalls, network security, etc.

Steps in summary:

  • Install NPM
  • Install a Simpler Web Server – Apache, Nginx, or even Simple Web Server
  • Install the CLI tools for OpenAPI and AsyncAPI
  • Script to identify API documents and use the CLIs

Steps …

As all the functionality is dependent on Node, we need both Niode.js and NPM (Node Package Manager). Installing the Node Version Manager (NVM) is the easiest way to do that for Linux, and Mac with the command:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

Windows has a separately produced binary called NVM for Windows (which will eventually be superseded by Runtime), which has an installer that can be downloaded from the GitHub releases part of the repo.

Once nvm is installed (and ideally in the OS’ PATH environment variable) we can complete the process with the command:

nvm install lts

Which will see the latest Long Term Support (LTS) version installed.

Open API

To install Open API using NPM:

npm install @openapitools/openapi-generator-cli -g

The command that we will need to wrap in a script is:

npx @openapitools/openapi-generator-cli generate -i <your-open-api-spec.yaml> -g html2 -o <your-output-folder-for-this-api>

As the output generated is index.html with subfolders for the stylesheet and Javascript needed, we recommend using the name of the API Spec file (without the postfix, e.g., .yaml) as the folder name.

AsyncAPI

Just Like the Open API command line, we need to install the Async version using the command line:

npm install -g @asyncapi/cli

The equivalent command to generate the HTML is pretty similar, but, note over time, the template-referenced version will evolve (i.e. @2.3.5 to be a newer version)

asyncapi generate fromTemplate <your-async-api-spec.yaml> @asyncapi/html-template@2.3.5 -o ./<your-output-folder-for-this-api> --force-write

Scripting the Process

As you can see, we need to tease out the API files from the source folder, which may contain other resources, even if such resources are schemas that get included in the API (as our APIs grow in scope, we’ll want to break the definitions up to keep things manageable. but also re-use common schema definitions.

The easiest way to do this is to have a text file providing the path and name of the API definition. Each type of API has its own file – removing the need to first work out which type of API needs to be run.

This also means we can read all the API list files to determine then if any API spec pages need to be removed.

Final Thoughts

One of the things we saw when adopting this approach is that the generating process did highlight an issue in the API YAML that the VS Code plugin for Open API didn’t flag, which was the accidental duplication of the operationId when defining an API (an error when creating related API definitions using a bit of cut, paste, and edit).

A static documentation generator is also available for GraphQL (https://2fd.github.io/graphdoc/); although we have not tested it, the available examples, while making the schema navigable, it isn’t as elegant in presenting the details as our Async and Open APIs,

Cloud Native Architecture book

31 Friday May 2024

Posted by in Books, General

≈ Leave a comment

Tags

, , , , , , , ,

It’s a busy time with books at the moment. I am excited, and pleased to hear that Fernando Harris‘ first book project has been published. It can be found on amazon.com and amazon.co.uk among sites.

Having been fortunate enough to be a reviewer of the book, I can say that what makes this book different from others that examine cloud-native architecture is its holistic approach to the challenge. Successful adoption of cloud-native approaches isn’t just technical (although this is an important element that the book addresses); it also considers organizational, processes, and people dimensions. Without these dimensions, the best technology in the world will only be successful as a result of chance rather than by intent.

As a result, this book guides and connects content to the Kubernetes technical content (technical how-to books we typically see from publishers like Manning and O’Reilly) and the more organizational leadership books that you might expect from IT Revolution (Gene Kim et al.).

A read I’d recommend to any architect or technical lead who wants to understand the different aspects of achieving cloud-native adoption rather than just the mastery of an individual technology.

Secure APIs (MEAP) book – Initial Impressions

24 Friday May 2024

Posted by in Book Reviews, General, manning

≈ Leave a comment

Tags

, , , , ,

My day job as a technical architect means I spend a lot of time working on and around technical non-functional needs, from observability to APIs. And APIs are everywhere (sometimes we don’t talk about things like the OpenTelemetry Protocol (OTLP) as APIs, but this is what it is). and I’ve written and blogged on the subject many times in the past.

One of the things I tend to do is read books on the subject – always on the lookout for new strategies, ideas, and techniques for handling an API’s number one challenge – security. With a new book on Secure APIs from José Haro Peralto being published by Manning (as a Manning author, I have the perks of looking at books published and in the Early Access Program).

The Early Access Program means that after the first couple of chapters have been written and go through initial review processes, they’re made available. However, the book is still in development and has not gone through a full copy edit process. However, the core ideas and messages are there in the book.

The book so far looks really good. It comes across as very practical and illustrative of the points it needs from the outset, with some nicely presented insights about why API Security is such an important consideration—54% of web traffic is API-driven, organizations see as many as 10 million attacks per day, and a breach typically costs $6.1 million. If you’re trying to make a case for investing in API security – there are some great references here.

The book doesn’t just look at implementing the code that powers the API contract but also the tools from firewalls to gateways. It engages in the process of figuring out what risks an API needs to mitigate and the consequences of failing to do so. While the first couple of chapters look at the broader landscape and ideas. We can expect a closer look at things like the OWASP Top 10 (a resource that should be mandatory learning for anyone going to implement APIs or web app development more generally) as the book progresses.

The first couple of chapters read well and are easy to absorb, and we’re looking forward to reading the coming chapters, which will discuss the nuts and bolts of securing APIs.

The only observation to be aware of at this point is that, while not explicitly stated, the illustrations suggest a strong bias to RESTful web services with the appearance of just the Open API Initiative logo. While REST is the most common API approach, gRPC, and GraphQL are continuing to make big inroads and are supported by the Asynchronous API Spec. I suspect this will be addressed given José’ background and expertise. I#m looking forward to the coming chapters.

Useful Quick Reference Links when Writing API Specs

13 Monday May 2024

Posted by in APIs & microservices, General, Technology

≈ Leave a comment

Tags

, , ,

Whether you’re writing Asynchronous or Open APIs unless you’re doing it pretty much constantly, it is useful to have links to the specific details, to quickly check the less commonly used keywords, or to check whether you’re not accidentally mixing OpenAPI with AsyncAPI or the differences between version 2 or version 3 of the specs. So here are the references I keep handy:

There are some useful ISO Specs for common data types like dates. Ideally, if you’re working in a specific industry domain, it is worth evaluating the industry standard definitions (even if you elect to use entire standardized objects). But when you’re not in such a position, it is at least work using standard ways of representing data—it saves on documentation effort.

Not a standard, but still an initiative to promote consistency back by the likes of Microsoft etc, so could provide some insights/ideas/templates for common data structures – https://schema.org/

There are, of course, a lot of technology-centered standards such as media streaming, use of HTTP, etc.

These and many more resources are in my Tech resources.

Fluent Bit Book – Blogged Extracts

12 Sunday May 2024

Posted by in Books, Fluentbit, manning

≈ Leave a comment

Tags

, , ,

The Calyptia team has been publishing some extracts from Fluent Bit with Kubernetes, you can check them out at:

Keep an eye on the Calyptia blog for more to come.

The book isn’t too far away from reaching publication, we’re a couple of weeks away from starting the final copy edit process.

Checking your OpenTelemetry pipeline with Telemetrygen

30 Tuesday Apr 2024

Posted by in Fluentbit, Fluentd, General, Technology

≈ Leave a comment

Tags

, , , , , , , , ,

Testing OpenTelemetry configuration pipelines without resorting to instrumented applications, particularly for traces, can be a bit of a pain. Typically, you just want to validate you can get an exported/generated signal through your pipeline, which may not be the OpenTelemetry Collector (e.g., FluentBit or commercial solutions such as DataDog). This led to the creation of Tracegen, and then the larger Telementrygen.

You can use Tracegen or Telemetrygen by either downloading and running the Go app from GitHub or using the Docker file. But there are a couple of challenges:

  • On initial investigation, these utilities appear wrapped up in the larger opentelemetry-collector-contrib. While potentially useful, shaking out your OTel pipelining is somewhat overkill.
  • We can install the app locally with the following command, but then we need to set up Golang in the environment. 
go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
  • Fortunately, there is a Docker image that just contains the tool, but to use it, we need to know what the parameters are to override the container defaults. The only irritant is that you either need to mess about with the container to get at the information (i.e., run the — help options) or install the utility (the parameters are not in the GitHub docs), so we’ve teased out all the options into the following table.

The following table’s Signal column All means it can be applied to Metrics, Traces, or Logs. Otherwise we’ve named the signal type that the parameter can be used with.

SignalParameter/FlagDescription
Trace–batchWhether to batch traces (default true)
All–ca-cert stringTrusted Certificate Authority to verify server certificate
Traces–child-spans intClient certificate file

All		–client-key stringClient private key file
All–duration durationFor how long to run the test
All-h, –help {traces|metrics|logs}help – with give you the basic help if no parameter is passed. Or the signal type help when used with the signal name e.g. telemetrygen traces --help
All–interval durationReporting interval (default 1s)
Traces–marshalWhether to marshal trace context via HTTP headers
All–mtlsWhether to require client authentication for mTLS
All–otlp-attributes map[string]stringCustom resource attributes to use. The value is expected in the format key=”value”. Note you may need to escape the quotes when using the tool from a cli. Flag may be repeated to set multiple attributes (e.g --otlp-attributes key1=\"value1\" --otlp-attributes key2=\"value2\")
All–otlp-endpoint stringDestination endpoint for exporting logs, metrics and traces
All–otlp-header map[string]stringCustom header to be passed along with each OTLP request. The value is expected in the format key=”value”. Note you may need to escape the quotes when using the tool from a cli. Flag may be repeated to set multiple headers (e.g --otlp-header key1=\"value1\" --otlp-header key2=\"value2\")
All–otlp-httpWhether to use HTTP exporter rather than a gRPC one
All–otlp-http-url-path stringWhich URL path to write to (default "/v1/traces")
All–otlp-insecureWhether to enable client transport security for the exporter’s grpc or http connection
All–rate intApproximately how many metrics per second each worker should generate. Zero means no throttling.
Traces–service stringService name to use (default "telemetrygen")
Traces–size intDesired minimum size in MB of string data for each trace generated. This can be used to test traces with large payloads, i.e. when testing the OTLP receiver endpoint max receive size.
Traces–span-duration durationThe duration of each generated span. (default 123µs)
Traces–status-code stringStatus code to use for the spans, one of (Unset, Error, Ok) or the equivalent integer (0,1,2) (default “0”)
All–telemetry-attributes map[string]stringNumber of traces to generate in each worker (ignored if the duration is provided) (default 1)
Traces–traces intNumber of traces to generate in each worker (ignored if duration is provided) (default 1)
All–workers intNumber of workers (goroutines) to run (default of 1)
Metrics–metric-type metricTypeNumber of metrics to generate in each worker (ignored if the duration is provided) (default 1)
Metrics–metrics intNumber of logs to generate in each worker (ignored if the duration is provided) (default 1)
Logs–body stringBody of the log (default “the message”)
Logs–logs intThe severity number of the log ranges from 1 to 24 (inclusive) (default 9)
Logs–severity-number int32The severity number of the log ranges from 1 to 24 (inclusive) (default 9)
Logs–severity-text stringSeverity text of the log (default “Info”)
All the configuration parameters for Telemetrygen

It is worth noting that while Tracegen has similar configuration parameters, they aren’t exactly the same in the CLI, often one dash rather than two in the name for example.

The following is a simple Docker compose file that can help you use the container to conduct local testing of your collector. In this configuration, we’re sending a trace to the host machine with HTTPS disabled.

services:
  web:
    image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest
    network_mode: host
    entrypoint:
      [
        "/telemetrygen",
        "traces",
        "--otlp-insecure",
        "--otlp-endpoint",
        "host.docker.internal:4317",
      ]

Observing the Observer (Fluent Bit monitoring)

25 Thursday Apr 2024

Posted by in Fluentbit, General, Technology

≈ Leave a comment

Tags

, , , , , , , ,

In the Fluent Bit book I touch upon the point that we should be observing the observer. After all, if we don’t monitor our observability stack, then we’ll be operating blind and may never know until things go catastrophically wrong, and we’re getting complaints that production business solutions are down. One of the peer review comments was it would be really good to have a visual representation in the book. While I’d love to incorporate such diagrams, for them to be readable, they do use up a lot of space on the printed page, and very long chapters can also put some readers off. So, given the point wasn’t a key theme, we simply couldn’t incorporate the diagram.

But the suggestion is a good one. So we’ve created the visual representation here.

Annotated diagram showing how Fluent Bit could be used to monitor an open-source observation stack.

The diagram shows how we can monitor our observation tech stack so we don’t become ‘blind’ without knowing about it. The Outflow from the right of the diagram would send a signal to a notification service, which could be as simple as email or as user-friendly as Slack. We’ve indicated which kinds of metrics, logs, and traces could offer the most value from our observation tech stack.

If we’re running everything within a Kubernetes cluster, it would be easy to say we don’t need such a sophisticated setup as we can use Kubernetes liveness probes if the containers are well configured. While it is true if one of our services starts to fail, a liveness check should pick it up and recycle the container. But such probes only worry about the HTTP response code, not the cause. If we don’t monitor and capture more information we’ll never understand the problem. At worst we could end up seeing Kubernetes starting and then killing our containers in a vicious cycle and struggling to resolve the cause. So collecting the logs and metrics remains just as important.

How to Publish Fluent Bit Metrics and Logs

To publish Fluent Bit’s metrics to Prometheus, we need to configure the fluentbit-metrics input plugin (it does sound odd as an input, but there are reasons that become clearer in the book). We then route the output that supports using Fluent Bit as a Prometheus node exporter or makes use of the remote write API.

The log output for Fluent Bit can be configured via the command line or in the SERVICE blog (using the attributes log_file and log_level in the configuration file. Today this is setting the log threshold and identifying the log file. We can then, of course, configure a tail input plugin against the file if we want to send the logs to OpenSearch. We can also set plugin-specific logging thresholds as overrides to the Fluent Bit wide setting in the SERVICE block of configuration.

Configuring the other monitoring tools

  • Grafana‘s configuration will allow it to publish Prometheus scrapable metrics and Traces that are OTLP compliant can be found documented here.
  • Prometheus provides metrics on itself (details here) and logging controls as part of its command line and generates logfmt or JSON logs, details here.
  • OpenSearch‘s logs can be accessed as documented here. The Logs are created with Log4j2, which means out of the box, it will be easy to parse them. Configuring the output of slow query reports does need to be switched on. OpenSearch also illustrates a pre OpenTelemetry/OpenMetrics approach to sharing internal metrics by writing them as logs. However, there are ways to convert such log events to OTLP Metrics with Fluent Bit.
  • Jaeger provides metrics endpoints that are Prometheus-compatible, along with JSON-based logs, and are documented here. There is some support for tracing.

InfoQ Article on Fluent Bit with MultiCloud

23 Tuesday Apr 2024

Posted by in Cloud, Fluentbit, Technology

≈ Leave a comment

Tags

, , ,

I’m excited to say that we’ve had an article on Fluent Bit and multi-cloud published on InfoQ. Check it out at https://www.infoq.com/articles/multi-cloud-observability-fluent-bit/ . This is another first for me.

As you may have guessed from the title, the article is about how Fluent Bit can support multi-cloud use cases. As part of the introduction, I walked through some of the challenges that aren’t so obvious when operating with a multi-cloud scenario. The following diagram illustrates that.

The book is now in its final peer review process with updates also being sent to the Early Access Program as well (MEAP).

Fluent Bit book cover

Fluent Bit with Kubernetes – more MEAP chapters

06 Saturday Apr 2024

Posted by in Books, Fluentbit, manning, Technology

≈ Leave a comment

Tags

, , , ,

12th April Update – The last chapter, a use case Appendix, and a couple of chapter updates are heading to the MEAP release.

We’ve not been blogging too much as we’ve been very focused on the book. For the keen readers who have signed up for the MEAP (Manning Early Access Programme) of the book, another 2 chapters are in the process of being made available.

The last chapter has been submitted to our editor along with the appendix, which includes an enterprise use case that outlines a business scenario and illustrates how Fluent Bit can be applied.

We’ve received the feedback from the second peer review and have started to address it. I’m sure that every Manning author will testify as to how helpful the process is. While I recommended some of the reviewers to my editor, I didn’t know others. All the feedback comes back anomalously. So publicly, thank you to the reviewers. Constructive feedback is key to how we ensure that we are getting our points across, but also how details we may have overlooked or thought obvious get put right.

Unfortunately, authors can’t always address every comment. Sometimes, that is down to the fact that the layout has to work within the publisher’s guidelines. Sometimes, we simply can’t fit in suggested content, as we’re ultimately working to an agreed timeline, and people can be put off by 800-page books. For me, and I suspect other authors, those extras aren’t ignored; they’re fuel for blog ideas and content.

We’ve one more peer review cycle where the reviewers get pretty much the entire book, and once any edits for that are needed, we move into the copy editing, which is done by Manning, and I just need to confirm edits don’t accidentally change the meaning and emphasis. This will be a time when we can start blogging and sharing more.

Fluent Bit the engine to power ChatOps – update

17 Sunday Mar 2024

Posted by in chatbots, Fluentbit, General

≈ 1 Comment

Tags

, , , , , ,

The other month, I described a presentation and demo (Fluent Bit – Powering Chat Ops) we’ll be doing for the Cloud Native Rejekts conference, which is the precursor event to KubeCon in Paris this week. Since that post, we’re excited to say that, with Patrick Stephens’s contributions from Chronosphere, the demo is now in the Fluent GitHub repo. It has been nicely packaged with a Docker Compose, so everything runs in a couple of containers.

In addition, if you want to see the presentation and hear us discuss the solution and explain how it works, we recorded part of the presentation dry run, which can be heard here (Demo) and here (Code overview).

I couldn’t be in Paris in person, so Patrick took the job of presenting in Paris, we tried to enable my remote participation but had audio issues. Hopefully, you’ll see the recording of Pat’s physical presentation here. But I did manage to collaborate in the demo:

This means that the original repo I mentioned can be viewed as a beta or upstream version (it’s cluttered with some generated code from Helidon, which we will eventually get around to exploiting and making the utility a native binary executable).