When configuring Fluentd we often need to provide credentials to access event sources, targets, and associated services such as notification tools like Slack and PagerDuty. The challenge is that we don’t want the credentials to be in clear text in the Fluentd configuration.

Using Env Vars

In the Logging In Action with Fluentd book, we illustrated how we can take the sensitive values from environment variables so the values don’t show up in the configuration file. But, we’ve seen regularly the question of how secure is this, can’t the environment variable be seen by everyone on that machine?

The answer to this question comes down to having a deeper understanding of how environment variables work. There is a really good explanation here. The long and short of it is that environment variables can only be seen by the process that creates the variable and any child process will receive a copy of the parent’s variables.

This means that if we create the variable in a shell, only that shell and any processes launched by that shell can see the environment variable. So as long as we don’t set variables up as part of a system-level configuration then we already have a level of security. So we could wrap the start of Fluentd with a script that sets the environment variables needed. Then everything launches that script.

An even better way?

Of course, the script sets the environment variable with a value, and where does that come from and how secure can that script be? There are several options for that …

• Set the value in a configuration file that is tightly controlled by access privileges. So unless you’re running as a specific set of privileges you can’t see the variable.
• Inject the values in from the outside world – we can do this when working with Kubernetes through the use of ConfigMaps.
• Create a level of indirection by accessing another source for storing the sensitive data values such as credentials. By using a secret store that has the means to disable access to the data. We have the means to more easily cut off access if we think a client has been compromised. With the sensitive details centrally stored we also have the means to change the credentials and everyone who should be able to get the credentials can benefit.

Incorporating Vault

This latter approach is how secrets managers such as Hashicorp Vault and Conjur among others. You can see this with a simple example provided by Hashicorp here. To see this in action the first step is to install Vault (open source version), which can be done a number of different ways (we’ve done it using Chocolatey but other package managers can do the something) with the command choco install vault.

This installs vault. We’ve then fired up Vault with the command vault server -dev. Note we are running the Vault server in its developer mode. Doing so simplifies the setup work for this exercise and shouldn’t be done for a production use case. This will result in a message like this. Note the instructions regarding setting environment variables VAULT_ADDR and we need to set a second VAULT_TOKEN to hold the Root Token included in the output as well.

Those variables need to be set in each shell which will want to interact with the Vault (back to our point about using environment variables and shell scripts carefully). These values are needed as they are used by the Vault CLI and language-provided SDKs. It is worth noting, that while these values are ‘sensitive’ you may well want to reuse them for any other software running on the client machine. The risks are mitigated by the ability to cut off the client by invalidating the token in the Vault server, configure secrets on the server in a manner that controls what is exposed, and permissions to interact with the secrets.

Storing a secret

To demonstrate the process we need to prime the Vault with a secret – using the simple KV storage mechanism we can do this with the command vault kv put secret/slack token=xoxb-735037803329-1110660446995-q6GrvjYukkzzYlBVJpzzigj7o This command tells the Vault using the KV storage type to create a namespace called secret/slack and add a key value pair, with my key called token and the value is in this case the Slack token as the book’s Chapter 4 demonstrates.

We can test the obtaining of the value by using the CLI to retrieve the result using the command vault kv get secret/creds. This returns both the secret and a lot of useful metadata. Fortunately, we can add an extra parameter that will just give us the value we want by including -field=token telling the CLI the key we’re interested in. The complete command now looks like vault kv get -field=passcode secret/creds.

How to incorporate the token retrieval into Fluentd

To incorporate the solution into our configuration we have several options:

• Use the Ruby SDK
• Use curl (the CLI has a nice feature that will translate the CLI instruction to the nearest curl equivalent)
• Call the Vault CLI directly

The Ruby code can be found here – and incorporating the code into the configuration is messy, there are some tricks we can apply to make it a bit neater.

We could use a command line such as curl (the CLI provides a handy option that will display the curl command equivalent to the CLI call (add -output-curl-string to the CLI). This has two downsides, firstly the CLI can process the response to do things like extract a specific attribute of the secret for us, the CURL option returns the full response. secondly, the HTTP header needs our authentication token which would put it into clear text in our configuration file.

Embedding the CLI into our Fluentd configuration is therefore looking like the cleanest option. This means we can go from a configuration looking like:

  <match *>
	@type slack
        token xoxb-735037803329-1110660446995-q6GrvjYukkzzYlBVJpzzigj7o
	username unifiedfluent
	icon_emoji :ghost: # if you don't want to use icon_url, delete this param.
	channel demo
	message Node2 says - %s
	message_keys message
	title %s
	title_keys time
	flush_interval 1s
        time_key time
  </match>

To something like:

<match *>
	@type slack
        token "#{`vault kv get -field=token secret/slack`}"
	username unifiedfluent
	icon_emoji :ghost: # if you don't want to use icon_url, delete this param.
	channel demo
	message Node2 says - %s
	message_keys message
	title %s
	title_keys time
	flush_interval 1s
        time_key time
  </match>

The all-important change is highlighted above, but let’s quickly cover a couple of key details. Firstly here we’re asking Fluentd to call Ruby – denoted by the use of "#{ and ending with }". Next, we’re taking advantage of the fact that Ruby when it encounters a backward tick ` will execute the instructions using a subshell (important, remembering the relationship for environment vars). The returned string from the CLI is the token required. As a result, Fluentd sees the line resolve to token xoxb-735037803329-1110660446995-q6GrvjYukkzzYlBVJpzzigj7o.

Why better?

This provides a good solution as we’ve not got any credentials in the configuration file for Fluentd. We can instruct the server to reject any requests using the token if we suspect the value has been hijacked. We have a central location to manage our secrets – making it easier to handle password rotation. The number of different credentials held on a client-server can be greatly reduced. While the final step is specific to Fluentd, the overall principles can be applied to many scenarios.