At Deloitte Platform Engineering, I help organisations build Kubernetes clusters and establish baseline policies and governance to ensure that they remain secure. A key aspect of cluster governance is defining how sensitive application configuration or ‘secrets’ are managed and used within the cluster.

Secrets such as passwords and API keys are often needed by applications to access secured resources, and due care is always needed to mitigate any unauthorised access to such resources. My colleague, Tim has written an excellent article about secrets, how secrets are managed with the Azure App Service and some principles to keep secrets secret.

So how do we manage secrets in Kubernetes? How do we integrate Secrets with our Continuous Integration and Continuous Deployment (CI/CD) practices? In this blog, we will explore:

• Secrets in Kubernetes
• Sealed Secrets in Kubernetes, and
• Mounting external secrets providers into Kubernetes

The Secrets

Secrets, in the context of Kubernetes, are objects that contain sensitive data such as passwords, tokens and credentials. Much like ConfigMap, Secrets can be mounted by containers that are hosted in the clusters as local files or environment variables, which can then be referred to by the applications hosted inside the containers. Access to Secrets can be controlled by implementing Role Based Authorisation Control (RBAC) rules.

Secrets can be created by either a manifest file or through the kubectl CLI. Here is a sample kubectl command to create a Secret, with two literal key-value pairs: ‘username’ and ‘password’:

kubectl create secret generic env-secret 
--from-literal=username=appusername 
--from-literal=password=Password123

The above command creates the following manifest:

We can then create a Role to allow users read-only access to the Secrets as follows:

Once the Secrets have been created in the cluster, we can mount them to a Pod as Environment Variables by specifying them in the Deployment or Pod manifest as follows (referring to the Secret created above):

As you can see here, it is impossible to include Secret creation in your CI/CD pipeline without storing the sensitive information as plain text in your source control. Our principles dictate that we should avoid storing sensitive information in a source control. Additionally, the sensitive information will be visible to whomever created the Secret, and this person might not have the authority to view that information.

So how do we avoid these issues? Introducing: the Sealed Secret.

The Sealed Secrets

A Sealed Secret is an open-source project that allows Secrets to be securely stored in a source control, enabling CI/CD to be implemented.

This solution consists of two main components:

• Sealed Secret Controller – The controller is installed on the cluster; its purpose is to decrypt Sealed Secret objects into plain Secret objects using its auto-rotating encryption key
• Kubeseal CLI – This client-side utility encrypts plain Secrets into Sealed Secrets

Refer to the official documentation for Sealed Secrets for instructions on how to install both the controller to your cluster, and the Kubeseal CLI to your development environment.

Let’s see how this works. Repeating the steps that we took earlier, we can use the Kubeseal CLI to convert the generated Secret into a Sealed Secret as follows:

kubectl create secret generic env-secret 
   --from-literal=username=appusername 
   --from-literal=password=Password123 
   --dry-run=client -o yaml | kubeseal -o yaml > SealedSecret.yaml

The kubectl command creates a Secret manifest, and then the kubeseal command converts them into a Sealed Secret manifest. It does this by using the public certificate that is provided by the Sealed Secret Controller running on the connected cluster. The output of this command is then piped into a file named ‘SealedSecret.yaml’. You can see the content of the file below:

As you can see, the username and password values have been encrypted. The only party that can decrypt them is the Kubernetes cluster that this Sealed Secret is intended for, as it is the only one with the private key required to decrypt the encrypted content.

Once the Sealed Secret is deployed into the intended cluster, the Sealed Secret Controller automatically decrypts it and creates a Secret object with the decrypted values.

The following diagram depicts the transformation of Secrets into Sealed Secrets and back.

There’s an important caveat with this setup. In order to create a Sealed Secret, you need:

• Access to the sensitive information or Secret
• Access to the source control where the Sealed Secret manifest needs to be stored
• Access to the Kubernetes cluster which needs to include enough privilege to use the Sealed Secret Controller, or
• Access to the public key that the cluster’s Sealed Secret Controller uses
• The technical knowledge to perform the Secret and Sealed Secret creation

With the above requirements, it is almost impossible to separate the concerns between the party that owns the sensitive information and the party who performs the Sealed Secret creation. This leads to secrets needing to be made available to a wider audience, and therefore increasing the risk of unauthorised use of such secrets.

The Secrets from the Vault

Major cloud providers offer native facilities that store and manage sensitive information. Some examples of these facilities are AWS’s Secret Manager, GCP’s Secret Manager and Azure’s Key Vault. All of these providers provide a way to make secrets available to the applications running on Kubernetes at runtime. In this article, we will focus on Azure Key Vault in particular.

Using these providers to manage secrets ensures that the sensitive information is only made available to the intended application at runtime. Access to the sensitive information can also be restricted to a limited audience, such as a high-level platform administrator or a Database administrator.

So how do we expose these secrets to your Kubernetes cluster?

Kubernetes Secrets Store CSI Driver allows Kubernetes cluster to mount secrets from external secrets management providers. To mount secrets from Azure Key Vault specifically, we need the the Azure Key Vault Provider for Secrets Store CSI Driver. Let's see how it works.

Suppose we already have an Azure Key Vault, and we already have two secrets in it named ‘username’ and ‘password’. First, we will need the credentials to access the Key Vault itself. The Secret Store CSI Driver provides several methods to do this, but we will use the Service Principal method in this example. We will need to create a Service Principal as well as the access policy needed to access the Key Vault data:

az ad sp create-for-rbac --skip-assignment --name http://sp-secret-store

az keyvault set-policy -n mykeyvault 
  --secret-permissions get 
  --spn [Insert Client ID created from the previous line]

Then you can create a Secret to store the credential in your cluster. This Secret will later be used by the CSI Driver to access the Key Vault:

kubectl create secret generic secrets-store-creds 
--from-literal=clientid=[Insert Client ID created from the previous step] 
--from-literal=clientsecret=[Insert Client secret created from the previous step]

Next we will need to define the Secret Provider Class, which tells the provider the ‘how’ and the ‘what’ to mount the secrets:

Then we can mount the Secret Store in a Pod or a Deployment definition as follows:

Upon Pod creation, the Secret will be mounted to the path: ‘/mnt/secret-store’ in the container. Then it will be mounted to a Secret named ‘env-secret’, and two environment variables will be created inside the container for the application to use. It is important to note that at this point, the Secret is available to any Pod in the same namespace.

Wrapping Up

Kubernetes Secrets are a great way to keep sensitive information inside the cluster with proper authorisation. However additional tools are required to make them work with your CI/CD processes, and to keep your Secret a secret.

Sealed Secret is a simple solution that enables Secret creation to be a part of the CI/CD processes, but not without making a compromise on the separation of concern.

Finally, CSI Secret Store will ensure the separation of concern, but it is a little more complex to set up.

Depending on the organisational requirements, all of these methods are viable options to integrate secret management into your CI/CD processes.