When doing GitOps, the desired state of your system will be stored as configuration files in a Git repository. The idea is that this desired state will be compared to the actual state of your system, and as soon as a drift between them is detected someone should be responsible for reconciling that drift. Who that someone is depends, but I've had good experience using Flux.

One tricky part, independent of what tool you use, is that almost all systems are dependent on secrets in some form. The question is how to store those secrets alongside the other configuration files in the Git repository?

It's not a good idea to store them in plain text, no matter if the repository is public or private, so we need a way of encrypting them.

Mozilla SOPS is an open source tool that can help us with this! It's easy to use and you can choose to encrypt entire files, or just specific fields within a file.

I'll show an example on how to use it to encrypt secrets in Kubernetes manifests, but it can be used for a lot more. The workflow will be something like this:

  1. A developer creates a new file that contains a secret. Before commiting and pushing it to the remote repository, they will encrypt the secrets using SOPS.
  2. The secret is encrypted safely in the repository, no need to worry who has access to it.
  3. At release time, the process responsible for applying the yaml (we'll get to this), needs to know how to decrypt the secret before applying it.

Handling secrets using PGP keys

In order to encrypt and decrypt secrets, we need a key. For this demo, we'll start out with the dev PGP keys. Be sure not to use those in any real use case. In a real use case you would create your own PGP keys where the private part hasn't been shared with the world. Later, we'll see how to use a real key using Azure Key Vault instead.

The keys contain a public and a private key. The private key is needed for decryption, but for encryption, only the public key is required. This means that we can commit the public key to the Git repository and let others encrypt secrets as well. The only one who will be able to decrypt secrets, is the one with the private key.

Encrypt a secret using the PGP dev keys

First install gnupg if you don't have that already:

brew install gnupg

Now, import the dev PGP key from Mozilla SOPS:

curl -sSL https://raw.githubusercontent.com/mozilla/sops/master/pgp/sops_functional_tests_key.asc | gpg --import -

This will install three dev keys that we can use to test things in this demo. If you run gpg --list-keys, you should see something like this:

pub   rsa2048 2019-08-29 [SC]
      FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
uid           [ultimate] SOPS Functional Tests Key 1 (https://github.com/mozilla/sops/) <secops@mozilla.com>
sub   rsa2048 2019-08-29 [E]

pub   rsa1024 2019-08-29 [SC]
      D7229043384BCC60326C6FB9D8720D957C3D3074
uid           [ unknown] SOPS Functional Tests Key 2 (https://github.com/mozilla/sops/) <secops@mozilla.com>
sub   rsa1024 2019-08-29 [E]

pub   rsa2048 2019-08-29 [SC]
      B611A2F9F11D0FF82568805119F9B5DAEA91FF86
uid           [ unknown] SOPS Functional Tests Key 3 (https://github.com/mozilla/sops/) <secops@mozilla.com>
sub   rsa2048 2019-08-29 [E]

Next, we'll create a Kubernetes secret:

kubectl create secret generic my-little-secret --from-literal=username=admin --from-literal=password=abC123! --dry-run=true --output=yaml > secret.yaml

It will base64 encode the values, so you get something looking like this:

apiVersion: v1
kind: Secret
metadata:
  name: my-little-secret
data:
  password: YWJDMTIzIQ==
  username: YWRtaW4=

Before commiting this file, we want to encrypt the password field. We can do that using the fingerprint of one of the dev PGP keys. Let's use the first one:

sops --encrypt -pgp FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 --encrypted-regex '^(password)$' secret.yaml > secret.enc.yaml

What we just did was telling SOPS to use the public key to encrypt the field that matches the provided regular expression in the file called secret.yaml and output it to a new file called secret.enc.yaml. We could have done an "in place" encryption instead (replace the contents of the file instead of creating a new one), but we'll do it like this for demo purposes.

If you look in the newly created file, you can see that the password field has been encrypted, but also that there is a new field called sops with a lot of extra information. That information says how and when this file was encrypted, but also what in the file was encrypted.

apiVersion: v1
kind: Secret
metadata:
    name: my-little-secret
data:
    password: ENC[AES256_GCM,data:JU0bk49SFkBY3HM0,iv:wNovfeFKoso3/IaaiGCOkAtLGpvD1y5z7xUOr2OyY6Q=,tag:o76uXfsFhSkNErRGXRpuXw==,type:str]
    username: YWRtaW4=
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2022-01-13T10:02:05Z"
    mac: ENC[AES256_GCM,data:h7lcuHW5rvPlAgFcnPA6gBBUqOMvuh/jDv1jf2/PY3XLlzJGagNwggO86b2fitZUUnc9LBtG4sKvS9xKchLXj7qQQpyLOFD84Di1WiaZrTiUyNDzfC9qfprve1hKPsA04lIz+bZjMYaGWnx/eUdbzON4+P2T2FA6LFLEmsk0XW8=,iv:sV7Iq+yf4G1Z/OtqTjbpLLeWnS2GBEdLLndaSjyZzmI=,tag:Z5m40Vn+B0ZrMk7J7HfZbg==,type:str]
    pgp:
        - created_at: "2022-01-13T10:02:05Z"
          enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMAyUpShfNkFB/AQf+MDhE0qbp8gz5GP9ECDCtXytomtMqDPFswl3gDXr0LeBL
            Pr0rthyd3JtwU4l3JYgkpcGlN8/2lrvjNrYD+KYpLfmnh4BYjiKbtd4bFANqo2g9
            PFjqdwStL5FHfuvJkdHP6YT3QN55rlSUibCFuY3EtcN9i0FThFUJoou+2NFAOfej
            mphIihEgxvHaTac8mgIagcQ7aM2ZOTpQDE3cw3QcfjyyaMYjCBkvMnO5WwR5TRCG
            uTB3YhVczMgbt5VvZUYPpiTbJHvDU7CcT0NUGP9NTHLp5njkYB6QghmQ1sPIo+vO
            kDDXGnfsghxFJVdVW/dyKcUGVPw5T7davY+hjwUydtJeAePH/UXAu047f32YGRPv
            vHl0z5X05KP4m4t8Fn7L8dsuR+UZ9qVTNRNAVrG8M8gVU46vY3ZzE/2xwmOG4TL1
            gHOSFSxPqX4VHdLkCN5vDmJ9jie7aFomBwtBY346Fw==
            =9gRQ
            -----END PGP MESSAGE-----
          fp: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
    encrypted_regex: ^(password)$
    version: 3.7.1

Decrypting a SOPS ecrypted secret

Thanks to the extra metada in the newly encrypted file, decrypting it will be as easy as running this:

sops --decrypt secret.enc.yaml

SOPS will read the contents of the file and see what public key was used and try to find the matching private key to use for decryption (as long as you have access to it). It will also figure out exactly what field to decrypt.

Edit a SOPS encrypted secret in VS Code

If you need to edit an already encrypted file, you can do so as long as you have access to the private key. By default, SOPS will use a default editor like VIM. If you're like me, you might want to use VS Code instead. We can do that by first setting the EDITOR environment variable.

export EDITOR="code --wait"
sops secret.enc.yaml

This should launch VS Code and open the decrypted version of the secret. You can do your edit and once you're done just save and close the file.

Handling secrets using Azure Key Vault

Setup

We need to create a key vault and a key, so let's do that first:

# Create a resource group
az group create \
  --location westus2 \
  --name MyResourceGroup

# Create the key vault
az keyvault create \
  --location westus2 \
  --name MyKeyVault \
  --resource-group MyResourceGroup

# Create the secret
az keyvault key create \
  --name MySopsKey \
  --vault-name MyKeyVault \
  --protection software \
  --ops encrypt decrypt

# Your user won't have encrypt/decrypt access by default, so you need to set that. First, get the objectId of your current user
az ad user show --id <your user id>

# Give your user access. You might want to add more permissions here based on your use case, but these will be enough for the rest of this demo
az keyvault set-policy \
  --name MyKeyVault \
  --object-id <objectId> \
  --key-permissions get encrypt decrypt

Encrypt a secret using the Azure Key Vault key

In order to encrypt a secret, we need to get a refence to the key id:

az keyvault key show --name MySopsKey \
  --vault-name MyKeyVault \
  --query key.kid

The key id should look something like this: https://mykeyvault.vault.azure.net/keys/MySopsKey/6cd0546319094954ba6d30e17e0331e1.

Now, we're ready to encrypt the secret:

sops --encrypt -azure-kv <key id> --encrypted-regex '^(password)$' secret.yaml > secret.enc.yaml

Just like before, we've created a new encrypted version of the file.

Decrypting a SOPS ecrypted secret

Since the the file contains the information about how it was encrypted, decrypting it will be the same as before:

sops --decrypt secret.enc.yaml

Decrypting secrets during deployment

Having encrypted secrets in your Git repository is great, but at some point you are going to have to decrypt them so that they can be used. Depending on how you are doing your deployments, there are a couple of different options:

GitOps using Flux

Flux is an awesome tool that enables a GitOps workflow, mainly for Kubernetes. They have great documentation on their site on how to use Flux in combination with SOPS, so I would recommend you to check that out. Flux is installed as an agent inside your Kubernetes cluster. It will monitor the Git repository that holds your desired configuration state. As soon as it detects a drift between the actual state (what's in the cluster) and the desired state, it will try to make the changes neccessary to reach the desired state again. In short, it will go something like this:

  1. A newly created (encrypted) secret is added to the Git repository.
  2. Flux will detect this new change and will want to apply that to the cluster, but it will have to decrypt it first. SOPS is no problem for Flux as long as it has access to the (private) key used for encryption, and it will be able to decrypt it before applying it to the cluster.

Other options

If you're not using Flux, then there are of course other options for decryption. If you rely on Azure Pipelines, GitHub Actions, Jenkins or whatever to deploy changes, you might end up with something like this instead:

  1. A newly created (encrypted) secret is added to the Git repository.
  2. A deployment pipeline is triggered based on this change and it will try to apply it. Again, it will have to decrypt the secret first. As long as the pipeline can run a SOPS command and has access to the (private) key, it will be as easy as inserting a pre deployment step that  decrypts the file.