Mozilla SOPS and GitOps Toolkit (Flux CD v2) to decrypt and apply Kubernetes secrets

Using GitOps way of working and tools like the GitOps toolkit (Flux CD v2) is great for applying configuration to your Kubernetes clusters but what about secrets and how can you store them securely in your repository? The perfect tool for this is Mozilla’s SOPS which uses a cloud based KMS, HashiCorp Vault or a PGP key to encrypt and decrypt your secrets and store them in encrypted form with the rest of your configuration in a code repostory. There is a guide in the Flux documentation about how to use SOPS but I did this slightly differently with a Google Cloud KMS.

Start by downloading the latest version of the Mozilla SOPS command-line binary. This is what makes SOPS so easy to use, there is not much you need to encrypt or decrypt secrets apart for an KMS system or a simple PGP key.

sudo wget -O /usr/local/bin/sops https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
sudo chmod 755 /usr/local/bin/sops

Next create the Google Cloud KMS, which I am using in my example.

$ gcloud auth application-default login
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?code_challenge=xxxxxxx&prompt=select_account&code_challenge_method=S256&access_type=offline&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=xxxxxxxxx-xxxxxxxxxxxxxxxxxx.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth


Enter verification code: xxxxxxxxxxxxxxx

Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
$ gcloud kms keyrings create sops --location global
$ gcloud kms keys create sops-key --location global --keyring sops --purpose encryption
$ gcloud kms keys list --location global --keyring sops
NAME                                                                           PURPOSE          ALGORITHM                    PROTECTION_LEVEL  LABELS  PRIMARY_ID  PRIMARY_STATE
projects/kubernetes-xxxxxx/locations/global/keyRings/sops/cryptoKeys/sops-key  ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE                  1           ENABLED

To encrypt secrets you need to create a .sops.yaml file in root your code repository.

creation_rules:
  - path_regex: \.yaml$
    gcp_kms: projects/kubernetes-xxxxxx/locations/global/keyRings/sops/cryptoKeys/sops-key
    encrypted_regex: ^(data|stringData)$

Let’s create a simple Kubernetes secret for testing.

$ cat secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

Encrypt your secret.yaml using SOPS with the following example.

$ sops -e secret.yaml
apiVersion: v1
kind: Secret
metadata:
    name: mysecret
    namespace: default
type: Opaque
data:
    username: ENC[AES256_GCM,data:<-HASH->,type:str]
    password: ENC[AES256_GCM,data:<-HASH->,type:str]
sops:
    kms: []
    gcp_kms:
    -   resource_id: projects/kubernetes-xxxxxx/locations/global/keyRings/sops/cryptoKeys/sops-key
        created_at: '2021-03-01T17:25:29Z'
        enc: <-HASH->
    azure_kv: []
    lastmodified: '2021-03-01T17:25:29Z'
    mac: ENC[AES256_GCM,data:<-HASH->,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.5.0

Alternatively you can encrypt and replace the file in-place.

$ sops -i -e secret.yaml

To decrypt the yaml file use sops -d or replace in-place using sops -i -d.

$ sops -d secret.yaml
apiVersion: v1
kind: Secret
metadata:
    name: mysecret
    namespace: default
type: Opaque
data:
    username: YWRtaW4=
    password: MWYyZDFlMmU2N2Rm

You can also edit an encrypted file with the default terminal editor by directly using the sops command without any options.

$ sops secret.yaml
File has not changed, exiting.

Let’s use the Flux CD Kustomize controller for this to decrypt Kubernetes secrets and apply to the specified namespace. First you need to create a GCP service account for Flux and grant the permission to decrypt.

Download the GCP json authentication file for the service account and create a new secret in the Flux namespace.

$ kubectl create secret generic gcp-auth -n gotk-system --from-file=./sops-gcp
$ kubectl get secrets -n gotk-system gcp-auth -o yaml
apiVersion: v1
data:
  sops-gcp: <-BASE64-ENCODED-GCP-AUTH-JSON->
kind: Secret
metadata:
  creationTimestamp: "2021-03-01T17:34:11Z"
  name: gcp-auth
  namespace: gotk-system
  resourceVersion: "1879000"
  selfLink: /api/v1/namespaces/gotk-system/secrets/gcp-auth
  uid: 10a14c1f-19a6-41a2-8610-694b12efefee
type: Opaque

You need to update the kustomize-controller deployment and add the volume mount for the sops GCP secret and the environment variable with the value where to find the Google application credential file. This is where my example is different to what is documented because I am not using integrated cloud authentication because my cluster is running locally.

...
    spec:
      containers:
...
      - env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /tmp/.gcp/credentials/sops-gcp
        name: manager
        volumeMounts:
        - mountPath: /tmp/.gcp/credentials
          name: sops-gcp
          readOnly: true
      volumes:
      - name: sops-gcp
        secret:
          defaultMode: 420
          secretName: sops-gcp
...

In the Kustomize object you enable the sops decryption provider and the controller automatically decrypts and applies secrets in the next reconcile loop.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: cluster
  namespace: gotk-system
spec:
  decryption:
    provider: sops
  interval: 5m0s
  path: ./clusters/cluster-dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: github-source

This takes a few minutes until the sync is completed and we can find out if the example secret got created correctly.

$ kubectl get secrets mysecret -n default -o yaml
apiVersion: v1
data:
  password: MWYyZDFlMmU2N2Rm
  username: YWRtaW4=
kind: Secret
metadata:
  name: mysecret
  namespace: default
  resourceVersion: "3439293"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: 4a009675-3c89-448b-bb86-6211cec3d4ea
type: Opaque

This is how to use SOPS and Flux CD v2 to decrypt and apply Kubernetes secrets using GitOps.

New Kubernetes GitOps Toolkit – Flux CD v2

I have been using the Flux CD operator for a few month to manage Kubernetes clusters in dev and prod and it is a great tool. When I initially reviewed Flux the first time back then, I liked it because of its simplicity but it was missing some important features such as the possibility to synchronise based on tags instead of a single branch, and configuring the Flux operator through the deployment wasn’t as good and intuitive, and caused some headaches.

A few days ago I stumbled across the new Flux CD GitOps Toolkit and it got my attention when I saw the new Flux v2 operator architecture. They’ve split the operator functions into three controller and using CRDs to configure Source, Kustomize and Helm configuration:

The feature which I was really waiting for was the support for Semantic Versioning semver in your GitRepository source. With this I am able to create platform releases, and can separate non-prod and prod clusters better which makes the deployment of configuration more controlled and flexible than previously with Flux v1.

You can see below the different release versions I’ve created in my cluster management repository:

The following two GitRepository examples; the first one syncs based on a static release tag 0.0.1 and the second syncs within a Semantic version range >=0.0.1 <0.1.0:

---
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  creationTimestamp: null
  name: gitops-system
  namespace: gitops-system
spec:
  interval: 1m0s
  ref:
    tag: 0.0.1
  secretRef:
    name: gitops-system
  url: ssh://github.com/berndonline/gitops-toolkit
status: {}
---
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  creationTimestamp: null
  name: gitops-system
  namespace: gitops-system
spec:
  interval: 1m0s
  ref:
    semver: '>=0.0.1 <0.1.0'
  secretRef:
    name: gitops-system
  url: ssh://github.com/berndonline/gitops-toolkit
status: {}

There are improvements for the Kustomize configuration to add additional overlays depending on your repository folder structure or combine this with another GitRepository source. In my example repository I have a cluster folder cluster-dev and a folder for common configuration:

.
|____cluster-dev
| |____kustomization.yaml
| |____hello-world_base
| | |____kustomization.yaml
| | |____deploy.yaml
|____common
  |____kustomization.yaml
  |____nginx-service.yaml
  |____nginx_base
    |____kustomization.yaml
    |____service.yaml
    |____nginx.yaml

You can add multiple Kustomize custom resources as you can see in my examples, one for the cluster specific config and a second one for the common configuration with can be applied to multiple clusters:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  creationTimestamp: null
  name: cluster-conf
  namespace: gitops-system
spec:
  interval: 5m0s
  path: ./cluster-dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: gitops-system
status: {}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  creationTimestamp: null
  name: common-con
  namespace: gitops-system
spec:
  interval: 5m0s
  path: ./common
  prune: true
  sourceRef:
    kind: GitRepository
    name: gitops-system
status: {}

Let’s install the Flux CD GitOps Toolkit. The toolkit comes again with its own command-line utility tk which you use to install and configure the operator . You find available CLI versions on the Github release page.

Set up a  new repository to store you k8s configuration:

$ git clone ssh://github.com/berndonline/gitops-toolkit
$ cd gitops-toolkit
$ mkdir -p ./cluster-dev/gitops-system

Generate the GitOps Toolkit manifests and store under gitops-system folder, afterwards apply the configuration to your k8s cluster:

$ tk install --version=latest \
    --export > ./cluster-dev/gitops-system/toolkit-components.yaml
$ kubectl apply -f ./cluster-dev/gitops-system/toolkit-components.yaml 
namespace/gitops-system created
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io created
role.rbac.authorization.k8s.io/crd-controller-gitops-system created
rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-system created
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-system created
service/notification-controller created
service/source-controller created
service/webhook-receiver created
deployment.apps/helm-controller created
deployment.apps/kustomize-controller created
deployment.apps/notification-controller created
deployment.apps/source-controller created
networkpolicy.networking.k8s.io/deny-ingress created

Check if all the pods are running and use the command tk check to see if the toolkit is working correctly:

$ kubectl get pod -n gitops-system
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-64f846df8c-g4mhv           1/1     Running   0          19s
kustomize-controller-6d9745c8cd-n8tth      1/1     Running   0          19s
notification-controller-587c49f7fc-ldcg2   1/1     Running   0          18s
source-controller-689dcd8bd7-rzp55         1/1     Running   0          18s
$ tk check
► checking prerequisites
✔ kubectl 1.18.3 >=1.18.0
✔ Kubernetes 1.18.6 >=1.16.0
► checking controllers
✔ source-controller is healthy
✔ kustomize-controller is healthy
✔ helm-controller is healthy
✔ notification-controller is healthy
✔ all checks passed

Now you can create a GitRepository custom resource, it will generate a ssh key local and displays the public key which you need to add to your repository deploy keys:

$ tk create source git gitops-system \
  --url=ssh://github.com/berndonline/gitops-toolkit \ 
  --ssh-key-algorithm=ecdsa \
  --ssh-ecdsa-curve=p521 \
  --branch=master \
  --interval=1m
► generating deploy key pair
ecdsa-sha2-nistp521 xxxxxxxxxxx
Have you added the deploy key to your repository: y
► collecting preferred public key from SSH server
✔ collected public key from SSH server:
github.com ssh-rsa xxxxxxxxxxx
► applying secret with keys
✔ authentication configured
✚ generating source
► applying source
✔ source created
◎ waiting for git sync
✗ git clone error: remote repository is empty

Continue with adding the Kustomize configuration:

$ tk create kustomization gitops-system \
  --source=gitops-system \
  --path="./cluster-dev" \
  --prune=true \
  --interval=5m
✚ generating kustomization
► applying kustomization
✔ kustomization created
◎ waiting for kustomization sync
✗ Source is not ready

Afterwards you can add your Kubernetes manifests to your repository and the operator will start synchronising the repository and apply the configuration which you’ve defined.

You can export the Source and Kustomize configuration:

$ tk export source git gitops-system \
 > ./cluster-dev/gitops-system/toolkit-source.yaml
$ tk export kustomization gitops-system \
 > ./cluster-dev/gitops-system/toolkit-kustomization.yaml

You basically finished installing the GitOps Toolkit and below you have some useful commands to reconcile the configured custom resources:

$ tk reconcile source git gitops-system
$ tk reconcile kustomization gitops-system

I was thinking of explaining how to setup a Kubernetes platform repository and do release versioning with the Flux GitOps Toolkit in one of my next articles. Please let me know if you have questions.