Framework Guides

Provision Cert-Manager and Let's Encrypt on Kubernetes using Terraform

TL;DR:

• Use the Cert-Manager Terraform module from the Kubestack catalog
• Provision Cert-Manager 1.14.3-kbst.0 on Kubernetes
• Configure Cert-Manager to use Let's Encrypt to issue certificates

Introduction

No matter if web site, web app or API, any service exposed to the internet must have transport layer security (TLS) enabled. And this requires a certificate signed by a certificate authority that browsers and other HTTP clients trust.

Following this tutorial, you will provision Cert-Manager and configure it to issue Let's Encrypt certificates via Kubernetes custom resources.

Let's Encrypt is a nonprofit certificate authority providing TLS certificates for free and fully automated. Cert-Manager is a Kubernetes operator, that can provision certificates from certificate authorities like Let's Encrypt automatically.

First step is to install Cert-Manager on the Kubernetes cluster. We will use the Kubestack Cert-Manager Terraform module for that. Like all Kubestack platform service modules, the Cert-Manager module bundles the Kubernetes YAML from upstream releases and packages it as a Terraform module.

The modules use the Kubestack maintained Kustomization provider to fully integrate the Cert-Manager Kubernetes resources into the Terraform lifecycle. It also allows to customize the configuration without modifications to the upstream YAML. This has the benefit that the custom configuration is significantly less likely to break on future updates.

Cert-Manager Installation

To install the Cert-Manager module, run the following kbst CLI command in the root of your repository.

# add cert-manager service to every cluster
# append --cluster-name NAME
# to only add to a single cluster
kbst add service cert-manager

Module Configuration

Now you have one *_service_cert-manager.tf file per cluster. The files are almost identical between AKS, EKS or GKE. Main difference is the aliased kustomization provider, that controls what cluster Cert-Manager will be installed on.

Leaving source and version aside, the configuration attribute is where we will now configure Let's Encrypt. All platform service modules allow adding additional resources to be applied alongside the bundled upstream YAML using additional_resources. We will use this to include our Let's Encrypt ClusterIssuer below.

Let's Encrypt ClusterIssuer

To configure Cert-Manager to issue certificates using Let's Encrypt we apply a ClusterIssuer custom resource. For this we first create a YAML file in manifests/cluster-issuer.yaml with the configuration for Let's Encrypt.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Apply ClusterIssuer alongside upstream YAML

Then we instruct the Cert-Manager module to apply our ClusterIssuer alongside the upstream YAML on the respective Kubernetes cluster. For this we add additional_resources to the external environment (usually apps or apps-prod) in our module configuration. Do this for every *_service_cert-manager.tf file in your repository.

configuration = {
  apps = {
+     additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }
  ops = {}
}

Patch ClusterIssuer to use Let's Encrypt staging

Let's Encrypt has strict API rate limits. Since the Kubestack ops environment does not run any application workloads, we don't need certificates that are trusted by browsers here. This is also a great opportunity to show how to patch upstream YAML using the Kubestack platform service modules and how to overwrite the inherited configuration from apps or apps-prod in ops.

configuration = {
  apps = {
    additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }
  ops = {
+   patches = [
+     {
+       patch = <<-EOF
+         - op: replace
+           path: /spec/acme/server
+           value: https://acme-staging-v02.api.letsencrypt.org/directory
+       EOF
+          
+       target = {
+         group   = "cert-manager.io"
+         version = "v1"
+         kind    = "ClusterIssuer"
+         name    = "letsencrypt"
+       }
+     }
+   ]
  }
}

Apply Changes

As with every change, we now follow the GitOps process. First, commit and push to start the peer review, then merge when the plan looks good. After the changes have been validated in the internal environment, promote the changes to the external environment.

The full workflow is documented on the GitOps process page.

But here's a short summary for convenience:

# create a new feature branch
git checkout -b add-cert-manager

# add the changes and commit them
git add .
git commit -m "Install cert-manager and let's encrypt issuer"

# push the changes to trigger the pipeline
git push origin add-cert-manager

Then follow the link in the output, to create a new pull request. Review the pipeline run. And merge the pull request, when everything is green.

Last but not least, promote the changes once you validated them in ops by setting a tag.

# make sure you're on the merge commit
git checkout main
git pull

# then tag the commit 
git tag apps-deploy-$(git rev-parse --short HEAD)

# finally push the tag, to trigger the pipeline to promote
git push origin apps-deploy-$(git rev-parse --short HEAD)