Framework Guides

Provision Cert-Manager and Let's Encrypt on Kubernetes using Terraform

TL;DR:

Use the Cert-Manager Terraform module from the Kubestack catalog
Provision Cert-Manager 1.8.2-kbst.0 on Kubernetes
Configure Cert-Manager to use Let's Encrypt to issue certificates

Introduction

No matter if web site, web app or API, any service exposed to the internet must have transport layer security (TLS) enabled. And this requires a certificate signed by a certificate authority that browsers and other HTTP clients trust.

Following this tutorial, you will provision Cert-Manager and configure it to issue Let's Encrypt certificates via Kubernetes custom resources.

Let's Encrypt is a nonprofit certificate authority providing TLS certificates for free and fully automated. Cert-Manager is a Kubernetes operator, that can provision certificates from certificate authorities like Let's Encrypt automatically.

First step is to install Cert-Manager on the Kubernetes cluster. We will use the Kubestack Cert-Manager Terraform module for that. Like all Kubestack cluster service modules, the Cert-Manager module bundles the Kubernetes YAML from upstream releases and packages it as a Terraform module.

The modules use the Kubestack maintained Kustomization provider to fully integrate the Cert-Manager Kubernetes resources into the Terraform lifecycle. It also allows to customize the configuration without modifications to the upstream YAML. This has the benefit that the custom configuration is significantly less likely to break on future updates.

Before we can provision Cert-Manager, we need a Kubestack repository. If you do not have a Kubestack repository yet, follow the Kubestack tutorial first. While the catalog modules can be used with any Terraform configuration, this tutorial assumes you have a Kubestack framework repository.

Cert-Manager Installation

Open Kubestack Cloud
Open the stack corresponding to your platform in Kubestack Cloud.
Add Cert-Manager cluster service
Your created stack may have one or more clusters. The clusters will have at least one node pool configured. And they will by default have at least one cluster service, Nginx Ingress configured.
To now add the Cert-Manager cluster service:
1. click + Add next to a cluster's name, and select cluster service from the dropdown
2. then select cert-manager and the latest version from the respective name and version dropdowns
Repeat these steps for every cluster in your platform that you want to have Cert-Manager installed on.
Export to Terraform
Once done, click the Export to Terraform button in the top right corner. After extracting the downloaded file, you will find multiple Terraform files in your repository. Files are grouped by managed Kubernetes solutions (aks, eks or gke), cluster prefix (kbst by default) and finally cloud region (e.g. europe-west4).
You will find a _cluster.tf for each cluster with its respective prefix and also a corresponding _service-cert-manager.tf with the same prefix. The cluster file defines the cluster and the service file installs the service on it. For more details please refer to the repository layout documentation page.
Copy the *-service-cert-manager.tf file(s) from the downloaded archive into your existing repository.

Module Configuration

Now you should have one _service-cert-manager.tf file per cluster. The files call the Kubestack Cert-Manager Terraform module which already bundles the upstream YAML. You can control the upstream release with the version attribute. Since we just created the files we're already using the latest available version.

The files are almost identical between AKS, EKS or GKE. Main difference is the aliased kustomization provider, that controls what cluster Cert-Manager will be installed on.

Leaving source and version aside, the configuration attribute is where we will now configure Let's Encrypt. The configuration attribute allows setting anything you can set in a kustomization.yaml file to customise the bundled upstream YAML. You can find all available configuration attributes, and the inheritance between environments, in the cluster service documentation.

module "eks_kbst_eu-west-1_service_cert-manager" {
  providers = {
    kustomization = kustomization.eks_kbst_eu-west-1
  }

  source  = "kbst.xyz/catalog/cert-manager/kustomization"
  version = "1.8.2-kbst.0"

  configuration = {
    apps = {}
    ops = {}
  }
}

module "aks_kbst_westeurope_service_cert-manager" {
  providers = {
    kustomization = kustomization.aks_kbst_westeurope
  }

  source  = "kbst.xyz/catalog/cert-manager/kustomization"
  version = "1.8.2-kbst.0"

  configuration = {
    apps = {}
    ops = {}
  }
}

module "gke_kbst_europe-west1_service_cert-manager" {
  providers = {
    kustomization = kustomization.gke_kbst_europe-west1
  }

  source  = "kbst.xyz/catalog/cert-manager/kustomization"
  version = "1.8.2-kbst.0"

  configuration = {
    apps = {}
    ops = {}
  }
}

Let's Encrypt ClusterIssuer

To configure Cert-Manager to issue certificates using Let's Encrypt we need to apply a ClusterIssuer custom resource. For this we first create a YAML file in manifests/cluster-issuer.yaml with the configuration for Let's Encrypt.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Apply ClusterIssuer alongside upstream YAML

Then we instruct the Cert-Manager module to apply our ClusterIssuer alongside the upstream YAML on the respective Kubernetes cluster. For this we add additional_resources to the apps or apps-prod key in our module configuration.

When you created your platform stack, you chose between shared or separate workload clusters. Select shared or separate below, to see the example configuration that inherits from apps or apps-prod respectively.

configuration = {
  apps = {
    additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }

  ops = {}
}

configuration_base_key = "apps-prod"
configuration = {
  apps-prod = {
    additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }

  apps = {}

  ops = {}
}

Patch ClusterIssuer to use Let's Encrypt staging

Let's Encrypt has strict API rate limits. Since the Kubestack ops environment does not run any application workloads, we don't need certificates that are trusted by browsers here. This is also a great opportunity to show how to patch upstream YAML using the Kubestack cluster service modules and how to overwrite the inherited configuration from apps or apps-prod in ops.

configuration = {
  apps = {
    additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }

  ops = {
    patches = [
      {
        patch = <<-EOF
          - op: replace
            path: /spec/acme/server
            value: https://acme-staging-v02.api.letsencrypt.org/directory
        EOF

        target = {
          group   = "cert-manager.io"
          version = "v1"
          kind    = "ClusterIssuer"
          name    = "letsencrypt"
        }
      }
    ]
  }
}

configuration_base_key = "apps-prod"
configuration = {
  apps-prod = {
    additional_resources = ["${path.root}/manifests/cluster-issuer.yaml"]
  }

  apps = {}

  ops = {
    patches = [
      {
        patch = <<-EOF
          - op: replace
            path: /spec/acme/server
            value: https://acme-staging-v02.api.letsencrypt.org/directory
        EOF

        target = {
          group   = "cert-manager.io"
          version = "v1"
          kind    = "ClusterIssuer"
          name    = "letsencrypt"
        }
      }
    ]
  }
}

Apply Changes

Like for every change, it's time to commit and push to to review. Merge when the plan looks good. And finally promote the changes, once they have been validated in ops.

The full workflow is documented on the GitOps process page.

But here's a short summary for convenience:

# create a new feature branch
git checkout -b add-cert-manager

# add the changes and commit them
git add .
git commit -m "Install cert-manager and let's encrypt issuer"

# push the changes to trigger the pipeline
git push origin add-cert-manager

Then follow the link in the output, to create a new pull request. Review the pipeline run. And merge the pull request, when everything is green.

Last but not least, promote the changes once you validated them in ops by setting a tag.

# make sure you're on the merge commit
git checkout main
git pull

# then tag the commit 
git tag apps-deploy-$(date -I)-0

# finally push the tag, to trigger the pipeline to promote
git push origin apps-deploy-$(date -I)-0