Cluster Pairs

TL;DR:

  • Manage one or more cluster pairs from a single repository.
  • Mix multiple regions and even different cloud providers.
  • Add, remove or update cluster pairs, without affecting other pairs in the repository.

Adding a cluster pair

Adding a cluster pair is just a change in the repository. After changing the files, follow the commit, push, review and merge change management process.

Adding a new cluster pair on AWS requires three steps. The example below adds a new EKS cluster pair named eks_one to an existing repository. The repository already has one cluster pair named eks_zero.

  1. Add the EKS module to clusters.tf

    module "eks_zero" {
    # existing module attributes excluded for brevity
    }
    # Module to add for the new cluster pair
    module "eks_one" {
    providers = {
    aws = aws.eks_one
    }
    source = "github.com/kbst/terraform-kubestack//aws/cluster?ref=v0.9.4-beta.0"
    configuration = var.clusters["eks_one"]
    }

    Multi cloud: Cluster pair modules can be of different types. You can mix and match AKS, EKS and GKE in the same clusters.tf file.

  2. Add configuration for the new cluster pair to config.auto.tfvars

    clusters = {
    eks_zero = {
    # existing cluster config excluded for brevity
    }
    # Configuration to add for new cluster pair
    eks_one = {
    apps = {
    name_prefix = "NAME_PREFIX"
    base_domain = "BASE_DOMAIN"
    cluster_instance_type = "t2.small"
    cluster_desired_capacity = "1"
    cluster_min_size = "1"
    cluster_max_size = "1"
    cluster_availability_zones = "eu-west-1a,eu-west-1b,eu-west-1c"
    }
    ops = {
    cluster_max_size = "1"
    cluster_availability_zones = "eu-west-1a,eu-west-1b"
    }
    }
    }

    Configuration attributes are EKS specific. Ops inherits configuration from apps. Refer to the cluster pair configuration documentation for details.

  3. Add the provider for the new cluster pair to providers.tf

    provider "aws" {
    alias = "eks_zero"
    region = "us-east-2"
    }
    # Provider to add for the new cluster pair
    provider "aws" {
    alias = "eks_one"
    region = "eu-west-1"
    }

    Multi region: AWS cluster pairs in different regions require the region set in the respective provider. The alias attribute is used to link a module to a specific provider. Refer to the Terraform documentation for more information on multiple provider instances.

Adding a new cluster pair on Microsoft Azure requires two steps. The example below adds a new AKS cluster pair named aks_one to an existing repository. The repository already has one cluster pair named aks_zero.

  1. Add the AKS module to clusters.tf

    module "aks_zero" {
    # existing cluster config excluded for brevity
    }
    # Module to add for the new cluster pair
    module "aks_one" {
    source = "github.com/kbst/terraform-kubestack//azurerm/cluster?ref=v0.9.4-beta.0"
    configuration = var.clusters["aks_one"]
    }

    Multi cloud: Cluster pair modules can be of different types. You can mix and match AKS, EKS and GKE in the same clusters.tf file.

  2. Add configuration for the new cluster pair to config.auto.tfvars

    clusters = {
    aks_zero = {
    # existing cluster config excluded for brevity
    }
    # Configuration to add for new cluster pair
    aks_one = {
    apps = {
    resource_group = "RESOURCE_GROUP"
    name_prefix = "NAME_PREFIX"
    base_domain = "BASE_DOMAIN"
    }
    ops = {}
    }
    }

    Configuration attributes are AKS specific. Ops inherits configuration from apps. Refer to the cluster pair configuration documentation for details.

    Multi region: The region is defined when creating the resource group. Specify different resource groups to have cluster pairs in different regions.

Adding a new cluster pair on Google Cloud requires two steps. The example below adds a new GKE cluster pair named gke_one to an existing repository. The repository already has one cluster pair named gke_zero.

  1. Add the GKE module to clusters.tf

    module "gke_zero" {
    # existing cluster config excluded for brevity
    }
    # Module to add for the new cluster pair
    module "gke_one" {
    source = "github.com/kbst/terraform-kubestack//google/cluster?ref=v0.9.4-beta.0"
    configuration = var.clusters["gke_one"]
    }

    Multi cloud: Cluster pair modules can be of different types. You can mix and match AKS, EKS and GKE in the same clusters.tf file.

  2. Add configuration for the new cluster pair to config.auto.tfvars

    clusters = {
    gke_zero = {
    # existing cluster config excluded for brevity
    }
    # Configuration to add for new cluster pair
    gke_one = {
    apps = {
    project_id = "PROJECT_ID"
    name_prefix = "NAME_PREFIX"
    base_domain = "BASE_DOMAIN"
    cluster_min_master_version = "1.13"
    cluster_min_node_count = 1
    cluster_max_node_count = 1
    region = "europe-west1"
    cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d"
    }
    ops = {
    cluster_node_locations = "europe-west1-b"
    }
    }
    }

    Configuration attributes are GKE specific. Ops inherits configuration from apps. Refer to the cluster pair configuration documentation for details.

    Multi region: The region attribute of the configuration defines a cluster's region. Node locations have to match the selected region. Setting different regions for ops and apps is discouraged because it breaks testability.

Updating a cluster pair

Updating the framework version is just a change in the repository. After changing the files, follow the commit, push, review and merge change management process.

Updating a cluster pair requires two changes in the repository:

  1. Updating the module version of the cluster pair in clusters.tf

    The version is specified using the ref URL parameter of the module's source attribute. Please refer to the releases page for available versions. Updates are only tested in order. Skipping versions may cause issues.

    Below example shows a diff of upgrading a GKE module from version v0.9.2-beta.0 to v0.9.4-beta.0.

    diff --git a/clusters.tf b/clusters.tf
    index ab41adc..6548b7e 100644
    --- a/clusters.tf
    +++ b/clusters.tf
    @@ -1,5 +1,5 @@
    module "gke_zero" {
    - source = "github.com/kbst/terraform-kubestack//google/cluster?ref=v0.9.2-beta.0"
    + source = "github.com/kbst/terraform-kubestack//google/cluster?ref=v0.9.4-beta.0"
    configuration = var.clusters["gke_zero"]
    }
  2. Updating the Dockerfile to update dependency versions

    Update your local Dockerfile to the one from the latest release to make sure you have the latest dependencies.

With the changes done, commit, push, review and merge as usual.

Updates causing a Terraform plan to delete and re-create resources may cause downtime. Teams have two options in such a case:

  • Postpone the update to a maintenance window, where the impact of unavailability is acceptable.
  • Not updating the existing cluster pair. But instead adding a cluster pair, then migrating workloads and finally removing the old cluster pair.

Removing a cluster pair

Removing a cluster pair is just another change in the repository. After changing the files, follow the commit, push, review and merge change management process.

To remove a cluster pair, make the following changes:

  • Remove the module instance from clusters.tf.
  • Cleanup the configuration in config.auto.tfvars.
  • For AWS, delete the aliased provider from providers.tf.

Refer to the step above about adding a cluster pair and undo the changes documented there.

Module Attributes

Cluster pair modules have two attributes to set their configuration and optionally overwrite their manifest path defined in each module's variables.tf.

variable "configuration" {
type = map(map(string))
description = "Map with per workspace cluster configuration."
}
variable "configuration_base_key" {
type = string
description = "The key in the configuration map all other keys inherit from."
default = "apps"
}
variable "manifest_path" {
type = string
description = "Path to Kustomize overlay to build."
default = null
}
  • configuration (required)

    Configuration hash map for the cluster module. Including apps and ops child maps. Refer to the dedicated cluster pair configuration documentation page for available configuration attributes. The apps and ops configuration follows the inheritance model.

    The example below shows the eks_zero cluster module referencing the var.clusters["eks_zero"] configuration in config.auto.tfvars:

    1. The module:

      module "eks_zero" {
      # [...]
      configuration = var.clusters["eks_zero"]
      }
    2. The configuration:

      clusters = {
      eks_zero = {
      apps = {
      # [...]
      }
      ops = {
      # [...]
      }
      }
      }
  • configuration_base_key (optional)

    For custom environments set configuration_base_key to the environment key all other's should inherit configuration from.

    The example shows the gke_zero cluster module referencing a custom environment configuration in config.auto.tfvars with three environments named apps-prod, apps-stage and ops whereby both apps-stage and ops inherit from apps-prod:

    1. The module:

      module "gke_zero" {
      # [...]
      configuration = var.clusters["gke_zero"]
      configuration_base_key = "apps-prod"
      }
    2. The configuration:

      clusters = {
      gke_zero = {
      apps-prod = {
      # [...]
      }
      apps-stage = {
      # [...]
      }
      ops = {
      # [...]
      }
      }
      }

    For a more complete example of what changes are required to have custom environments please refer to this custom environments example repository.

  • manifest_path (optional)

    Path to this cluster pair's Kustomize overlay directory. By default, this points to manifests/overlays/apps or manifests/overlays/ops depending on the current Terraform workspace. The apps and ops overlays follow the inheritance model.

    Multi cloud repositories, for example, can require different Kubernetes resource configurations per cloud provider. To achieve this, extend the overlays with an apps and ops overlay per cloud provider. Then overwrite the manifest_path of each cluster pair to point to the provider specific overlay.

    The example below shows this for a repository with a EKS and a GKE cluster pair.

    1. The extended overlays directory layout:

      manifests/overlays/
      ├── eks
      │ ├── apps
      │ └── ops
      └── gke
      ├── apps
      └── ops
    2. The overwritten manifest_path attributes for the cluster pairs:

      module "eks_zero" {
      # [...]
      manifest_path = "manifests/overlays/eks/${terraform.workspace}"
      }
      module "gke_zero" {
      # [...]
      manifest_path = "manifests/overlays/gke/${terraform.workspace}"
      }

Use this feature to adapt your repository to use-cases that require multi-level inheritance. Take a look at the multi-cluster-multi-tenant-kustomize repository for an example of multiple inheritance levels.

When you overwrite the path, consider the impact on the apps and ops overlay inheritance. Do not break testability.