Cluster Service Modules

TL;DR:

  • Cluster service modules provision Kubernetes resources on clusters provisioned using cluster modules.
  • Consider cluster service modules for anything that's required before deploying application workloads.
  • Kubernetes resources managed by cluster service modules are fully integrated into the Terraform lifecycle.
  • Cluster services can be from the catalog or your own custom manifests.

What are cluster service modules

Cluster service modules are Terraform modules that provision Kubernetes resources on a Kubernetes cluster. Kubestack differentiates between two types of modules. Cluster modules and cluster service modules.

  1. Cluster modules provision managed Kubernetes clusters and their required infrastructure dependencies using the respective cloud's Terraform provider.
  2. Cluster service modules provision Kubernetes resources on top of Kubernetes clusters from upstream releases using the Kubestack maintained Kustomization provider.

The Kubestack catalog curates Terraform modules for popular Kubernetes services that are continuosly updated and tested.

Ultimately, Kubestack's cluster service modules are just Terraform modules that implement standard Kubestack inputs and outputs. This provides a unified developer experience and makes for seamless integration with cluster modules.

Most importantly, the Kubestack specific inheritance based configuration all modules implement is a key enabler for Kubestack's reliable GitOps automation.

Lifecycle integration

Due to their shared nature, just like changes to the infrastructure itself, changes to cluster services need to be validated before they can affect applications.

Cluster service modules, using the Kustomization provider, fully integrate their Kubernetes resources into the Terraform lifecycle.

This ensures that cluster manifests are part of Kubestack's gitops process and that changes are validated against ops before they are promoted to apps.

Module attributes

Cluster service modules accept the following attributes:

  • source (required)

    Source of the module as required by Terraform. Cluster service modules are available from the Kubestack module registry.

    # source for a fictitious example module
    source = "kbst.xyz/catalog/example/kustomization"
  • version (required)

    Version of the module to be installed. The version for Kubestack cluster service modules consists of the upstream version and a -kbst.x packaging suffix. The packaging suffix counter x is incremented for new module versions that do not change the upstream version.

    # fictitious example module version
    version = "0.1.0-kbst.0"

The -kbst.x packaging suffix means version constraint operators like =, !=, >, >=, <, <= and ~> can not be used for cluster module versions.

  • providers (required)

    The providers attribute is used to explicitly define which cluster a cluster service is provisioned on. Kubestack cluster modules provide a kubeconfig output, that can be used to configure a kustomization alias provider. Explicitly pass this alias provider into the cluster service module.

    # fictitious example alias provider
    providers = {
    kustomization = kustomization.example
    }
  • configuration (optional)

    Map of per environment configuration objects. Following Kubestack's inheritance model. The cluster service module specific configuration object is documented below. Defaults to { apps = {}, ops = {}, loc = {} }.

    configuration = {
    apps = {
    # configuration attributes documented below
    }
    ops = {
    # inherits from apps
    }
    loc = {
    # inherits from apps
    }
    }
  • configuration_base_key (optional)

    Name of the key in the configuration map all others inherit from. Key must exist in configuration map. Defaults to apps.

    configuration = {
    apps-prod = {
    # every environment inhertis from apps-prod
    # because configuration_base_key is set to apps-prod
    }
    apps-stage = {
    # inherits from apps-prod
    }
    ops = {
    # inherits from apps-prod
    }
    loc = {
    # inherits from apps-prod
    }
    }
    configuration_base_key = "apps-prod"

Examples

Below examples show how the kustomization provider is configured using the kubeconfig output of a cluster module. And how the provider is explicitly passed into a fictitious cluster service module.

module "eks_zero" {
# cluster module attributes excluded for brevity
}
provider "kustomization" {
# alias reflects cluster module name
alias = "eks_zero"
# uses kubeconfig output from cluster module
kubeconfig_raw = module.eks_zero.kubeconfig
}
module "eks_zero_example" {
providers = {
# explicitly pass provider alias
kustomization = kustomization.eks_zero
}
source = "kbst.xyz/catalog/example/kustomization"
version = "0.1.0-kbst.0"
configuration = {
apps = {
# overwrites namespace for all module resources
namespace = "new-namespace"
}
ops = {}
loc = {}
}
}
module "aks_zero" {
# cluster module attributes excluded for brevity
}
provider "kustomization" {
# alias reflects cluster module name
alias = "aks_zero"
# uses kubeconfig output from cluster module
kubeconfig_raw = module.aks_zero.kubeconfig
}
module "aks_zero_example" {
providers = {
# explicitly pass provider alias
kustomization = kustomization.aks_zero
}
source = "kbst.xyz/catalog/example/kustomization"
version = "0.1.0-kbst.0"
configuration = {
apps = {
# overwrites namespace for all module resources
namespace = "new-namespace"
}
ops = {}
loc = {}
}
}
module "gke_zero" {
# cluster module attributes excluded for brevity
}
provider "kustomization" {
# alias reflects cluster module name
alias = "gke_zero"
# uses kubeconfig output from cluster module
kubeconfig_raw = module.gke_zero.kubeconfig
}
module "gke_zero_example" {
providers = {
# explicitly pass provider alias
kustomization = kustomization.gke_zero
}
source = "kbst.xyz/catalog/example/kustomization"
version = "0.1.0-kbst.0"
configuration = {
apps = {
# overwrites namespace for all module resources
namespace = "new-namespace"
}
ops = {}
loc = {}
}
}

Configuration

The configuration input variable is of type map(object({...})). All attributes below are attributes of the inner object.

  • additional_resources (optional)

    List of paths to YAML files or Kustomizations to append to the module's default list of resources. Defaults to [].

    # add one more resource to the resources the module provisions
    additional_resources = [
    "${path.root}/manifests/resource.yaml"
    ]
  • common_annotations (optional)

    A map of strings to set commonAnnotations in the kustomization. Adds or overwrites the respective annotations on all Kubernetes resources the module provisions.

    common_annotations = {
    "example-annotation" = var.example_annotation
    }
  • common_labels (optional)

    A map of strings to set commonLabels in the kustomization. Adds or overwrites the respective labels and if applicable labelSelectors on all Kubernetes resources the module provisions. Changes to immutable labelSelectors will lead to a destory and recreate plan of affected Kubernetes resources.

    common_labels = {
    "example-label" = var.example_label
    }
  • config_map_generator (optional)

    A list of objects that configure Kustomize configMapGenerators. Generates Kubernetes configMaps that will be provisioned by the module.

    config_map_generator = [{
    # name (required)
    # Sets 'metadata.name' of the configMap resource
    name = "example"
    # namespace (required)
    # Sets 'metadata.namespace' of the configMap resource
    namespace = "example"
    # behavior (optional)
    # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
    behavior = "create"
    # literals (optional)
    # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the configMap.
    literals = [
    "KEY=VALUE"
    ]
    # envs (optional)
    # List of paths (strings) to env files (one KEY=VALUE pair per line).
    # Sets 'data[KEY] = VALUE' in the configMap per line in the env file.
    envs = [
    "${path.root}/manifests/env"
    ]
    # files (optional)
    # List of paths (strings) to files.
    # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
    # Overwrite by prefixing 'customkey='.
    files = [
    "${path.root}/manifests/cfg.ini" # data["cfg.ini"] = file_content
    "prod.ini=${path.root}/manifests/cfg.ini" # data["prod.ini"] = file_content
    ]
    # options (optional)
    # Same as top level 'generator_options' but specific to this configMap.
    options = {
    # see generator_options
    }
    }]
  • generator_options (optional)

    Object defining Kustomize generatorOptions. This generator_options attribute will apply to all resources that are generated inside the module. To set options only for a specific config_map_generator or secret_generator use their options child attribute.

    generator_options = {
    # annotations (optional)
    # Sets 'metadata.annotations' on the generated resources. Defaults to '{}'.
    annotations = {
    example-annotation = "example"
    }
    # labels (optional)
    # Sets 'metadata.labels' on the generated resources. Defaults to '{}'.
    labels = {
    example-label = "example"
    }
    # disable_name_suffix_hash (optional)
    # Disables hash suffix of generated resource's 'metadata.name'.
    # Defaults to 'false'.
    disable_name_suffix_hash = true
    }
  • images (optional)

    A list of objects defining the Kustomize images transformer to patch container image attributes.

    images = [{
    # Refers to the 'pod.spec.container.name' to modify the 'image' attribute of.
    name = "busybox"
    # Customize the 'registry/name' part of the image. The part before the ':'
    new_name = "new_name"
    # Customize the 'tag' part of the image. The part after the ':'.
    new_tag = "new_tag"
    # Replace the 'tag' part of an image with a 'digest'.
    digest = "sha256:..."
    }]
  • name_prefix/name_suffix (optional)

    Strings to set a Kustomize namePrefix or nameSuffix transformer to add a prefix or suffix to resources' metadata.name.

    name_prefix = "prefix-"
    name_suffix = "-suffix"
  • namespace (optional)

    String to configure the Kustomize namespace transformer to set or overwrite metadata.namespace for all resources.

    namespace = "example"
  • patches (optional)

    List of objects to configure Kustomize patches to apply to resources.

    patches = [{
    # path (optional)
    # Path to a file that defines a strategic merge or JSON patch.
    path = "${path.root}/manifests/patch.yaml"
    # patch (optional)
    # Inline string that defines a strategic merge or JSON patch.
    patch = <<-EOF
    - op: replace
    path: /metadata/name
    value: newname
    EOF
    # target (optional)
    # Target one or multiple resources to be patched.
    target = {
    group = ""
    version = "v1"
    kind = "ConfigMap"
    name = "example"
    namespace = "example"
    label_selector = "key=value"
    annotation_selector = "key=value"
    }
    }]
  • replicas (optional)

    A list of objects to configure the Kustomize replicas transformer for Deployment, ReplicationController, ReplicaSet or StatefulSet resources.

    replicas = [{
    # Refers to the 'metadata.name' of the resource to scale
    name = "example"
    # Sets the desired number of replicas.
    count = 5
    }]
  • resources (optional)

    List of paths to YAML files or Kustomizations. Defaults to concat(["./variant"], additional_resources). By default cluster service modules include the upstream resources, defined by the module's variant.

    Overwriting resources entirely is possible. But only useful in rare cases. Most use cases should prefer addtional_resources to append resources and keep upstream resources included.

    # overwrite the resources the module provisions
    resources = [
    "${path.module}/variant"
    ]
  • secret_generator (optional)

    A list of objects that define Kustomize secretGenerators to generate Kubernetes Secrets.

    secret_generator = [{
    # name (required)
    # Sets 'metadata.name' of the secret resource
    name = "example"
    # namespace (required)
    # Sets 'metadata.namespace' of the secret resource
    namespace = "example"
    # behavior (optional)
    # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
    behavior = "create"
    # type (optional)
    # 'type' attribute of the secret to generate.
    type = "generic"
    # literals (optional)
    # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the secret.
    literals = [
    "KEY=VALUE"
    ]
    # envs (optional)
    # List of paths (strings) to env files (one KEY=VALUE pair per line).
    # Sets 'data[KEY] = VALUE' in the secret per line in the env file.
    envs = [
    "${path.root}/manifests/env"
    ]
    # files (optional)
    # List of paths (strings) to files.
    # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
    # Overwrite by prefixing 'customkey='.
    files = [
    "${path.root}/manifests/cfg.ini" # data["cfg.ini"] = file_content
    "prod.ini=${path.root}/manifests/cfg.ini" # data["prod.ini"] = file_content
    ]
    # options (optional)
    # Same as top level 'generator_options' but specific to this secret.
    options = {
    # see generator_options
    }
    }]
  • variant (optional)

    Name of one of the module variants. Defaults to a module specific default variant. The chosen variant defines the default upstream manifests the module includes. Available variants are documented on the catalog page of each respective cluster service module.

    variant = "example"
  • low-level arguments

    In addition to the convenience kustomization attributes documented above, cluster service modules also pass the following low-level attributes through to the kustomization provider. These Kustomization attributes are less useful in the Terraform context and as such are not documented here.

    • components (optional) List of paths to Kustomize components.
    • crds (optional) List of paths to Kustomize CRDs.
    • generators (optional) List of paths to Kustomize generators.
    • transformers (optional) List of paths to Kustomize transformers.
    • vars (optional) List of objects to define Kustomize vars.

Custom manifests

To provision custom manifests on clusters Kubestack provides the custom-manifests module. This module accepts the same configuration attributes as cluster service modules documented above. But unlike the cluster service modules it does not bundle any upstream manifests. So it can be used to integrate bespoke Kubernetes YAML into the Kubestack automation.

The custom-manifests module is the recommended migration path for any existing automation that previously relied on the deprecated cluster manifests feature.

Cluster services vs. application workloads

Not all Kubernetes manifests should be part of the infrastructure automation repository. Include manifests that provide services shared between applications and environments. Everything that's required before the cluster is ready to run workloads. Exclude manifests that merely define workloads.

Examples of manifests that are commonly included:

  • Ingress controllers or service meshes, but not the ingress resources or mesh resources.
  • Namespaces, including their RBAC and quota manifests.
  • Operators, and their custom resource definitions. But not their custom resources.
  • Admission controllers for policy enforcement and the policies.
  • Monitoring and logging exporters/agents/forwarders.

Deploying application workloads as part of the Kubestack automation is discouraged, because it breaks separation of concerns and often leads to circular dependencies.

However, there is no technical limitation that prevents the custom-manifests module, or the underlying provider from being used for application manifests. You can do so inside the Kubestack repository and pipeline. Or preferably, from a separate repository and pipeline.

Usage example

To include custom YAML manifests:

  1. place the YAML files under manifests/
  2. call the custom-manifests module and reference the YAML files as resources

Below example uses the custom-manifests module to create a namespace from YAML and set an env label from the current Terraform workspace:

module "example_custom_manifests" {
providers = {
kustomization = kustomization.example
}
source = "kbst.xyz/catalog/custom-manifests/kustomization"
version = "0.1.0"
configuration = {
apps = {
namespace = "example-${terraform.workspace}"
resources = [
"${path.root}/manifests/example/namespace.yaml",
"${path.root}/manifests/example/deployment.yaml",
"${path.root}/manifests/example/service.yaml"
]
common_labels = {
"env" = terraform.workspace
}
}
ops = {}
loc = {}
}
}