Framework Documentation

Extending Kubestack

TL;DR:

• Use custom OpenTofu modules to extend your platform with on-premise or cloud resources.
• Integrate custom modules into the framework by adopting Kubestack metadata and configuration inheritance.
• Provision any Kubernetes YAML using the kustomization overlay module.

Introduction

You can extend a Kubestack platform to integrate with existing infrastructure or to provision resources that Kubestack does not provide modules for. There are two main use-cases.

1. For infrastructure requirements you can write custom OpenTofu modules and integrate them with the environments and outputs from the Kubestack modules.
2. For Kubernetes manifests you want to provision as part of your platform, Kubestack provides the kustomization overlay module. Using the overlay module, you can integrate and configure any upstream YAML the same way as all platform feature modules.

Cluster Module Outputs

To integrate with cloud resources that are created by Kubestack modules, the modules provide OpenTofu outputs. You can combine the outputs with OpenTofu data sources, to query information about created resources that you may need as inputs to other resources.

Output current_config

The current_config output has the config used for the cluster in the current environment.

hclCopy
# module.eks_gc0_eu-west-1.current_config
eks_gc0_eu-west-1_current_config = {
  "base_domain"                = "kubestack.example.com"
  "cluster_availability_zones" = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  "cluster_version"            = "1.30"
  "default_node_pool" = {
    "desired_capacity" = 3
    "instance_types"   = ["t3a.xlarge"]
    "max_size"         = 9
    "min_size"         = 3
  }
  "name_prefix" = "gc0"
}

Output current_metadata

The current_metadata output holds all the information that the metadata module generated for the cluster.

hclCopy
# module.eks_gc0_eu-west-1.current_metadata
eks_gc0_eu-west-1_current_metadata = {
  "domain"          = "aws.kubestack.example.com"
  "fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
  "label_namespace" = "kubestack.com/"
  "labels" = {
    "kubestack.com/cluster_domain"          = "aws.kubestack.example.com"
    "kubestack.com/cluster_fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
    "kubestack.com/cluster_name"            = "gc0-ops-eu-west-1"
    "kubestack.com/cluster_provider_name"   = "aws"
    "kubestack.com/cluster_provider_region" = "eu-west-1"
    "kubestack.com/cluster_workspace"       = "ops"
  }
  "name" = "gc0-ops-eu-west-1"
  "tags" = [
    "gc0-ops-eu-west-1",
    "ops",
    "aws",
    "eu-west-1",
  ]
}

Output kubeconfig

The kubeconfig output, provides a KUBECONFIG file that can be used to connect and authenticate to the Kubernetes API server of the respective cluster. This output is used to configure the kustomization providers used for platform feature modules.

The following examples show how to use it with different OpenTofu providers focused on Kubernetes.

hclCopy
provider "kustomization" {
  alias = "gke_gc0_europe-west1"

  kubeconfig_raw = module.gke_gc0_europe-west1.kubeconfig
}

hclCopy
locals {
  gke_gc0_europe-west1_kubeconfig = yamldecode(module.gke_gc0_europe-west1.kubeconfig)
}

provider "kubernetes" {
  alias = "gke_gc0_europe-west1"

  host                   = local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["server"]
  cluster_ca_certificate = base64decode(local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["certificate-authority-data"])
  token                  = local.gke_gc0_europe-west1_kubeconfig["users"][0]["user"]["token"]
}

OpenTofu Modules

Kubestack modules, are just OpenTofu modules. And consequently they can coexist alongside your custom modules in the same root module.

Query Using Data Sources

OpenTofu data sources can be used to query information about an existing resource. The primary resource is usually the cluster. Once you have the data source, you can access the attributes of the respective cluster resource.

The following examples show how to query information about an AKS, EKS or GKE clusters using a data source and the Kubestack module outputs.

hclCopy
data "aws_eks_cluster" "eks_gc0_eu-west-1" {
  provider = aws.eks_gc0_eu-west-1

  name = module.eks_gc0_eu-west-1.current_metadata.name
}

Unique Names per Environment

Kubestack uses OpenTofu workspaces for its environments. When you create cloud resources, names usually have to be unique for one account or even globally. To make sure names are unique per environment, you can include the environment name in the resource name. The easiest way to achieve that is to include the terraform.workspace in the name.

hclCopy
locals {
  example_name = "example-${terraform.workspace}"
}

Kubestack itself goes a step further and includes a name_prefix, the terraform.workspace and the region in names. This provides names that are unique per environment, region and cloud provider.

The module that implements this, is used by all other Kubestack modules. And you can use the common metadata module to generate names and other metadata in your custom modules aswell.

hclCopy
module "example_metadata" {
  source = "github.com/kbst/terraform-kubestack//common/metadata?ref=<version>"

  name_prefix = "example"
  base_domain = var.base_domain

  provider_name   = "gcp" # kubestack uses aws, gcp or azure
  provider_region = module.gke_gc0_europe-west1.current_config["region"]

  # it's good practice to namespace labels
  label_namespace = "example.com/"
}

You can then access the same metadata that all Kubestack cluster modules provide. See the section about the current_metadata OpenTofu output above for more details.

hclCopy
output "example_metadata_name" {
  value = module.example_metadata.name
}

output "example_metadata_labels" {
  value = module.example_metadata.labels
}

Inheritance Based Configuration

To build custom modules that implement the same inheritance based configuration as Kubestack modules, use the common configuration module.

hclCopy
module "example_configuration" {
  source = "github.com/kbst/terraform-kubestack//common/configuration?ref=<version>"

  configuration = var.configuration
  base_key      = var.configuration_base_key
}

The configuration module expects to be given the configuration, as a map of maps. Where the keys of the outer map, are the environment names. And the base_key sets which environment all others inherit from. Most commonly you'll use this module inside another module, and the configuration and the base_key are passed in using variables when calling your module.

Kubernetes Manifests

Kubestack provides the kustomization overlay module to apply any Kubernetes YAML. The overlay module accepts the same configuration attributes as platform feature modules. You can use it to integrate any Kubernetes YAML into your Kubestack platform the same way as any platform feature.

Usage example

To include custom YAML manifests:

1. place the YAML files in a directory under manifests/
2. call the overlay module and reference the YAML files as resources

Below example uses the overlay module to apply fictitious upstream YAML. It also sets an env label for all resources with the current OpenTofu workspace as the value.

hclCopy
module "example_custom_manifests" {
  providers = {
    kustomization = kustomization.gke_gc0_europe-west1
  }

  source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=<version>"

  configuration = {
    apps = {
      namespace = "example-${terraform.workspace}"

      resources = [
        "${path.module}/manifests/example/upstream.yaml"
      ]

      common_labels = {
        "env" = terraform.workspace
      }
    }

    ops = {}
  }
}

Overlay Configuration Attributes

The overlay module configuration attributes are identical to platform feature module configuration attributes. Use resources to specify a Kustomization or the individual YAML files to provision.

hclCopy
module "example_custom_manifests" {
  providers = {
    kustomization = kustomization.example_cluster_alias
  }

  # kustomization overlay module, use the framework version
  # that all other modules in your repository use
  source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=<version>"

  configuration = {
    apps = {
      # list of paths to YAML files or Kustomizations to deploy (required)
      resources = [
        # kustomization
        "${path.module}/manifests/example-kustomization",

        # or individual YAML files
        "${path.module}/manifests/example/namespace.yaml",
        "${path.module}/manifests/example/deployment.yaml",
        "${path.module}/manifests/example/service.yaml",
      ]

      # set annotations on all Kubernetes resources
      # (optional), defaults to null
      common_annotations = {
        "example-annotation" = var.example_annotation
      }

      # set labels on all Kubernetes resources
      # (optional), defaults to null
      common_labels = {
        "example-label" = var.example_label
      }

      # generate Kubernetes configMaps
      # (optional), defaults to null
      config_map_generator = [{
        # name (required)
        # Sets 'metadata.name' of the configMap resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the configMap resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"  

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the configMap.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the configMap per line in the env file.
        envs = [
          "${path.module}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.module}/manifests/cfg.ini",          # data["cfg.ini"] = file_content
          "prod.ini=${path.module}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this configMap.
        options = {
          # see generator_options
        }
      }]

      # generate Kubernetes secrets
      # (optional), defaults to null
      secret_generator = [{
        # name (required)
        # Sets 'metadata.name' of the secret resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the secret resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"

        # type (optional)
        # 'type' attribute of the secret to generate.
        type = "generic"

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the secret.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the secret per line in the env file.
        envs = [
          "${path.module}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.module}/manifests/cfg.ini",          # data["cfg.ini"] = file_content
          "prod.ini=${path.module}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this secret.
        options = {
          # see generator_options
        }
      }]

      # set options for all configMap and secret generators
      # (optional), defaults to null
      generator_options = {
        # annotations (optional)
        # Sets 'metadata.annotations' on the generated resources. Defaults to '{}'.
        annotations = {
          example-annotation = "example"
        }
        
        # labels (optional)
        # Sets 'metadata.labels' on the generated resources. Defaults to '{}'.
        labels = {
          example-label = "example"
        }

        # disable_name_suffix_hash (optional)
        # Disables hash suffix of generated resource's 'metadata.name'.
        # Defaults to 'false'.
        disable_name_suffix_hash = true
      }

      # patch image names, tags or digests
      # (optional), defaults to null
      images = [{
        # Refers to the 'pod.spec.container.name' to modify the 'image' attribute of.
        name     = "busybox"
        
        # Customize the 'registry/name' part of the image. The part before the ':'
        new_name = "new_name"
        
        # Customize the 'tag' part of the image. The part after the ':'.
        new_tag  = "new_tag"

        # Replace the 'tag' part of an image with a 'digest'.
        digest   = "sha256:..."
      }]

      # prefix or suffix to add to resource names
      # (optional), default to null
      name_prefix = "prefix-"
      name_suffix = "-suffix"

      # namespace to set for all resources
      # (optional), default to null
      namespace = "example"

      # patches to apply to resources
      # (optional), default to null
      patches = [{
        # path (optional)
        # Path to a file that defines a strategic merge or JSON patch.
        path = "${path.module}/manifests/patch.yaml"

        # patch (optional)
        # Inline string that defines a strategic merge or JSON patch.
        patch = <<-EOF
          - op: replace
            path: /metadata/name
            value: newname
        EOF

        # target (optional)
        # Target one or multiple resources to be patched.
        target = {
          group               = ""
          version             = "v1"
          kind                = "ConfigMap"
          name                = "example"
          namespace           = "example"
          label_selector      = "key=value"
          annotation_selector = "key=value"
        }
      }]

      # set replicas attribute of deployments and similar resources
      # (optional), default to null
      replicas = [{
        # Refers to the 'metadata.name' of the resource to scale
        name = "example"
        
        # Sets the desired number of replicas.
        count = 5
      }]
    }

    ops  = {}
  }
}

low-level arguments

In addition to the convenience kustomization attributes documented above, platform feature modules also pass the following low-level attributes through to the kustomization provider. These Kustomization attributes are less useful in the OpenTofu context and as such are not documented here.

• components (optional) List of paths to Kustomize components.
• crds (optional) List of paths to Kustomize CRDs.
• generators (optional) List of paths to Kustomize generators.
• transformers (optional) List of paths to Kustomize transformers.
• vars (optional) List of objects to define Kustomize vars.