Cluster Modules

TL;DR:

  • Cluster modules provision managed Kubernetes clusters and their direct cloud infrastructure dependencies.
  • Manage one or more clusters and mix multiple regions and even different cloud providers in the same root module.

What are cluster modules

Cluster modules are Terraform modules that provision a AKS, EKS or GKE managed Kubernetes cluster. Kubestack differentiates between two types of modules. Cluster modules and cluster service modules.

  1. Cluster modules provision managed Kubernetes clusters and their required infrastructure dependencies using the respective cloud's Terraform provider.
  2. Cluster service modules provision Kubernetes resources on top of Kubernetes clusters from upstream releases using the Kubestack maintained Kustomization provider.

Ultimately, Kubestack's cluster modules are just Terraform modules that implement standard Kubestack inputs and outputs. This provides a unified developer experience and makes for seamless integration with cluster service modules.

Most importantly, the Kubestack specific inheritance based configuration all modules implement is a key enabler for Kubestack's reliable GitOps automation.

Lifecycle integration

Cluster modules are implemented to provision cloud resources according to Kubestack's infrastructure environments and inheritance model.

They ensure a unified naming scheme, that works the same for:

  • one or multiple clusters
  • in one or more regions
  • on a single or multiple different cloud providers
  • across multiple infrastructure environments

In addition, cluster modules come in two variants:

  1. The cluster variant, that uses the respective cloud's Terraform provider and provisions real cloud resources.
  2. And a cluster-local variant, that uses the Terraform KinD provider to simulate a local development cluster.

By default ops and apps use the cluster variant. Only the kbst local apply command uses the cluster-local variant and the loc environment.

Modules are purpose built to provide multi-cluster and multi-cloud support. Integrate all resources fully into the Terraform plan/apply lifecycle. And are continuously tested to provide reliable automation following Kubestack's GitOps flow.

Module attributes

Cluster modules accept the following attributes:

  • source (required)

    Source of the module as required by Terraform. Cluster modules are available from GitHub.

    # source and version for a fictitious example module
    source = "github.com/kbst/terraform-kubestack//example/cluster?ref=v0.1.0"

Making cluster module's available from the Kubestack module registry, the same way as cluster service modules, is a work in progress.

  • configuration (required)

    Map of per environment configuration objects. Following Kubestack's inheritance model. The cluster module specific configuration is documented below.

    configuration = {
    apps = {
    # configuration attributes documented below
    }
    ops = {
    # inherits from apps
    }
    loc = {
    # inherits from apps
    }
    }
  • configuration_base_key (optional)

    Name of the key in the configuration map all others inherit from. Key must exist in configuration map. Defaults to apps.

    configuration = {
    apps-prod = {
    # every environment inhertis from apps-prod
    # because configuration_base_key is set to apps-prod
    }
    apps-stage = {
    # inherits from apps-prod
    }
    ops = {
    # inherits from apps-prod
    }
    loc = {
    # inherits from apps-prod
    }
    }
    configuration_base_key = "apps-prod"

Examples

Below examples show provider specific usage of the AKS, EKS and GKE cluster modules. Multiple, different cluster modules can be added to the same Terraform root module for multi-cluster, multi-region or multi-cloud use cases.

EKS requires the cluster module and a aws provider alias to configure the desired region. The alias attribute is used to pass a specific provider into a specific module.

module "eks_example" {
providers = {
aws = aws.eks_example
}
source = "github.com/kbst/terraform-kubestack//aws/cluster?ref=v0.14.1-beta.0"
configuration = {
apps = {
name_prefix = "example"
base_domain = "infra.example.com"
cluster_instance_type = "m5a.xlarge"
cluster_min_size = 3
cluster_max_size = 6
cluster_desired_capacity = 1
cluster_availability_zones = "eu-west-1a,eu-west-1b,eu-west-1c"
}
ops = {
cluster_instance_type = "t3a.medium"
cluster_min_size = 2
cluster_max_size = 2
}
loc = {}
}
}
provider "aws" {
alias = "eks_example"
region = "eu-west-1"
}

AKS requires the cluster module and a resource_group attribute. The resource group defines the region.

module "aks_example" {
source = "github.com/kbst/terraform-kubestack//azurerm/cluster?ref=v0.14.1-beta.0"
configuration = {
apps = {
resource_group = "example-resource-group"
name_prefix = "example"
base_domain = "infra.example.com"
default_node_pool_vm_size = "Standard_D4_v4"
default_node_pool_min_count = 3
default_node_pool_max_count = 6
}
ops = {
default_node_pool_vm_size = "Standard_B2s"
default_node_pool_min_count = 2
default_node_pool_max_count = 2
}
loc = {}
}
}

GKE requires the cluster module and a region attribute to configure the desired region. The *_node_count attributes for GKE are per node_location.

module "gke_example" {
source = "github.com/kbst/terraform-kubestack//google/cluster?ref=v0.14.1-beta.0"
configuration = {
apps = {
project_id = "example-project-id"
name_prefix = "example"
base_domain = "infra.example.com"
cluster_machine_type = "e2-standard-4"
cluster_min_node_count = 1
cluster_max_node_count = 2
region = "europe-west1"
cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d"
}
ops = {
cluster_machine_type = "e2-medium"
cluster_min_node_count = 1
cluster_max_node_count = 1
cluster_node_locations = "europe-west1-b,europe-west1-c"
}
loc = {}
}
}

Configuration

Cluster modules have two types of attributes. The common attributes are the same for all cluster modules. But the specific attributes are different for each module.

Common attributes

  • name_prefix (required)

    The name_prefix is concatenated with the current workspace and region: name_prefix-workspace-region. Whenever cluster modules create resources, the resulting name is used.

    Use a name_prefix that reflects the scope of the cluster. For example, the company, department, product or team name. Keep the prefix short to avoid running into character limits.

    name_prefix = "example"
  • base_domain (required)

    base_domain becomes a part of the fully qualified domain name (FQDN) in the format cluster_name.provider_name.base_domain.

    The provider_name is defined by the respective cluster modules:

    • AKS: azure
    • EKS: aws
    • GKE: gcp
    base_domain = "infra.example.com"

Specific attributes

  • cluster_availability_zones (required)

    Comma-separated list of availability zones to use for cluster nodes.

    cluster_availability_zones = "eu-west-1a,eu-west-1b,eu-west-1c"
  • cluster_instance_type (required)

    AWS EC2 instance type to use for cluster nodes.

    cluster_instance_type = "m5.xlarge"
  • cluster_desired_capacity (required)

    Desired number of worker nodes.

    cluster_desired_capacity = 1
  • cluster_max_size (required)

    Maximum number of worker nodes.

    cluster_max_size = 1
  • cluster_min_size (required)

    Minimum number of worker nodes.

    cluster_min_size = 1
  • cluster_version (optional)

    Desired Kubernetes control plane version. Defaults to the latest available version at resource creation. The value must be configured and increased to upgrade the version when desired. Downgrades are not supported by EKS.

    cluster_version = "1.19.8"
  • worker_root_device_encrypted (optional)

    Whether to encrypt root device volumes of worker nodes. Defaults to true.

    worker_root_device_encrypted = true
  • worker_root_device_volume_size (optional)

    Size in GB for root device volumes of worker nodes. Defaults to 20 GB.

    worker_root_device_volume_size = 20
  • cluster_aws_auth_map_roles (optional)

    Sets mapRoles attribute in the aws-auth configmap in the kube-system namespace.

    # each mapRoles entry maps an IAM role to a username and set of groups
    cluster_aws_auth_map_roles = <<-MAPROLES
    - rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin
    username: kubernetes-admin
    groups:
    - system:masters
    MAPROLES

    For more information please refer to the AWS EKS guide on user roles.

  • cluster_aws_auth_map_users (optional)

    Sets mapUsers attribute in the aws-auth configmap in the kube-system namespace.

    # each mapUsers entry maps an IAM role to a static username and set of groups
    cluster_aws_auth_map_users = <<-MAPUSERS
    - userarn: arn:aws:iam::000000000000:user/Alice
    username: alice
    groups:
    - system:masters
    MAPUSERS

    For more information please refer to the AWS EKS guide on user roles.

  • cluster_aws_auth_map_accounts (optional)

    Sets mapAccounts attribute in the aws-auth configmap in the kube-system namespace.

    # automatically map IAM ARN from these accounts to username.
    # NOTE: Always use quotes to avoid the account numbers being recognized as numbers
    # instead of strings by the yaml parser.
    cluster_aws_auth_map_accounts = <<-MAPACCOUNTS
    - "012345678901"
    - "456789012345"
    MAPACCOUNTS

    For more information please refer to the AWS EKS guide on user roles.

  • enabled_cluster_log_types (optional)

    Comma-separated list of cluster log types to enable. Defaults to "api,audit,authenticator,controllerManager,scheduler".

    # enable only a subset of log types
    enabled_cluster_log_types = "audit,authenticator"
  • resource_group (required)

    The Azure resource group to place the cluster resource in. The resource group in Azure determines the region.

    resource_group = "RESOURCE_GROUP_NAME"

    To query the list of available resource groups you can use az group list.

  • kubernetes_version (optional)

    Desired Kubernetes control plane version. Defaults to the latest available version at resource creation. Update value to start a version upgrade. Downgrades are not supported by AKS.

    kubernetes_version = "1.18.19"
  • dns_prefix (optional)

    The DNS prefix used for the Kubernetes API. Defaults to api.

    dns_prefix = "api"
  • sku_tier (optional)

    The SKU tier used for the AKS cluster. Defaults to Free.

    sku_tier = "Basic"
  • vnet_address_space (optional)

    Comma-separated list of vnet address space CIDRs. Defaults to "10.0.0.0/8".

    vnet_address_space = "10.0.0.0/8"
  • subnet_address_prefixes (optional)

    Comma-separated list of subnet address prefixes to configure. Defaults to "10.1.0.0/16".

    subnet_address_prefixes = "10.1.0.0/16,10.2.0.0/16"
  • subnet_service_endpoints (optional)

    Comma-separated list of subnet service endpoints to configure for the subnet.

    subnet_service_endpoints = "Microsoft.Sql,Microsoft.AzureCosmosDB"
  • network_plugin (optional)

    Type of network plugin to use. Defaults to kubenet.

    network_plugin = "azure"
  • network_policy (optional)

    Network policy option. Defaults to calico.

    network_plugin = "calico"
  • service_cidr (optional)

    Service CIDR. Defaults to 10.0.0.0/16.

    service_cidr = "10.0.0.0/16"
  • dns_service_ip (optional)

    DNS service IP. Defaults to 10.0.0.10.

    dns_service_ip = "10.0.0.10"
  • pod_cidr (optional)

    POD CIDR. Defaults to 10.244.0.0/16.

    pod_cidr = "10.244.0.0/16"
  • max_pods (optional)

    Max number of PODs. Defaults to AKS default.

    max_pods = 250
  • default_node_pool_name (optional)

    Name for the default node pool. Defaults to default.

    default_node_pool_name = "default"
  • default_node_pool_type (optional)

    Type of the node pool. Defaults to VirtualMachineScaleSets. Possible values are VirtualMachineScaleSets or AvailabilitySet. Only use AvailabilitySet for backwards compatibility with older clusters.

    default_node_pool_type = "VirtualMachineScaleSets"
  • default_node_pool_enable_auto_scaling (optional)

    Whether to enable auto scaling for the default node pool. Defaults to true.

    default_node_pool_enable_auto_scaling = false
  • default_node_pool_min_count (optional)

    Minimum number of worker nodes in the default node pool.

    default_node_pool_min_count = 3
  • default_node_pool_max_count (optional)

    Maximum number of worker nodes in the default node pool.

    default_node_pool_max_count = 9
  • default_node_pool_node_count (optional)

    Initial number of worker nodes in the default node pool.

    default_node_pool_node_count = 1
  • default_node_pool_vm_size (optional)

    VM size to use for the default node pool. Defaults to Standard_D1_v2.

    default_node_pool_vm_size = "Standard_D1_v2"
  • default_node_pool_os_disk_size_gb (optional)

    Size for the disk volume to use for the default node pool. Defaults to 30 GB.

    default_node_pool_os_disk_size_gb = 30
  • project_id (required)

    ID of the Google Cloud project to place the resources in.

    project_id = "PROJECT_ID"
  • region (required)

    The Google Cloud region to use for resources.

    region = "europe-west1"
  • cluster_node_locations (required)

    Comma-separated list of Google Cloud availability zones to use for nodes.

    # `region` and `cluster_node_locations` must match
    cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d"
  • cluster_min_master_version (required)

    Minimum master version for new clusters. Does not have effect for existing clusters.

    cluster_min_master_version = "1.19"
  • cluster_daily_maintenance_window_start_time (optional)

    Start time of the daily maintenance window. Defaults to 03:00 GMT.

    cluster_daily_maintenance_window_start_time = "03:00"
  • remove_default_node_pool (optional)

    Whether to remove the default node pool after cluster creation. Defaults to true.

    remove_default_node_pool = false

    Default node pools don't support auto-scaling. Kubestack removes the default node pool by default and creates a different one as the default.

  • cluster_initial_node_count (optional)

    Initial number of worker nodes in the default node pool per availability zone.

    cluster_initial_node_count = 1
  • cluster_min_node_count (optional)

    Minimum number of worker nodes in the default node pool per availability zone.

    cluster_min_node_count = 3
  • cluster_max_node_count (optional)

    Maximum number of worker nodes in the default node pool per availability zone.

    cluster_max_node_count = 9
  • cluster_extra_oauth_scopes (optional)

    Comma-separated list of additional OAuth scopes for worker nodes. Defaults to "".

    cluster_extra_oauth_scopes = "https://www.googleapis.com/auth/source.read_only,https://www.googleapis.com/auth/compute.readonly"
  • cluster_disk_size_gb (optional)

    Disk size in GB for worker nodes. Defaults to 100 GB.

    cluster_disk_size_gb = 100
  • cluster_disk_type (optional)

    Disk type used for worker nodes. Defaults to pd-standard. Available values are pd-standard or pd-ssd.

    cluster_disk_type = "pd-standard"
  • cluster_image_type (optional)

    Operating system image type for worker nodes. Defaults to COS.

    cluster_image_type = "COS"
  • cluster_machine_type (optional)

    Machine type used for worker nodes. Defaults to n1-standard-1.

    cluster_machine_type = "n1-standard-1"

    Custom machine types can be specified using custom-CPUs-MEMORY:

    # for 4 vCPUs and 4 GB of memory
    cluster_machine_type = "custom-4-4096"
  • cluster_preemptible (optional)

    Whether to use preemptible machines for the worker nodes. Defaults to false.

    cluster_preemptible = false
  • cluster_auto_repair (optional)

    Whether to enable cluster auto repair. Defaults to true.

    cluster_auto_repair = true
  • cluster_auto_upgrade (optional)

    Whether to enable cluster auto upgrades. Defaults to true.

    cluster_auto_upgrade = true