Cluster Pair Configuration

TL;DR:

Format

The Kubestack root module has a single clusters variable defined in variables.tf.

variable "clusters" {
description = "Map, holding configuration of all clusters."
type = map(map(map(string)))
}

Using a single variable reduces the number of steps necessary to add a new cluster pair.

The example configuration below consists of:

  1. the top level cluster hash map
  2. three example clusters: eks_zero, gke_zero and aks_zero
  3. each cluster's apps and ops configuration
# configuration excluding attributes
# available attributes are documented below
clusters = {
eks_zero = {
apps = {}
ops = {}
}
gke_zero = {
apps = {}
ops = {}
}
aks_zero = {
apps = {}
ops = {}
}
}

Module configuration includes both common attributes shared between AKS, EKS and GKE as well as provider specific attributes.

The configuration for cluster pairs follows the inheritance model. Consider the consequences for testability before overwriting values. Differences between ops and apps risk that a configuration applied against ops does fail when applied against apps.

Common Attributes

  • name_prefix (required)

    name_prefix becomes part of the cluster name along with the current Terraform workspace and the cluster's region. Values are concatenated to a cluster name in the format [name_prefix]-[workspace]-[region]. The region part of the cluster name follows the respective cloud provider's format.

    Consider using a name_prefix that reflects the purpose of the cluster. For smaller organizations the company or product name might be a good choice. For bigger organizations consider using department, team or product names. Keep the name_prefix short. Cloud providers almost always limit the number of characters for resource names.

    The examples below show resulting cluster names for different name_prefix values. The examples assume that the region is eu-west-1 on AWS.

    • name_prefix = "acmeinc" becomes:
      • acmeinc-ops-eu-west-1
      • acmeinc-apps-eu-west-1
    • name_prefix = "marketing" becomes:
      • marketing-ops-eu-west-1
      • marketing-apps-eu-west-1
    • name_prefix = "shop" becomes:
      • shop-ops-eu-west-1
      • shop-apps-eu-west-1
  • base_domain (required)

    base_domain becomes a part of the fully qualified domain name (FQDN) in the format [cluster_name].[provider_name].[base_domain].

    The examples below show the FQDN for different providers. Cluster modules set unique provider_name values. The base_domain for all three cluster pairs in this example is infra.example.com. The cluster names each use a provider specific region.

    • AKS:
      • acmeinc-ops-westeurope.azure.infra.example.com
      • acmeinc-apps-westeurope.azure.infra.example.com
    • EKS:
      • acmeinc-ops-eu-west-1.aws.infra.example.com
      • acmeinc-apps-eu-west-1.aws.infra.example.com
    • GKE:
      • acmeinc-ops-europe-west1.gcp.infra.example.com
      • acmeinc-apps-europe-west1.gcp.infra.example.com

The region influences the cluster name and ultimately the FQDN. It is defined differently depending on the provider.

  • AKS inherits the region from the resource_group
  • EKS inherits the region from the AWS provider's region attribute
  • GKE requires the region as an attribute of the cluster resource

Learn more in the respective provider's section about adding cluster pairs.

Provider Specific Attributes

  • cluster_availability_zones (required)

    Comma separated list of availability zones to use for cluster nodes. E.g. cluster_availability_zones = "eu-west-1a,eu-west-1b,eu-west-1c"

  • cluster_instance_type (required)

    AWS EC2 instance type to use for cluster nodes. E.g. cluster_instance_type = "m5.xlarge"

  • cluster_desired_capacity (required)

    Desired number of worker nodes. E.g. cluster_desired_capacity = "1"

  • cluster_max_size (required)

    Maximum number of worker nodes. E.g. cluster_max_size = "1"

  • cluster_min_size (required)

    Minimum number of worker nodes. E.g. cluster_min_size = "1"

  • root_device_encrypted (optional)

    Whether to encrypt root device volumes of worker nodes. Defaults to true. E.g. root_device_encrypted = true

  • root_device_volume_size (optional)

    Size in GB for root device volumes of worker nodes. Defaults to 20 GB. E.g. root_device_volume_size = 20

  • cluster_aws_auth_map_roles (optional)

    Sets mapRoles attribute in the aws-auth configmap in the kube-system namespace. E.g.:

    # each mapRoles entry maps an IAM role to a username and set of groups
    # Each username and group can optionally contain template parameters:
    # 1) "{{AccountID}}" is the 12 digit AWS ID.
    # 2) "{{SessionName}}" is the role session name.
    cluster_aws_auth_map_roles = <<MAPROLES
    - roleARN: arn:aws:iam::000000000000:role/KubernetesAdmin
    username: kubernetes-admin
    groups:
    - system:masters
    MAPROLES

    For more information please refer to the AWS-IAM-Authenticator config format documentation.

  • cluster_aws_auth_map_users (optional)

    Sets mapUsers attribute in the aws-auth configmap in the kube-system namespace. E.g.:

    # each mapUsers entry maps an IAM role to a static username and set of groups
    cluster_aws_auth_map_users = <<MAPUSERS
    - userARN: arn:aws:iam::000000000000:user/Alice
    username: alice
    groups:
    - system:masters
    MAPUSERS

    For more information please refer to the AWS-IAM-Authenticator config format documentation.

  • cluster_aws_auth_map_accounts (optional)

    Sets mapAccounts attribute in the aws-auth configmap in the kube-system namespace. E.g.:

    # automatically map IAM ARN from these accounts to username.
    # NOTE: Always use quotes to avoid the account numbers being recognized as numbers
    # instead of strings by the yaml parser.
    cluster_aws_auth_map_accounts = <<MAPACCOUNTS
    - "012345678901"
    - "456789012345"
    MAPACCOUNTS

    For more information please refer to the AWS-IAM-Authenticator config format documentation.

  • resource_group (required)

    The Azure resource group to place the cluster resource in. The resource group in Azure determines the region. E.g. resource_group = "RESOURCE_GROUP_NAME"

    To query the list of available resource groups you can use az group list.

  • dns_prefix (optional)

    The DNS prefix used for the Kubernetes api. Defaults to api. E.g. dns_prefix = "api"

  • default_node_pool_name (optional)

    Name for the default node pool. Defaults to default. E.g. default_node_pool_name = "default"

  • default_node_pool_type (optional)

    Type of the node pool. Defaults to VirtualMachineScaleSets. Possible values are VirtualMachineScaleSets or AvailabilitySet. Only use AvailabilitySet for backwards compatibility with older clusters. E.g. default_node_pool_type = "VirtualMachineScaleSets"

  • default_node_pool_enable_auto_scaling (optional)

    Whether to enable auto scaling for the default node pool. Defaults to true. E.g. default_node_pool_enable_auto_scaling = false

  • default_node_pool_min_count (optional)

    Minimum number of worker nodes in the default node pool. E.g. default_node_pool_min_count = "3"

  • default_node_pool_max_count (optional)

    Maximum number of worker nodes in the default node pool. E.g. default_node_pool_max_count = "9"

  • default_node_pool_node_count (optional)

    Initial number of worker nodes in the default node pool. E.g. default_node_pool_node_count = "1"

  • default_node_pool_vm_size (optional)

    VM size to use for the default node pool. Defaults to Standard_D1_v2. E.g. default_node_pool_vm_size = "Standard_D1_v2"

  • default_node_pool_os_disk_size_gb (optional)

    Size for the disk volume to use for the default node pool. Defaults to 30 GB. E.g. default_node_pool_os_disk_size_gb = "30"

  • project_id (required)

    ID of the Google Cloud project to place the resources in. E.g. project_id = "PROJECT_ID"

  • region (required)

    The Google Cloud region to use for resources. E.g. region = "europe-west1".

  • cluster_node_locations (required)

    Comma separated list of Google Cloud availability zones to use for nodes. E.g. cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d". Note: region and cluster_node_locations must match.

  • cluster_min_master_version (required)

    Minimum master version for new clusters. Does not have effect for existing clusters. E.g. cluster_min_master_version = "1.13" will create a cluster using the latest 1.13.x patch version.

  • cluster_daily_maintenance_window_start_time (optional)

    Start time of the daily maintenance window. Defaults to 03:00 GMT. E.g. cluster_daily_maintenance_window_start_time = "03:00"

  • remove_default_node_pool (optional)

    Whether to remove the default node pool after cluster creation. Defaults to true. E.g. remove_default_node_pool = false

    Default node pools don't support auto-scaling. Kubestack removes the default node pool by default and creates a different one as the default.

  • cluster_initial_node_count (optional)

    Initial number of worker nodes in the default node pool per availability zone. E.g. cluster_initial_node_count = "1"

  • cluster_min_node_count (optional)

    Minimum number of worker nodes in the default node pool per availability zone. E.g. cluster_min_node_count = "3"

  • cluster_max_node_count (optional)

    Maximum number of worker nodes in the default node pool per availability zone. E.g. cluster_max_node_count = "9"

  • cluster_extra_oauth_scopes (optional)

    Comma separated list of additional OAuth scopes for worker nodes. Defaults to "". E.g. cluster_extra_oauth_scopes = "https://www.googleapis.com/auth/source.read_only,https://www.googleapis.com/auth/compute.readonly"

  • cluster_disk_size_gb (optional)

    Disk size in GB for worker nodes. Defaults to 100 GB. E.g. cluster_disk_size_gb = 100

  • cluster_disk_type (optional)

    Disk type used for worker nodes. Defaults to pd-standard. Available values are pd-standard or pd-ssd. E.g. cluster_disk_type = "pd-standard"

  • cluster_image_type (optional)

    Operating system image type for worker nodes. Defaults to COS. E.g. cluster_image_type = "COS"

  • cluster_machine_type (optional)

    Machine type used for worker nodes. Defaults to n1-standard-1. E.g. cluster_machine_type = "n1-standard-1". Custom machine types can be specified using custom-CPUs-MEMORY, e.g. cluster_machine_type = "custom-4-4096" for an instance with 4 virtual CPU cores and 4 GB of memory.

  • cluster_preemptible (optional)

    Whether to use preemptible machines for the worker nodes. Defaults to false. E.g. cluster_preemptible = false

  • cluster_auto_repair (optional)

    Whether to enable cluster auto repair. Defaults to true. E.g. cluster_auto_repair = true

  • cluster_auto_upgrade (optional)

    Whether to enable cluster auto upgrades. Defaults to true. E.g. cluster_auto_upgrade = true