Node-Pools

TL;DR:

Node-pools are attached to clusters by calling node-pool modules.
A cluster has one or more node-pools.

Adding or Removing Node-Pools

Every Kubestack provisioned cluster comes with a default node-pool. The default node-pool is configured as part of the cluster module. Additional node-pools can be added or removed by adding or removing node-pool modules. The easiest way to do so is to ask your AI coding agent that has learned the Kubestack skill to scaffold the module for you.

Following the skill, your agent only makes changes to local files, and you can see the changes with git status. To have the changes take effect, you have to commit, push, merge and promote them following the GitOps flow.

Adding a Node-Pool

Adding a node-pool adds one file to your repository that calls the node-pool module. Ask your agent to scaffold the node-pool for you. Your agent configures the node-pool module to use the correct provider and attach to the correct cluster automatically. You can of course also write the configuration yourself, the configuration section below documents all available attributes.

AmazonAzureGoogleScaleway

Adding a node pool to your EKS cluster requires specifying the cluster name and the pool name.

Add an extra node-pool to the eks_gc0_eu-west-1 cluster

Removing a Node-Pool

You can also ask your agent to remove a node-pool for you.

# list all platform components
git ls-files '*_node_pool_*.tf'
aks_gc0_westeurope_node_pool_extra.tf
eks_gc0_eu-west-1_node_pool_extra.tf
gke_gc0_europe-west1_node_pool_extra.tf

# then ask your agent to remove the desired node-pool by <name>
Remove the eks_gc0_eu-west-1_node_pool_extra node-pool

Configuration

Node-pool module configuration is provider specific. See the commenteded examples below for available attributes.

AmazonAzureGoogleScaleway

module "eks_gc0_eu-west-1_node_pool_extra" {
  providers = {
    aws = aws.eks_gc0_eu-west-1
  }

  source = "github.com/kbst/terraform-kubestack//aws/cluster/node-pool?ref=<version>"

  cluster          = module.eks_gc0_eu-west-1.cluster
  cluster_metadata = module.eks_gc0_eu-west-1.current_metadata

  configuration = {
    apps = {
      # name of the node pool (required)
      name = "extra"

      # list of EC2 instance types to use (required)
      instance_types = ["t3a.xlarge", "t3a.large"]

      # min and max number of nodes (required)
      min_size = 3
      max_size = 9

      # desired number of nodes
      # (optional), the autoscaler owns the node count
      desired_capacity = 3

      # list of availability zones to use
      # (required)
      availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]

      # AMI type to use for nodes
      # (optional), default depends on instance types
      # AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD or AL2023_x86_64_NVIDIA
      ami_type = "AL2023_ARM_64_STANDARD"

      # instance metadata options for the nodes
      # (optional), defaults to null
      metadata_options = {
        http_endpoint               = "enabled"
        http_tokens                 = "required"
        http_put_response_hop_limit = 1
        http_protocol_ipv6          = "disabled"
        instance_metadata_tags      = "disabled"
      }

      # list of VPC subnet IDs to use for nodes
      # (optional), defaults to cluster's existing subnets
      vpc_subnet_ids = ["subnet-01234567890abcdef"]

      # secondary CIDR to associate with the cluster's VPC
      # (optional), defaults to null
      vpc_secondary_cidr = "10.1.0.0/16"
      
      # newbits to pass to cidsubnet function for new subnets
      # (optional), defaults to null
      # https://developer.hashicorp.com/terraform/language/functions/cidrsubnet
      vpc_subnet_newbits = 2
      
      # offset added to cidrsubnet's netnum parameter for new subnets
      # (optional), defaults to 1
      vpc_subnet_number_offset = 0

      # whether to map public IPs to nodes
      # (optional), defaults to true
      vpc_subnet_map_public_ip = false

      # Kubernetes taints to add to nodes
      # (optional), defaults to []
      taints = [
        {
          key    = "taint-key1"
          value  = "taint-value1"
          effect = "NO_SCHEDULE"
        },
        {
          key    = "taint-key2"
          value  = "taint-value2"
          effect = "PREFER_NO_SCHEDULE"
        }
      ]

      # additional AWS tags to add to node pool resources
      # (optional), defaults to {}
      tags = {
        "k8s.io/cluster-autoscaler/node-template/label/nvidia.com/gpu" = "true"
        "k8s.io/cluster-autoscaler/node-template/taint/dedicated"      = "nvidia.com/gpu=true"
      }

      # additional Kubernetes labels to add to nodes
      # (optional), defaults to {}
      labels = {
        "example.com/example" = "true"
      }
    }

    ops = {}
  }
}

Example new subnets

module "eks_gc0_eu-west-1_node_pool_new_subnets" {
  providers = {
    aws = aws.eks_gc0_eu-west-1
  }

  source = "github.com/kbst/terraform-kubestack//aws/cluster/node-pool?ref=<version>"

  cluster          = module.eks_gc0_eu-west-1.cluster
  cluster_metadata = module.eks_gc0_eu-west-1.current_metadata

  configuration = {
    apps = {
      name = "new-subnets"

      instance_types   = ["t3a.medium", "t3a.small"]
      desired_capacity = 1
      min_size         = 1
      max_size         = 3

      availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]

      # use the last three /18 subnets of the cluster's CIDR
      # https://www.davidc.net/sites/default/subnets/subnets.html?network=10.0.0.0&mask=16&division=23.ff4011
      vpc_subnet_newbits = 2
    }

    ops = {}
  }
}

Example new CIDR and subnets

module "eks_gc0_eu-west-1_node_pool_new_subnets_secondary_cidr" {
  providers = {
    aws = aws.eks_gc0_eu-west-1
  }

  source = "github.com/kbst/terraform-kubestack//aws/cluster/node-pool?ref=<version>"

  cluster          = module.eks_gc0_eu-west-1.cluster
  cluster_metadata = module.eks_gc0_eu-west-1.current_metadata

  configuration = {
    apps = {
      name = "new-subnets-secondary-cidr"

      instance_types   = ["t3a.medium", "t3a.small"]
      desired_capacity = 1
      min_size         = 1
      max_size         = 3

      availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]

      # add a secondary CIDR to the VPC and create subnets for it
      # https://www.davidc.net/sites/default/subnets/subnets.html?network=10.1.0.0&mask=16&division=23.ff4011
      vpc_secondary_cidr = "10.1.0.0/16"

      # with 3 zones we need at least 3 subnets, use the first three /18 subnets
      vpc_subnet_newbits       = 2
      vpc_subnet_number_offset = 0
    }

    ops = {}
  }
}