Framework Documentation

TL;DR:

• Desired configuration is set for the mission critical apps infrastructure environment.
• All other environments inherit configuration from apps.
• The inherited configuration can be overwritten per environment.

Inheritance from the mission critical apps to the loc and ops infrastructure environments is a cornerstone of Kubestack's reliable GitOps automation. The ops environment serves the purpose of validating changes before they are promoted to the apps environment. Configuration drift risks rendering this protection ineffective.

Inheritance makes differences explicit. By default, everything is inherited. But if necessary, individual attributes of the inherited configuration can be overwritten. Explicit differences do not prevent configuration drift. But they make it easier to spot.

Kubestack implements inheritance to reduce the risk of configuration drift and increase automation reliability. The fewer differences there are, between the environment a change was validated against, and the environment it is promoted to, the less likely the promotion is to fail.

Kubestack implements the inheritance model for both cluster modules and cluster service modules.

All Kubestack modules accept two input variables:

1. configuration
2. configuration_base_key

Configuration expects a map where the keys are the names of the Terraform workspaces. And the values are the per workspace configurations. The configuration_base_key defaults to apps and controls which environment all others inherit their configuration from.

Default environments

Given Kubestack's default environment names, loc, ops and apps, this is the basic structure of the configuration map:

configuration = {
  apps = {}
  ops = {}
  loc = {}
}

Custom environments

For custom environments, consider this example with separate clusters for the application production and staging environment. Organizing the environments like this limits the blast radius from infra automation with the ops environment. And the blast radius between application staging and application production with separate clusters.

configuration = {
  apps-prod = {}
  apps-stage = {}
  ops = {}
  loc = {}
}
configuration_base_key = "apps-prod"

The inheritance is implemented in the common/configuration module that all other Kubestack modules use internally. The module loops through all keys in the base environment (apps by default) and the current environment (determined by the value of the terraform.workspace variable, e.g. ops). If the key exists in the current environment, the value from the current environment is used. If it does not, the value from the base environment is used. This results in the following inheritance behaviour (assuming the default environment names):

1. loc and ops inherit everything from apps
2. any attribute that is set in loc or ops overwrites the inherited value from apps
3. loc and ops can add attributes that are not set in apps

To explain this, take a look at the following examples:

1. A hypothetical example showing the inheritance rules >>
2. A practical example for scaling cluster and cluster service modules >>

Consider the following fictitious configuration:

module "configuration" {
  source = "github.com/kbst/terraform-kubestack//common/configuration"
  configuration = {
    apps = {
      apps_key1 = "from_apps"
      apps_key2 = "from_apps"
    }
    ops = {
      ops_key   = "from_ops"
      apps_key1 = "from_ops"
    }
    loc = {
      loc_key = "from_loc"
    }
  }
  base_key = "apps"
}
output "apps_merged" {
  value = module.configuration.merged["apps"]
}
output "ops_merged" {
  value = module.configuration.merged["ops"]
}
output "loc_merged" {
  value = module.configuration.merged["loc"]
}

This will result in the following outputs:

Outputs:
apps_merged = {
  "apps_key1" = "from_apps"
  "apps_key2" = "from_apps"
}
# ops overwrites apps_key1
# ops inherits apps_key2 unchanged
# ops adds ops_key
ops_merged = {
  "apps_key1" = "from_ops"
  "apps_key2" = "from_apps"
  "ops_key" = "from_ops"
}
# loc inherits apps_key1 and apps_key2 unchanged
# loc adds loc_key
loc_merged = {
  "apps_key1" = "from_apps"
  "apps_key2" = "from_apps"
  "loc_key" = "from_loc"
}

The practical examples below show how to use inheritance to scale cluster modules and cluster service modules per environment.

Cluster modules

The scaling configuration for a cluster is specified in the apps hash map. Half the compute resources would be wasted, if the ops environment used the exact same scaling settings.

Avoiding configuration differences that break the automation is important. And money spent on making the automation more reliable is an investment into sustainability for any team.

But a configuration that works for a three node cluster isn't likely to fail for a 30 node cluster. And within reason, neither is it, if ops uses burstable or shared-core nodes with reduced vCPUs and memory.

Below examples for AKS, EKS and GKE each auto-scale apps between 3 and 18 nodes (4 vCPUs, 16 GB memory) and ops between 3 and 6 nodes (2 vCPU burstable, 4 GB memory).

Cluster service modules

Similarly to how it makes sense to scale the cluster nodes differently for apps and ops. It also makes sense to scale cluster services differently. Since cluster modules and cluster service modules share the configuration inheritance, this works very similar. Just the configuration attributes are now Kubernetes/Kustomize specific and not AKS, EKS or GKE specific any more.

Below example scales the Nginx Ingress controller up to three replicas on apps and down to one replica on ops. The module defaults to two replicas.

configuration = {
  apps = {
    replicas = [{
      name = "ingress-nginx-controller"
      count = 3
    }]
  }
  ops = {
    replicas = [{
      name = "ingress-nginx-controller"
      count = 1
    }]
  }
  loc = {}
}