Inheritance Model

TL;DR:

  • Desired configuration is set for the mission critical apps infrastructure environment.
  • All other environments inherit configuration from apps.
  • The inherited configuration can be overwritten per environment.

Why inheritance

Inheritance from the mission critical apps to the loc and ops infrastructure environments is a cornerstone of Kubestack's reliable GitOps automation. The ops environment serves the purpose of validating changes before they are promoted to the apps environment. Configuration drift risks rendering this protection ineffective.

Inheritance makes differences explicit. By default, everything is inherited. But if necessary, individual attributes of the inherited configuration can be overwritten. Explicit differences do not prevent configuration drift. But they make it easier to spot.

Kubestack implements inheritance to reduce the risk of configuration drift and increase automation reliability. The fewer differences there are, between the environment a change was validated against, and the environment it is promoted to, the less likely the promotion is to fail.

Implementation

Kubestack implements the inheritance model for both cluster modules and cluster service modules.

All Kubestack modules accept two input variables:

  1. configuration
  2. configuration_base_key

Configuration expects a map where the keys are the names of the Terraform workspaces. And the values are the per workspace configurations. The configuration_base_key defaults to apps and controls which environment all others inherit their configuration from.

For historical reasons, configuration is of type map(map(string)) for cluster modules and type map(object()) for the newer cluster service modules. Object input variables have only recently become viable with bug free support for optional object type attributes with Terraform v0.15.0. The feature is still experimental but necessary for the cluster service modules. The intention is to migrate cluster modules as well once the feature is marked stable by upstream.

Default environments

Given Kubestack's default environment names, loc, ops and apps, this is the basic structure of the configuration map:

configuration = {
apps = {}
ops = {}
loc = {}
}

Custom environments

For custom environments, consider this example with separate clusters for the application production and staging environment. Organizing the environments like this limits the blast radius from infra automation with the ops environment. And the blast radius between application staging and application production with separate clusters.

configuration = {
apps-prod = {}
apps-stage = {}
ops = {}
loc = {}
}
configuration_base_key = "apps-prod"

Inheritance rules

The inheritance is implemented in the common/configuration module that all other Kubestack modules use internally. The module loops through all keys in the base environment (apps by default) and the current environment (determined by the value of the terraform.workspace variable, e.g. ops). If the key exists in the current environment, the value from the current environment is used. If it does not, the value from the base environment is used. This results in the following inheritance behaviour (assuming the default environment names):

  1. loc and ops inherit everything from apps
  2. any attribute that is set in loc or ops overwrites the inherited value from apps
  3. loc and ops can add attributes that are not set in apps

To explain this, take a look at the following examples:

  1. A hypothetical example showing the inheritance rules >>
  2. A practical example for scaling cluster and cluster service modules >>

Hypothetical example

Consider the following fictitious configuration:

module "configuration" {
source = "github.com/kbst/terraform-kubestack//common/configuration"
configuration = {
apps = {
apps_key1 = "from_apps"
apps_key2 = "from_apps"
}
ops = {
ops_key = "from_ops"
apps_key1 = "from_ops"
}
loc = {
loc_key = "from_loc"
}
}
base_key = "apps"
}
output "apps_merged" {
value = module.configuration.merged["apps"]
}
output "ops_merged" {
value = module.configuration.merged["ops"]
}
output "loc_merged" {
value = module.configuration.merged["loc"]
}

This will result in the following outputs:

Outputs:
apps_merged = {
"apps_key1" = "from_apps"
"apps_key2" = "from_apps"
}
# ops overwrites apps_key1
# ops inherits apps_key2 unchanged
# ops adds ops_key
ops_merged = {
"apps_key1" = "from_ops"
"apps_key2" = "from_apps"
"ops_key" = "from_ops"
}
# loc inherits apps_key1 and apps_key2 unchanged
# loc adds loc_key
loc_merged = {
"apps_key1" = "from_apps"
"apps_key2" = "from_apps"
"loc_key" = "from_loc"
}

It is not possible to overwrite an inherited value with null to remove the attribute from the configuration.

Practical examples

The practical examples below show how to use inheritance to scale cluster modules and cluster service modules per environment.

Cluster modules

The scaling configuration for a cluster is specified in the apps hash map. Half the compute resources would be wasted, if the ops environment used the exact same scaling settings.

Avoiding configuration differences that break the automation is important. And money spent on making the automation more reliable is an investment into sustainability for any team.

But a configuration that works for a three node cluster isn't likely to fail for a 30 node cluster. And within reason, neither is it, if ops uses burstable or shared-core nodes with reduced vCPUs and memory.

Below examples for AKS, EKS and GKE each auto-scale apps between 3 and 18 nodes (4 vCPUs, 16 GB memory) and ops between 3 and 6 nodes (2 vCPU burstable, 4 GB memory).

configuration = {
apps = {
# abbreviated example configuration
# ...
cluster_instance_type = "m5a.xlarge"
cluster_min_size = 3
cluster_max_size = 18
}
ops = {
# smaller, cheaper machine_type
cluster_instance_type = "t3a.medium"
# lower autoscaling min/max
cluster_min_size = 3
cluster_max_size = 6
}
loc = {}
}
configuration = {
apps = {
# abbreviated example configuration
# ...
default_node_pool_vm_size = "Standard_D4_v4"
default_node_pool_min_count = 3
default_node_pool_max_count = 18
}
ops = {
# smaller, cheaper machine_type
default_node_pool_vm_size = "Standard_B2s"
# lower autoscaling min/max
default_node_pool_min_count = 3
default_node_pool_max_count = 6
}
loc = {}
}
configuration = {
apps = {
# abbreviated example configuration
# ...
cluster_machine_type = "e2-standard-4"
cluster_min_node_count = 1
cluster_max_node_count = 6
# GKE min/max are per location
cluster_node_locations = "us-central1-a,us-central1-b,us-central1-c"
}
ops = {
# smaller, cheaper machine_type
cluster_machine_type = "e2-medium"
# lower autoscaling min/max
cluster_min_node_count = 1
cluster_max_node_count = 2
}
loc = {}
}

Cluster service modules

Similarly to how it makes sense to scale the cluster nodes differently for apps and ops. It also makes sense to scale cluster services differently. Since cluster modules and cluster service modules share the configuration inheritance, this works very similar. Just the configuration attributes are now Kubernetes/Kustomize specific and not AKS, EKS or GKE specific any more.

Below example scales the Nginx Ingress controller up to three replicas on apps and down to one replica on ops. The module defaults to two replicas.

configuration = {
apps = {
replicas = [{
name = "ingress-nginx-controller"
count = 3
}]
}
ops = {
replicas = [{
name = "ingress-nginx-controller"
count = 1
}]
}
loc = {}
}