Framework

TL;DR:

• Desired configuration is set for the apps infrastructure environment.
• Ops inherits configuration from apps.
• The inherited configuration can be overwritten.

Inheritance from the apps to the ops infrastructure environment is a cornerstone of Kubestack's reliable GitOps automation. The ops environment serves the purpose to test and validate configuration before it is applied to the apps environment. Configuration drift risks rendering this protection ineffective.

Inheritance makes differences explicit. By default, everything is inherited. But if necessary, parts of the inherited configuration can be overwritten. Explicit differences do not prevent configuration drift. But they make it easier to spot.

By adopting inheritance Kubestack reduces the risk for configuration drift and increases automation reliability by avoiding potential errors and enforcing discipline across teams.

Kubestack implements the inheritance model for both infrastructure configuration and cluster manifests. The implementation depends on the repository to define both apps and ops. Terraform to implement the inheritance based on the workspace. And the CI/CD pipeline to select the workspace and running Terraform.

• Repository:
  • defines apps & ops configuration in config.auto.tfvars
  • defines apps & ops overlays under manifests/overlays
• Terraform:
  • for apps workspace
    • uses apps configuration
    • uses apps overlay
  • for ops workspace
    • uses merged apps + ops configuration
    • uses merged apps + ops overlay
• CI/CD Pipeline:
  • selects workspace apps or ops
  • runs terraform plan and terraform apply

The infrastructure configuration implements the inheritance model by having a map per cluster pair. Both apps and ops are keys of the cluster_pair map as shown below.

cluster_pair = {
  apps = {}
  ops = {}
}

By default, the configuration is specified in the apps hash map. Any value that is additionally set in the ops hash map, will overwrite the value from the apps hash map. Overwriting values can be useful to reduce the size of the ops cluster by using fewer nodes or also smaller instance types.

While avoiding configuration differences that break the automation is important, it may also be undesirable to have ops be the same size as apps because ops does not run any workloads.

Example

The example below configures an apps cluster's min node count to 1 and max node count to 10. It also sets the cluster's node locations to three availability zones in GCP's europe-west1 region. This apps cluster will auto scale between 3 and 30 nodes because node counts are per location. The ops cluster overwrites the auto scaling and node location attributes. This ops cluster would not auto scale and would have 2 nodes because max node count is set to 1 and it has two node locations instead of three.

A configuration like this may be a good compromise. It keeps a similar cluster architecture, with multiple nodes and locations. But it also reduces costs for the ops cluster.

clusters = {
  gke_zero = {
    apps = {
      # [...]
      cluster_min_node_count = 1
      cluster_max_node_count = 10
      region                 = "europe-west1"
      cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d"
    }
    ops = {
      cluster_max_node_count = 1
      cluster_node_locations = "europe-west1-b,europe-west1-c"
    }
  }
}

Kubestack uses Kustomize to maintain cluster manifests. Kustomize also uses inheritance. Its inheritance is implemented using bases and overlays. Overlays can inherit from one or more bases or even overlays. Bases inherit from nowhere.

Kubestack implements the inheritance for cluster manifests with a directory for all bases and two Kustomize overlays. By default, reference your bases in apps/kustomization.yaml and ops/kustomization.yaml will inherit everything from apps. If necessary, the ops overlay can be used to overwrite configuration inherited from apps. Follow the example below to try this for yourself.

The overlays directory layout for a fresh Kubestack repository looks like this:

manifests/overlays/
├── apps
│   └── kustomization.yaml
└── ops
    └── kustomization.yaml

To understand how the inheritance works, let's take a look at the two kustomization.yaml files.

First, manifests/overlays/apps/kustomization.yaml inherits from the Nginx ingress controller base.

# [...]
resources:
# Nginx ingress is included by default
- ../../bases/nginx/default-ingress

Second, manifests/overlays/ops/kustomization.yaml inherits everything from the apps overlay by referencing it under resources. This also includes the Nginx ingress controller.

# [...]
resources:
- ../apps

Example

The included Nginx ingress controller is configured to have two replicas. Following the smaller ops cluster example above, let's decrease the replicas for ops to one.

1. Add the built-in replicas field to ops/kustomization.yaml:

   # [...]
   resources:
   - ../apps
   replicas:
   - name: nginx-ingress-controller
     count: 1
   

2. To understand the differences, take a look at the kustomize build output of the two overlays:

   • kustomize build manifests/overlays/apps
   • kustomize build manifests/overlays/ops

   Compare how apps has replicas: 2, while ops has replicas: 1.

Refer to the Kustomize docs for more supported fields.

For more information on cluster manifests:

• Check the repository layout docs for details on Kustomize bases and overlays,
• or learn more about working with cluster manifests.