Inheritance Model
TL;DR:
- Desired configuration is set for the apps infrastructure environment.
- Ops inherits configuration from apps.
- The inherited configuration can be overwritten.
Why inheritance
Inheritance from the apps to the ops infrastructure environment is a cornerstone of Kubestack's reliable GitOps automation. The ops environment serves the purpose to test and validate configuration before it is applied to the apps environment. Configuration drift risks rendering this protection ineffective.
Inheritance makes differences explicit. By default, everything is inherited. But if necessary, parts of the inherited configuration can be overwritten. Explicit differences do not prevent configuration drift. But they make it easier to spot.
By adopting inheritance Kubestack reduces the risk for configuration drift and increases automation reliability by avoiding potential errors and enforcing discipline across teams.
Implementation
Kubestack implements the inheritance model for both infrastructure configuration and cluster manifests. The implementation depends on the repository to define both apps and ops. Terraform to implement the inheritance based on the workspace. And the CI/CD pipeline to select the workspace and running Terraform.
- Repository:
- defines apps & ops configuration in
config.auto.tfvars
- defines apps & ops overlays under
manifests/overlays
- defines apps & ops configuration in
- Terraform:
- for apps workspace
- uses apps configuration
- uses apps overlay
- for ops workspace
- uses merged apps + ops configuration
- uses merged apps + ops overlay
- for apps workspace
- CI/CD Pipeline:
- selects workspace apps or ops
- runs
terraform plan
andterraform apply
Infrastructure configuration
The infrastructure configuration implements the inheritance model by having a map per cluster module. Both apps
and ops
are keys of the cluster_module
map as shown below.
cluster_module = {apps = {}ops = {}}
By default, the configuration is specified in the apps hash map. Any value that is additionally set in the ops hash map, will overwrite the value from the apps hash map. Overwriting values can be useful to reduce the size of the ops cluster by using fewer nodes or also smaller instance types.
While avoiding configuration differences that break the automation is important, it may also be undesirable to have ops be the same size as apps because ops does not run any workloads.
Example
The example below configures an apps cluster's min node count to 1 and max node count to 10. It also sets the cluster's node locations to three availability zones in GCP's europe-west1 region. This apps cluster will auto scale between 3 and 30 nodes because node counts are per location. The ops cluster overwrites the auto scaling and node location attributes. This ops cluster would not auto scale and would have 2 nodes because max node count is set to 1 and it has two node locations instead of three.
A configuration like this may be a good compromise. It keeps a similar cluster architecture, with multiple nodes and locations. But it also reduces costs for the ops cluster.
clusters = {gke_zero = {apps = {# [...]cluster_min_node_count = 1cluster_max_node_count = 10region = "europe-west1"cluster_node_locations = "europe-west1-b,europe-west1-c,europe-west1-d"}ops = {cluster_max_node_count = 1cluster_node_locations = "europe-west1-b,europe-west1-c"}}}
Cluster manifests
Kubestack uses Kustomize to maintain cluster manifests. Kustomize also uses inheritance. Its inheritance is implemented using bases and overlays. Overlays can inherit from one or more bases or even overlays. Bases inherit from nowhere.
Kubestack implements the inheritance for cluster manifests with a directory for all bases and two Kustomize overlays. By default, reference your bases in apps/kustomization.yaml
and ops/kustomization.yaml
will inherit everything from apps. If necessary, the ops overlay can be used to overwrite configuration inherited from apps. Follow the example below to try this for yourself.
The default manifest path can be overwritten using the cluster module's manifest_path
attribute.
The overlays
directory layout for a fresh Kubestack repository looks like this:
manifests/overlays/├── apps│ └── kustomization.yaml└── ops└── kustomization.yaml
To understand how the inheritance works, let's take a look at the two kustomization.yaml
files.
First, manifests/overlays/apps/kustomization.yaml
inherits from the Nginx ingress controller base.
# [...]resources:# Nginx ingress is included by default- ../../bases/nginx/default-ingress
Second, manifests/overlays/ops/kustomization.yaml
inherits everything from the apps overlay by referencing it under resources
. This also includes the Nginx ingress controller.
# [...]resources:- ../apps
Example
The included Nginx ingress controller is configured to have two replicas. Following the smaller ops cluster example above, let's decrease the replicas for ops to one.
Add the built-in
replicas
field toops/kustomization.yaml
:# [...]resources:- ../appsreplicas:- name: nginx-ingress-controllercount: 1To understand the differences, take a look at the
kustomize build
output of the two overlays:kustomize build manifests/overlays/apps
kustomize build manifests/overlays/ops
Compare how apps has
replicas: 2
, while ops hasreplicas: 1
.
Refer to the Kustomize docs for more supported fields.
For more information on cluster manifests:
- Check the repository layout docs for details on Kustomize bases and overlays,
- or learn more about working with cluster manifests.