Repository Layout

TL;DR:

  • Kubestack follows a convention over configuration approach.
  • The layout follows common recommendations for Terraform configurations.
  • The hierarchy of Kustomize overlays and the Terraform workspace determine the cluster services configuration.

Repository root

Kubestack follows a convention over configuration approach. The repository layout is primarily influenced by Terraform conventions. So teams familiar with Terraform should feel right at home. On top of the Terraform configuration, the repository also includes a Dockerfile for an image that provides all requirements. Last but not least, it holds the configuration for cluster services in Kustomize bases and overlays.

A freshly bootstrapped repository looks like this:

.
├── ci-cd
│   ├── Dockerfile
│   └── nss-wrapper
├── manifests
│   ├── bases
│   └── overlays
├── clusters.tf
├── config.auto.tfvars
├── providers.tf
├── state.tf
├── variables.tf
└── versions.tf

Let's take a look at the directories and files in the repository root:

  • ci-cd/

    The ci-cd directory includes the Dockerfile. The Dockerfile builds an image to use for bootstrapping, pipeline runs and disaster recovery. The image has all the required dependencies installed. Most notably the CLIs of all supported cloud providers, as well as Terraform itself, the Kubestack Kustomize provider and a couple of other useful tools like Kubectl and Kustomize.

  • manifests

    The manifests directory holds the Kustomize bases and overlays for the cluster services. Kustomize bases and overlays, and the inheritance model of the cluster services manifests are documented in the manifests directory section below.

    Cluster services are fully integrated into the Terraform lifecycle. This is further documented as part of the cluster services documentation.

  • clusters.tf

    This file defines the desired cluster pairs by loading the cluster module and passing the respective configuration. Modules are self-contained packages of reusable Terraform configurations.

  • config.auto.tfvars and variables.tf

    These two files hold the main configuration for Kubestack cluster pairs. The configuration format is documented under cluster pair configuration. Input variables are used to set values specific to a cluster and its workspace. Terraform requires variables to be defined. Variables are defined in variables.tf. Kubestack makes use of a Terraform feature to automatically load values for input variables from files named *.auto.tfvars.

  • providers.tf: configures Terraform providers, if necessary

    Providers integrate cloud provider APIs into Terraform. In providers.tf we configure the Terraform provider for your chosen cloud. The majority of provider configuration, most importantly API credentials, is read from the environment automatically. Some cloud providers require per provider configuration. Refer to the documentation about adding a cluster pair for details.

  • state.tf

    This file holds the configuration for the Terraform remote state. Terraform stores state about your infrastructure. It uses this state to map the real world resources to your configuration, keep track of metadata and to improve performance.

    Because we are using Terraform in a team and CI/CD scenario it is important to store state centrally and ensure only one change is applied at any given time. Both of these are achieved by using remote state with locking.

    Backends are not Kubestack specific. You can use any backend Terraform supports. Please refer to the Terraform documentation for backends.

  • versions.tf

    The versions file defines the required Terraform version.

Manifests directory

The manifests/ directory has two subdirectories named bases/ and overlays/.

manifests/
├── bases
│   └── nginx
└── overlays
├── apps
└── ops

Kubestack uses Kustomize to manage Kubernetes cluster manifests. Kustomize is based on inheritance instead of templating. It allows customizing a set of Kubernetes resources for different environments. How this works for apps and ops is documented as part of the inheritance model.

Bases and overlays are directories with a set of Kubernetes manifests in YAML format and additionally a Kustomization.yaml. Overlays can inherit from one or more bases or even overlays. Bases inherit from nowhere.

Bases are either your own bespoke bases or bases from the Kubestack catalog.

Overlays can overwrite any attribute of the inherited configuration. For more information on Kustomize, refer to its community website or documentation on Github.