GitOps Process

TL;DR:

  • Kubestack's GitOps process is designed to make infrastructure automation reliable.
  • Kubestack differentiates between infrastructure and application environments.
  • Any change follows the same four steps.
  • Git branches and tags determine the environment and pipeline steps.

GitOps prerequisites

Adopting GitOps requires reliable infrastructure automation. Kubestack combines an intuitive change management process with two dedicated environments, called ops and apps, to achieve this.

The change management process uses Git branches and tags to trigger the pipeline. The pipeline will either provide feedback, validate changes on the ops environment or change the apps environment. Configuration changes and Terraform plans can be reviewed before they are applied.

This approach enables teams to jointly maintain the configuration of cluster infrastructure and services in a reliable and automated way.

Infrastructure environments

The Kubestack framework uses Terraform workspaces to provide the independent ops and apps environments. These infrastructure environments should not be confused with application environments.

  • The purpose of the ops environment is to validate changes before they get applied to the apps environment.
  • The purpose of the apps environment is to provide a stable environment to run your applications.

Environments are ubiquitous both for infrastructure and applications. Kubestack explicitly avoids common application environment names for its infrastructure environments.

Keeping infrastructure and application environments independent is key for reliable automation. This separation also allows teams to work on infrastructure and applications at the same time. Calling the environments ops and apps aims to make their purpose clear and avoid misunderstandings.

Custom environments

By default, Kubestack uses the two environments ops and apps. But it is possible to use custom names and also more than two environments. This allows alternative cluster-environment architectures like for example ops, apps-stage and apps-prod for teams where staging and production are required to be on different clusters.

This requires modifications to the following areas:

  1. Adapt the cluster module attributes to your custom environments
  2. Align the Kustomize overlays to match these environments as well
  3. Create Terraform workspaces that match your environment names
  4. Modify the pipeline and triggers to promote changes to your custom environments

Here is an example repository, showing the changes required for custom environments. The rest of the documentation will focus on the default ops and apps architecture. Custom environments will require you to adapt instructions to your custom setup.

Making changes

Kubestack GitOps Flow

The Kubestack workflow is very similar to Github's Git-flow. Changing and applying configuration is a four-step process:

  1. Change

    Make changes to the configuration in a new branch. Commit the changed configuration. Validate your changes by pushing the new branch. The pipeline runs Terraform plan against the ops cluster.

  2. Review

    Request a peer review of your changes. Team members review the changes and the Terraform plan. If reviewers require changes, make additional commits in the branch.

  3. Merge

    If approved, merge your changes to master, to apply them against the ops environment. After applying to ops was successful, the pipeline runs Terraform plan against the apps environment.

  4. Set tag

    Review the previous apps environment plan and tag the merge commit to apply the same changes to the apps environment.

Now that we know how to make changes and trigger the pipeline, let's take a closer look at the triggers and pipeline steps.

Pipeline triggers and steps

Any commit pushed to a remote repository should trigger a pipeline run. The pipeline selects the correct workspace and runs the required steps. The branch or tag of the commit determines the correct workspace and steps.

Kubestack knows three types of triggers:

  1. Triggers from feature branches.

    Feature branch pipeline-runs provide Terraform plan for the ops workspace as feedback for reviewers. Teams should peer review the changes and the plan before merging the feature branch into the master branch.

    The pipeline runs the following Terraform commands for feature branches:

    1. terraform init
    2. terraform workspace select ops
    3. terraform plan
  2. Triggers from the master branch.

    Master branch pipeline-runs validate changes by applying them against the ops workspace. Terraform apply can fail even if the prior plan was successful. Applying changes to the dedicated ops environment reduces the risk of this happening when applying to the critical apps environment.

    The pipeline runs the following Terraform commands for the master branch:

    1. terraform init
    2. terraform workspace select ops
    3. terraform plan
    4. terraform apply
    5. terraform workspace select apps
    6. terraform plan
  3. Triggers from deploy-tags.

    Deploy-tag pipeline-runs apply the configuration against the apps workspace. Tags point to a specific commit in the Git repository. By using tags the exact same configuration is applied to the ops and apps workspace.

    The pipeline runs the following Terraform commands for deploy-tags:

    1. terraform init
    2. terraform workspace select apps
    3. terraform plan
    4. terraform apply