GitOps Process

TL;DR:

Kubestack's GitOps process is designed to make infrastructure automation reliable.
Kubestack differentiates between infrastructure and application environments.
Any change follows the same review, validate, and promote workflow.
Git branches and tags determine the environment and pipeline steps.
AI coding agents edit files locally, the pipeline owns applying changes to infrastructure.

GitOps prerequisites

Adopting GitOps requires reliable infrastructure automation. Kubestack combines an intuitive change management process with at least one internal and one external environment to achieve this.

The change management process uses Git branches and tags to trigger the pipeline. The pipeline will either provide feedback for peer-reviews, validate changes against the internal environment or promote the change to the external environment.

This enables teams to jointly maintain the configuration of Kubernetes infrastructure and Kubernetes services in a reliable and automated way.

Infrastructure environments

The Kubestack framework uses OpenTofu workspaces to provide independent infrastructure environments. By default, there are either two or three environments ops, apps or ops, apps and apps-prod depending on your infrastructure environment choice. But the number of environments and their names can be changed.

ops: validates changes against real cloud infrastructure before they get promoted to the mission critical environment
apps: provides the stable environment for mission critical applications (home to either non-prod and prod or only non-prod namespaces)
(optional) apps-prod: provides the stable environment for mission critical applications (home to prod namespaces only)

These infrastructure environments should not be confused with application environments. Kubestack explicitly avoids common application environment names like dev, staging or prod for its infrastructure environments.

Keeping infrastructure and application environments independent is key for reliable automation. Clear separation of concerns also allows teams to work on infrastructure and applications simultaneously.

Calling the environments ops, apps and apps-prod aims to make their purpose clear and avoid misunderstandings.

Making changes

Kubestack GitOps Flow

The Kubestack workflow is very similar to GitHub's Git-flow. Changing and applying configuration is a four-step process. Each Git push triggers a pipeline run. The pipeline will run OpenTofu, which will send the new committed state to both the cloud provider or Kubernetess API. Synchronizing the new desired state with the current state continuously is the responsibility of the active cloud and Kubernetes control planes.

Change

Make changes to the configuration in a new branch. Commit the changed configuration. Validate your changes by pushing the new branch. The pipeline runs tofu plan against the ops environment.
Review

Request a peer review of your changes. Team members review the changes and the OpenTofu plan. If reviewers require changes, make additional commits in the branch.
Validate

If approved, merge your changes to the main branch, to apply them against the ops environment. After applying to ops was successful, the pipeline runs tofu plan against the apps environment.
Promote

Review the previous apps environment plan and tag the merge commit to promote the same changes to the apps environment.

Agents write, the pipeline applies: When you work through an AI coding agent that has learned the Kubestack skill, the same workflow applies unchanged. Your agent only edits and validates files locally, it never applies changes to infrastructure. Every change the agent proposes still has to pass peer review and pipeline validation before the pipeline applies it. This separation of concerns is what makes non-deterministic LLM behavior safe: it can produce a bad proposal, but the proposal never reaches your running platform without a human reviewing the OpenTofu plan first.

Pipeline triggers and steps

Any commit pushed to a remote repository should trigger a pipeline run. The branch or tag of the commit that triggered the run determines the correct environment and steps. The pipeline selects the workspace for the environment and runs the required steps.

Kubestack knows three types of triggers:

Triggers from feature branches.

Feature branch pipeline-runs provide a tofu plan for the ops workspace as feedback for reviewers. Teams should peer review the changes and the plan before merging the feature branch into the main branch.

The pipeline runs the following tofu commands for feature branches:
1. tofu init
2. tofu workspace select ops
3. tofu plan
Triggers from the main branch.

Main branch pipeline-runs validate changes by applying them against the ops workspace. tofu apply can fail even if the prior plan was successful. Applying changes to the dedicated ops environment reduces the risk of this happening when applying to the critical apps environment.

The pipeline runs the following tofu commands for the main branch:
1. tofu init
2. tofu workspace select ops
3. tofu plan
4. tofu apply
5. tofu workspace select apps
6. tofu plan
Triggers from deploy-tags.

Deploy-tag pipeline-runs apply the configuration against the apps workspace. Tags point to a specific commit in the Git repository. By using tags the exact same configuration is promoted from the ops to the apps environment.

The pipeline runs the following tofu commands for deploy-tags:
1. tofu init
2. tofu workspace select apps
3. tofu plan
4. tofu apply