Build a Local Lab

Motivation

All great frameworks have easy to use local development environments. Similarly, the Kubestack GitOps framework provides a local development environment for infrastructure automation. This tutorial will help you setup a complete GitOps lab on your local machine. Including Kubernetes nodes running as Docker containers using Kubernetes in Docker (KinD) and optionally even a local pipeline triggered using Git hooks.

Having a local lab can be useful to:

  • Learn more about GitOps.
  • Try the Kubestack framework.
  • Use the lab as a local development environment.

Following this tutorial you will:

  1. Bootstrap your local GitOps repository.
  2. Configure your lab infrastructure.
  3. Provision the local clusters.
  4. Optional: Test the GitOps flow.

Bootstrap your local repository

# Download and unpack the KinD starter. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/kubestack-starter-kind-v0.9.0-beta.0.zip
unzip kubestack-starter-kind-v0.9.0-beta.0.zip
rm kubestack-starter-kind-v0.9.0-beta.0.zip
# Change into the KinD starter directory and initialize your Git repo.
cd kubestack-starter-kind
git init .
git add .
git commit -m "Initialized from KinD starter"

To make it easier to provide all prerequisites like Terraform, Kind, Kustomize and more we provide a container image and use it for bootstrapping now and also CI/CD later.

# Build the bootstrap container
docker build -t kubestack-starter-kind .
# Exec into the bootstrap container
docker run --rm -ti \
-v `pwd`:/infra \
-v /var/run/docker.sock:/var/run/docker.sock \
kubestack-starter-kind

Configure your lab infrastructure

The KinD starter comes pre-configured. But you can change the defaults by editing the file config.auto.tfvars and change the setting for apps. Optionally, overwrite settings for ops. By default, the configuration from apps is inherited.

  • name_prefix (required)

    Defaults to name_prefix = "kind".

  • base_domain (required)

    Defaults to base_domain = "infra.127.0.0.1.xip.io".

  • extra_nodes (optional)

    By default, KinD starter clusters come with a single control-plane node. You can add additional control-plane and/or worker nodes using the extra_nodes configuration attribute.

    Extra nodes are in addition to the first control-plane node. For example, to create a cluster with 3 control-plan and 3 worker nodes, specify 2 extra control-plane and 3 extra worker nodes like this: extra_nodes = "control-plane,control-plane,worker,worker,worker".

Provision the local clusters

  1. Terraform init

    # Initialize Terraform
    terraform init
  2. Terraform workspace for apps and ops

    # Create apps and ops workspaces
    terraform workspace new apps
    terraform workspace new ops

    Terraform's workspaces control which environment the configuration is applied to. The ops workspace is currently selected because it was created last.

  3. Terraform apply for ops

    # To bootstrap the ops environment run apply
    terraform apply --auto-approve
  4. Terraform apply for apps

    # To bootstrap the apps environment select the apps workspace and run apply
    terraform workspace select apps
    terraform apply --auto-approve

Commit changes

# Exit the bootstrap container
exit
# Commit the configuration
git add .
git commit -m "Add cluster configuration"

At this point you have two local Kubernetes clusters. You can see the nodes by running:

# See the Kubernetes cluster nodes running as Docker containers
docker ps

With the local clusters provisioned, you can switch between the ops and apps infrastructure environment by selecting the terraform workspace. terraform workspace select ops switches to the ops workspace and ensures all following terraform commands are run against the ops environment. Likewise, terraform workspace select apps does the same for the apps environment.

You can now either run terraform commands manually to try your changes locally. Or you can continue and also set up a local GitOps pipeline.

Optional: Try the GitOps flow

The GitOps flow defines the workspace to select and terraform commands to run based on how it was triggered. For the local lab we will emulate the GitOps flow using Git hooks. Lets set this up.

  1. Create a local Git remote

    git init --bare .git/localremote
    git remote add origin .git/localremote
    git push -u origin master
  2. Create a new branch

    git checkout -b localpipeline
  3. Create the pipeline

    cat > kbst-local-pipeline <<'EOF'
    #!/usr/bin/env bash
    set -e
    workspace_path=$1
    state_path=$2
    ref_name=$3
    docker_volumes="-v ${workspace_path}:/infra -v ${state_path}:/infra/terraform.tfstate.d -v /var/run/docker.sock:/var/run/docker.sock"
    docker_env="-e TF_IN_AUTOMATION=1"
    docker_image="kubestack-starter-kind"
    #
    #
    # Build image
    docker build -t $docker_image .
    #
    #
    # Terraform init
    docker run \
    --rm \
    $docker_volumes \
    $docker_env \
    $docker_image \
    terraform init
    #
    #
    # Terraform workspace select
    case "$ref_name" in
    refs/tags/apps-deploy-*)
    terraform_workspace="apps"
    ;;
    *)
    terraform_workspace="ops"
    ;;
    esac
    docker run \
    --rm \
    $docker_volumes \
    $docker_env \
    $docker_image \
    terraform workspace select $terraform_workspace
    #
    #
    # Terraform apply
    case "$ref_name" in
    refs/heads/master)
    terraform_args="apply --auto-approve"
    ;;
    refs/tags/apps-deploy-*)
    terraform_args="apply --auto-approve"
    ;;
    *)
    terraform_args="plan"
    ;;
    esac
    docker run \
    --rm \
    $docker_volumes \
    $docker_env \
    $docker_image \
    terraform $terraform_args
    EOF
    chmod +x kbst-local-pipeline
  4. Create the hook in the remote to fire the pipeline

    cat > .git/localremote/hooks/post-update <<'EOF'
    #!/usr/bin/env bash
    set -e
    ref_name=$1
    if [ ! -d "../workspace" ]; then
    git clone -l . ../workspace
    fi
    cd ../workspace
    # we're running as a hook of localremote
    # and localremote is a bare repository
    # we have to fix GIT_DIR to have workspace
    # correctly have a working dir
    export GIT_DIR=.git
    git log -1
    git fetch origin --tags --prune
    ref_origin="${ref_name/'refs/heads'/'remotes/origin'}"
    hash=$(git rev-parse --verify ${ref_origin})
    git reset --hard $hash
    workspace_path=$(realpath .)
    state_path=$(realpath ../../terraform.tfstate.d)
    exec ./kbst-local-pipeline $workspace_path $state_path $ref_name
    EOF
    chmod +x .git/localremote/hooks/post-update
  5. Commit and push the pipeline

    If we now commit our changes and push them to our local remote, our newly created pipeline will be triggered for the first time. Because it's triggered from a branch it will run against the ops environment but because the branch is not master, it will only run terraform plan and not terraform apply.

    git add kbst-local-pipeline
    git commit -m "Add local pipeline"
    git push origin localpipeline
  6. Merge into master branch

    Next, we merge the localpipeline branch into master and push this merge to trigger a pipeline run for the master branch. This will apply the changes to the ops environment.

    git checkout master
    git merge localpipeline
    git push origin master
  7. Set a apps-deploy tag

    Finally, to apply the changes we just validated against the ops environment also against the apps environment set and push an apps-deploy tag. Triggered from a tag, the pipeline selects the apps environment and runs terraform plan.

    git tag apps-deploy-0
    git push origin apps-deploy-0

Recap

To recap, you have created a local repository holding the configuration for both an ops and an apps environment. Each has a local Kubernetes cluster powered by KinD (Kubernetes in Docker) and you can select the Terraform workspace to control which one of the environments changes are applied to.

If you also followed the optional GitOps section, you even have a local GitOps pipeline and can simulate the entire GitOps flow from creating a feature branch and getting feedback on the changes to validating the changes against ops and finally applying them against apps.

You can keep the local lab to use it as a development environment. Or optionally, tear it down using terraform destroy

  1. Optional: Destroy the local lab

    # Exec into the bootstrap container again
    docker run --rm -ti \
    -v `pwd`:/infra \
    -v /var/run/docker.sock:/var/run/docker.sock \
    kubestack-starter-kind
    # Destroy the apps environment
    terraform workspace select apps
    terraform destroy --auto-approve
    # Destroy the ops environment
    terraform workspace select ops
    terraform destroy --auto-approve
    # Exit the bootstrap container
    exit
  2. Optional: Cleanup localhost

    # to cleanup the repository, simply delete it
    cd ..
    rm -rf kubestack-starter-kind/
    # and cleanup the unused Docker images
    docker system prune

Next

If you're ready, lets continue the tutorial and apply what we learned to start automating real cloud infrastructure.