Framework

Provision Infrastructure

Overview

In this first part of the cloud tutorial you will:

Prepare your work environment.
Setup authentication.
Configure the infrastructure.
Provision the remote state.
Bootstrap your infrastructure.
Set up DNS.

Prepare your work environment

Bootstrap your infrastructure automation. Set-up your infrastructure repository, prepare your cloud environment and get your first cluster pair up and running.

You can easily add more cluster pairs in different regions, on different cloud providers or both later. But for now pick one cloud and let's get started.

# Download and unpack the EKS starter. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/kubestack-starter-eks-v0.9.0-beta.0.zip
unzip kubestack-starter-eks-v0.9.0-beta.0.zip
rm kubestack-starter-eks-v0.9.0-beta.0.zip
# Change into the EKS starter directory and initialize your Git repo.
cd kubestack-starter-eks
git init .
git add .
git commit -m "Initialized from EKS starter"

# Download and unpack the AKS starter. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/kubestack-starter-aks-v0.9.0-beta.0.zip
unzip kubestack-starter-aks-v0.9.0-beta.0.zip
rm kubestack-starter-aks-v0.9.0-beta.0.zip
# Change into the AKS starter directory and initialize your Git repo.
cd kubestack-starter-aks
git init .
git add .
git commit -m "Initialized from AKS starter"

# Download and unpack the GKE starter. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/kubestack-starter-gke-v0.9.0-beta.0.zip
unzip kubestack-starter-gke-v0.9.0-beta.0.zip
rm kubestack-starter-gke-v0.9.0-beta.0.zip
# Change into the GKE starter directory and initialize your Git repo.
cd kubestack-starter-gke
git init .
git add .
git commit -m "Initialized from GKE starter"

To make it easier to provide all prerequisites like the cloud provider command line utilities, Terraform, Kustomize and more we provide a container image and use it for bootstrapping now and also CI/CD later.

# Build the bootstrap container
docker build -t kbst-infra-automation:bootstrap .
# Exec into the bootstrap container
docker run --rm -ti \
    -v `pwd`:/infra \
    kbst-infra-automation:bootstrap

Setup authentication

Use your personal user to create the automation user.

# run aws configure and follow the instructions
# make sure to set a default region
aws configure

Create the automation user and attach the AWS managed AdministratorAccess policy to it.

# Create the user
aws iam create-user --user-name kubestack-automation
# Attach the managed admin policy to the user
aws iam attach-user-policy \
   --policy-arn arn:aws:iam::aws:policy/AdministratorAccess \
   --user-name kubestack-automation

Replace your personal credentials.

# Create keys for the user
ACCESS_KEY_JSON=$(aws iam create-access-key --user-name kubestack-automation)
# Set the access key
aws configure set aws_access_key_id $(echo $ACCESS_KEY_JSON | jq -r .AccessKey.AccessKeyId)
# Set the secret access key
aws configure set aws_secret_access_key $(echo $ACCESS_KEY_JSON | jq -r .AccessKey.SecretAccessKey)

Use your personal account to create the service principal.

# run az login and follow the instructions
az login

Select the subscription to use.

# List your subscriptions
az account list --query "[].{name:name,subscription_id:id}" --output table
# Select your subscription by ID
read -p "Subscription ID: " SUBSCRIPTION_ID

Create a service principal and configure role and permissions.

# Create service principal and assign the contributor role
AZ_SP_JSON=$(az ad sp create-for-rbac \
   --name="kubestack-automation" \
   --role="Contributor" \
   --scopes="/subscriptions/${SUBSCRIPTION_ID}" \
   --output="json")
# Also add and grant active directory permissions
az ad app permission add \
  --id $(echo $AZ_SP_JSON | jq -r .appId) \
  --api 00000002-0000-0000-c000-000000000000 \
  --api-permissions 1cda74f2-2616-4834-b122-5cb1b07f8a59=Role
az ad app permission grant \
  --id $(echo $AZ_SP_JSON | jq -r .appId) \
  --api 00000002-0000-0000-c000-000000000000 \
  --expires never
az ad app permission admin-consent \
  --id $(echo $AZ_SP_JSON | jq -r .appId)

Authenticate as the service principal.

# For the CLI
az login --service-principal \
         --username $(echo $AZ_SP_JSON | jq -r .appId) \
         --password $(echo $AZ_SP_JSON | jq -r .password) \
         --tenant $(echo $AZ_SP_JSON | jq -r .tenant)

# For Terraform
export ARM_CLIENT_ID=$(echo $AZ_SP_JSON | jq -r .appId)
export ARM_CLIENT_SECRET=$(echo $AZ_SP_JSON | jq -r .password)
export ARM_SUBSCRIPTION_ID="${SUBSCRIPTION_ID}"
export ARM_TENANT_ID=$(echo $AZ_SP_JSON | jq -r .tenant)

Get the storage account access key

# Select the resource group
az group list --query "[].{name:name}" --output table
read -p "Resource group: " RESOURCE_GROUP

# Select the storage account name
az storage account list --query "[].{name:name}" --output table
read -p "Storage account name: " STORAGE_ACCOUNT

export ARM_ACCESS_KEY=$(az storage account keys list --account-name ${STORAGE_ACCOUNT} --resource-group ${RESOURCE_GROUP} | jq -r '.[0].value')

Safe the ARM_* environment variables for CI/CD
```
env | grep ARM_ > ~/.azure/KBST_AUTH_AZ
```

Use your personal account to create the service account.

# run gcloud init and follow the instructions
gcloud init

Create the service account, assign a role and create keys.

# Use the current project
PROJECT=$(gcloud config get-value project)
# Create the service account
gcloud iam service-accounts create kubestack-automation \
  --description "SA used for Kubestack Github Actions" \
  --display-name "kubestack-automation"
# Assign the owner role to the service account
gcloud projects add-iam-policy-binding ${PROJECT} \
  --member serviceAccount:kubestack-automation@${PROJECT}.iam.gserviceaccount.com \
  --role roles/owner
# Create service account keys
gcloud iam service-accounts keys create \
  ~/.config/gcloud/application_default_credentials.json \
  --iam-account kubestack-automation@${PROJECT}.iam.gserviceaccount.com

Activate the service account and stop using your personal account.

gcloud auth activate-service-account --key-file ~/.config/gcloud/application_default_credentials.json

Configure the infrastructure

Follow the steps below to configure your cluster pair.

config.auto.tfvars
Set the required variables for apps. Optionally, overwrite settings for ops. By default, the configuration from apps is inherited.
- name_prefix (required)
- base_domain (required)
- cluster_availability_zones (required)
providers.tf
Set the region attribute to your desired region in the file providers.tf.

Edit the file config.auto.tfvars and set the required variables for apps. Optionally, overwrite settings for ops. By default, the configuration from apps is inherited.

name_prefix (required)
base_domain (required)
resource_group (required)

Edit the file config.auto.tfvars and set the required variables for apps. Optionally, overwrite settings for ops. By default, the configuration from apps is inherited.

project_id (required)
name_prefix (required)
base_domain (required)
region (required)
cluster_node_locations (required)

Provision the remote state

Terraform requires remote state to run in CI/CD. You are free to use any supported remote state storage. Below instructions use your cloud provider's object storage.

Object storage bucket names have to be globally unique. The instructions below append the short git hash of our initial commit to achieve this.

Run the following commands to create a bucket in AWS S3 and change the state.tf file to configure Terraform to use the created bucket for remote state.

# Create bucket and configure remote state
BUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`
REGION=`aws configure get region`
aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION --create-bucket-configuration LocationConstraint=$REGION --acl private
aws s3api put-bucket-versioning --bucket $BUCKET_NAME --region $REGION --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket $BUCKET_NAME --region $REGION --server-side-encryption-configuration '{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }
  ]
}'
cat > state.tf <<EOF
terraform {
  backend "s3" {
    bucket = "${BUCKET_NAME}"
    region = "${REGION}"
    key    = "tfstate"
  }
}
EOF

Run the following commands to create a storage container in Azure blob storage and change the state.tf file to configure Terraform to use the created storage container for remote state.

# Create storage container and configure remote state
STORAGE_CONTAINER=terraform-state-kubestack-`git rev-parse --short HEAD`
az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT
cat > state.tf <<EOF
terraform {
  backend "azurerm" {
    storage_account_name = "${STORAGE_ACCOUNT}"
    container_name       = "${STORAGE_CONTAINER}"
    key                  = "tfstate"
  }
}
EOF

Run the following commands to create a bucket in Google cloud storage and change the state.tf file to configure Terraform to use the created bucket for remote state.

# Set the location of your multi-regional bucket
# valid values are `asia`, `eu` or `us`
read -p "Bucket location: " LOCATION

# Create bucket and configure remote state
BUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`
gsutil mb -l $LOCATION gs://$BUCKET_NAME
gsutil versioning set on gs://$BUCKET_NAME
cat > state.tf <<EOF
terraform {
  backend "gcs" {
    bucket = "${BUCKET_NAME}"
  }
}
EOF

Bootstrap your infrastructure

With the remote state setup, initialize Terraform to use the remote state and create the two workspaces ops and apps.

Terraform init
```
# Initialize Terraform
terraform init
```
Terraform workspace for apps and ops
```
# Create apps and ops workspaces
terraform workspace new apps
terraform workspace new ops
```
Terraform's workspaces control which environment the configuration is applied to. The ops workspace is currently selected because it was created last.

Terraform apply for ops

# To bootstrap the ops environment run apply
terraform apply --auto-approve

Terraform apply for apps

# To bootstrap the apps environment select the apps workspace and run apply
terraform workspace select apps
terraform apply --auto-approve

Set up DNS

Kubestack's per environment DNS zones ensure full test ability. The name_prefix and base_domain variables set in config.auto.tfvars define the zone names.

To make DNS names in those zones resolvable, follow these steps:

Get the ops zones' name and nameservers.
Get the apps zones' name and nameservers.
Set NS records in the base_domain for each zone.

Get ops zone name

# get ops zone name and name_servers
terraform workspace select ops
terraform state show module.eks_zero.module.cluster.aws_route53_zone.current

Get apps zone name

# get apps zone name and name_servers
terraform workspace select apps
terraform state show module.eks_zero.module.cluster.aws_route53_zone.current

Set NS records
Set NS in the base_domain for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.

Get ops zone name

# get ops zone name and name_servers
terraform workspace select ops
terraform state show module.aks_zero.module.cluster.azurerm_dns_zone.current

Get apps zone name

# get zone name and name_servers
terraform workspace select apps
terraform state show module.aks_zero.module.cluster.azurerm_dns_zone.current

Set NS records
Set NS in the base_domain for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.

Get ops zone name

# get ops zone dns_name and name_servers
terraform workspace select ops
terraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current

Get apps zone name

# get apps zone dns_name and name_servers
terraform workspace select apps
terraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current

Set NS records
Set NS in the base_domain for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.

Example

NS queries should return the cluster's name servers for your cluster domain similar to how they do for the DNS zones below.

# ops example
dig NS kbst-ops-us-east1.gcp.infra.kubestack.com
# apps example
dig NS kbst-apps-us-east1.gcp.infra.kubestack.com

[...]
;; ANSWER SECTION:
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS	ns-cloud-a1.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS	ns-cloud-a2.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS	ns-cloud-a3.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS	ns-cloud-a4.googledomains.com.

Commit changes

# Exit the bootstrap container
exit

# Commit the configuration
git add .
git commit -m "Add cluster configuration"

Recap

You now have the GitOps repository in place. You provisioned prerequisites like authentication credentials and the remote state. Finally, you also bootstrapped both the ops and apps infrastructure environments and configured DNS for each.

Continue with the second part of the tutorial to add the automation.