Provision Infrastructure
Overview
In this second part of the Kubestack tutorial you will:
- Setup authentication.
- Provision the remote state.
- Bootstrap your infrastructure.
- Set up DNS.
To make it easier to provide all prerequisites like the cloud provider command line utilities, Terraform, Kustomize and more we provide a container image and use it for bootstrapping now and also CI/CD later.
# Build the bootstrap containerdocker build -t kbst-infra-automation:bootstrap .# Exec into the bootstrap containerdocker run --rm -ti \-v `pwd`:/infra \kbst-infra-automation:bootstrap
Setup authentication
Use your personal user to create the automation user.
Login with your personal user.
# run aws configure and follow the instructions# make sure to set a default regionaws configureCreate the automation user and attach the AWS managed
AdministratorAccess
policy to it.# Create the useraws iam create-user --user-name kubestack-automation# Attach the managed admin policy to the useraws iam attach-user-policy \--policy-arn arn:aws:iam::aws:policy/AdministratorAccess \--user-name kubestack-automationReplace your personal credentials.
# Create keys for the userACCESS_KEY_JSON=$(aws iam create-access-key --user-name kubestack-automation)# Set the access keyaws configure set aws_access_key_id $(echo $ACCESS_KEY_JSON | jq -r .AccessKey.AccessKeyId)# Set the secret access keyaws configure set aws_secret_access_key $(echo $ACCESS_KEY_JSON | jq -r .AccessKey.SecretAccessKey)
Use your personal account to create the service principal.
Login with your personal account.
# run az login and follow the instructionsaz loginSelect the subscription to use.
# List your subscriptionsaz account list --query "[].{name:name,subscription_id:id}" --output table# Select your subscription by IDread -p "Subscription ID: " SUBSCRIPTION_IDCreate a service principal and configure role and permissions.
# Create service principal and assign the contributor roleAZ_SP_JSON=$(az ad sp create-for-rbac \--name="kubestack-automation" \--role="Contributor" \--scopes="/subscriptions/${SUBSCRIPTION_ID}" \--output="json")# Also add and grant active directory permissionsaz ad app permission add \--id $(echo $AZ_SP_JSON | jq -r .appId) \--api 00000002-0000-0000-c000-000000000000 \--api-permissions 1cda74f2-2616-4834-b122-5cb1b07f8a59=Roleaz ad app permission grant \--id $(echo $AZ_SP_JSON | jq -r .appId) \--api 00000002-0000-0000-c000-000000000000 \--expires neveraz ad app permission admin-consent \--id $(echo $AZ_SP_JSON | jq -r .appId)Authenticate as the service principal.
# For the CLIaz login --service-principal \--username $(echo $AZ_SP_JSON | jq -r .appId) \--password $(echo $AZ_SP_JSON | jq -r .password) \--tenant $(echo $AZ_SP_JSON | jq -r .tenant)# For Terraformexport ARM_CLIENT_ID=$(echo $AZ_SP_JSON | jq -r .appId)export ARM_CLIENT_SECRET=$(echo $AZ_SP_JSON | jq -r .password)export ARM_SUBSCRIPTION_ID="${SUBSCRIPTION_ID}"export ARM_TENANT_ID=$(echo $AZ_SP_JSON | jq -r .tenant)Get the storage account access key
# Select the resource groupaz group list --query "[].{name:name}" --output tableread -p "Resource group: " RESOURCE_GROUP# Select the storage account nameaz storage account list --query "[].{name:name}" --output tableread -p "Storage account name: " STORAGE_ACCOUNTexport ARM_ACCESS_KEY=$(az storage account keys list --account-name ${STORAGE_ACCOUNT} --resource-group ${RESOURCE_GROUP} | jq -r '.[0].value')Safe the ARM_* environment variables for CI/CD
env | grep ARM_ > ~/.azure/KBST_AUTH_AZ
Use your personal account to create the service account.
Login with your personal account.
# run gcloud init and follow the instructionsgcloud initCreate the service account, assign a role and create keys.
# Use the current projectPROJECT=$(gcloud config get-value project)# Create the service accountgcloud iam service-accounts create kubestack-automation \--description "SA used for Kubestack Github Actions" \--display-name "kubestack-automation"# Assign the owner role to the service accountgcloud projects add-iam-policy-binding ${PROJECT} \--member serviceAccount:kubestack-automation@${PROJECT}.iam.gserviceaccount.com \--role roles/owner# Create service account keysgcloud iam service-accounts keys create \~/.config/gcloud/application_default_credentials.json \--iam-account kubestack-automation@${PROJECT}.iam.gserviceaccount.comActivate the service account and stop using your personal account.
gcloud auth activate-service-account --key-file ~/.config/gcloud/application_default_credentials.json
Provision the remote state
Terraform requires remote state to run in CI/CD. You are free to use any supported remote state storage. Below instructions use your cloud provider's object storage.
Object storage bucket names have to be globally unique. The instructions below append the short git hash of our initial commit to achieve this.
Run the following commands to create a bucket in AWS S3 and change the state.tf
file to configure Terraform to use the created bucket for remote state.
# Create bucket and configure remote stateBUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`REGION=`aws configure get region`aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION --create-bucket-configuration LocationConstraint=$REGION --acl privateaws s3api put-bucket-versioning --bucket $BUCKET_NAME --region $REGION --versioning-configuration Status=Enabledaws s3api put-bucket-encryption --bucket $BUCKET_NAME --region $REGION --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'cat > state.tf <<EOFterraform {backend "s3" {bucket = "${BUCKET_NAME}"region = "${REGION}"key = "tfstate"}}EOF
Run the following commands to create a storage container in Azure blob storage and change the state.tf
file to configure Terraform to use the created storage container for remote state.
# Create storage container and configure remote stateSTORAGE_CONTAINER=terraform-state-kubestack-`git rev-parse --short HEAD`az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNTcat > state.tf <<EOFterraform {backend "azurerm" {storage_account_name = "${STORAGE_ACCOUNT}"container_name = "${STORAGE_CONTAINER}"key = "tfstate"}}EOF
Run the following commands to create a bucket in Google cloud storage and change the state.tf
file to configure Terraform to use the created bucket for remote state.
# Set the location of your multi-regional bucket# valid values are `asia`, `eu` or `us`read -p "Bucket location: " LOCATION
# Create bucket and configure remote stateBUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`gsutil mb -l $LOCATION gs://$BUCKET_NAMEgsutil versioning set on gs://$BUCKET_NAMEcat > state.tf <<EOFterraform {backend "gcs" {bucket = "${BUCKET_NAME}"}}EOF
Bootstrap your infrastructure
With the remote state setup, initialize Terraform to use the remote state and create the two workspaces ops
and apps
.
Terraform init
# Initialize Terraformterraform initTerraform workspace for apps and ops
# Create apps and ops workspacesterraform workspace new appsterraform workspace new opsTerraform's workspaces control which environment the configuration is applied to. The ops workspace is currently selected because it was created last.
Terraform apply for ops
# To bootstrap the ops environment run applyterraform apply --auto-approveTerraform apply for apps
# To bootstrap the apps environment select the apps workspace and run applyterraform workspace select appsterraform apply --auto-approve
Set up DNS
Kubestack's per environment DNS zones ensure full test ability. The name_prefix
and base_domain
variables set in config.auto.tfvars
define the zone names.
To make DNS names in those zones resolvable, follow these steps:
- Get the ops zones' name and nameservers.
- Get the apps zones' name and nameservers.
- Set NS records in the
base_domain
for each zone.
Get ops zone name
# get ops zone name and name_serversterraform workspace select opsterraform state show module.eks_zero.module.cluster.aws_route53_zone.current[0]Get apps zone name
# get apps zone name and name_serversterraform workspace select appsterraform state show module.eks_zero.module.cluster.aws_route53_zone.current[0]Set NS records
Set NS in the
base_domain
for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.
Get ops zone name
# get ops zone name and name_serversterraform workspace select opsterraform state show module.aks_zero.module.cluster.azurerm_dns_zone.current[0]Get apps zone name
# get zone name and name_serversterraform workspace select appsterraform state show module.aks_zero.module.cluster.azurerm_dns_zone.current[0]Set NS records
Set NS in the
base_domain
for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.
Get ops zone name
# get ops zone dns_name and name_serversterraform workspace select opsterraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current[0]Get apps zone name
# get apps zone dns_name and name_serversterraform workspace select appsterraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current[0]Set NS records
Set NS in the
base_domain
for both the ops and apps DNS zone. How to set these NS record depends on your DNS provider. Compare the example below to understand the desired outcome.
Example
NS queries should return the cluster's name servers for your cluster domain similar to how they do for the DNS zones below.
# ops exampledig NS kbst-ops-us-east1.gcp.infra.kubestack.com# apps exampledig NS kbst-apps-us-east1.gcp.infra.kubestack.com
[...];; ANSWER SECTION:kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a1.googledomains.com.kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a2.googledomains.com.kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a3.googledomains.com.kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a4.googledomains.com.
Commit changes
# Exit the bootstrap containerexit
# Commit the configurationgit add .git commit -m "Add cluster configuration"
Recap
You now have the GitOps repository in place. You provisioned prerequisites like authentication credentials and the remote state. Finally, you also bootstrapped both the ops and apps infrastructure environments and configured DNS for each.
Continue with the second part of the tutorial to add the automation.