Set up Automation

Overview

To set up the automation we need to:

  1. Push your repository.
  2. Set up pipeline credentials.
  3. Add the pipeline file.
  4. Follow the GitOps process.

Github is used for the step-by-step instructions. But you can use other Git hosting and CI/CD providers too. The required steps tend to be similar.

Example pipelines for other CI/CD systems are available on Github.

Push your repository

  1. Create the remote repository

    Create a new repository and give it a descriptive name. For example infrastructure-automation.

    Github create new repository

  2. Push the repository

    Follow option two from Github's instructions to push your existing repository.

    Github push an existing repository

Set up pipeline credentials

  1. Encode the AWS credentials file using base64.

    cat .user/.aws/credentials | base64 -w 0 && echo
  2. Find secrets under your repository settings and add a secret named KBST_AUTH_AWS with the base64 encoded credentials from the previous step as the value.

    Github add action secret

  1. Encode the service principal credentials using base64

    cat .user/.azure/KBST_AUTH_AZ | base64 -w 0 && echo
  2. Find secrets under your repository settings and add a secret named KBST_AUTH_AZ with the base64 encoded credentials from the previous step as the value.

    Github add action secret

  1. Encode the service account key using base64

    cat .user/.config/gcloud/application_default_credentials.json | base64 -w 0 && echo
  2. Find secrets under your repository settings and add a secret named KBST_AUTH_GCLOUD with the base64 encoded credentials from the previous step as the value.

    Github add action secret

Add the pipeline file

  1. First create a new branch to work in.

    git checkout -b ghactions
  2. Add the pipeline as .github/workflows/main.yaml.

    mkdir -p .github/workflows
    cat > .github/workflows/main.yaml <<'EOF'
    name: deploy
    on:
    push:
    branches:
    - "*" # run for branches
    tags:
    - "*" # run for tags
    jobs:
    deploy:
    runs-on: ubuntu-latest
    env:
    TF_IN_AUTOMATION: true
    KBST_DOCKER_ARGS: --rm -v ${{ github.workspace }}:/infra
    KBST_DOCKER_IMAGE: kbst:${{ github.sha }}
    steps:
    - uses: actions/checkout@v1
    #
    #
    # Build image
    - name: Build image
    env:
    DOCKER_BUILDKIT: 1
    run: docker build -t $KBST_DOCKER_IMAGE .
    #
    #
    # Authenticate cloud providers
    - name: Authenticate providers
    env:
    KBST_AUTH_AWS: ${{ secrets.KBST_AUTH_AWS }}
    KBST_AUTH_AZ: ${{ secrets.KBST_AUTH_AZ }}
    KBST_AUTH_GCLOUD: ${{ secrets.KBST_AUTH_GCLOUD }}
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    -e KBST_AUTH_AWS \
    -e KBST_AUTH_AZ \
    -e KBST_AUTH_GCLOUD \
    $KBST_DOCKER_IMAGE
    #
    #
    # Terraform init
    - name: Terraform init
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform init
    #
    #
    # Select ops or apps workspace
    - name: Select ops workspace
    if: startsWith(github.ref, 'refs/tags/apps-deploy-') == false
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform workspace select ops
    - name: Select apps workspace
    if: startsWith(github.ref, 'refs/tags/apps-deploy-')
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform workspace select apps
    #
    #
    # Terraform plan against current workspace
    - name: Terraform plan
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform plan --out=tfplan --input=false
    #
    #
    # Terraform apply against current workspace
    # for master branch or apps-deploy tag
    - name: Terraform apply
    if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/apps-deploy-')
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform apply --input=false tfplan
    #
    #
    # Terraform plan against apps workspace
    # after applying to ops,
    # run plan against apps
    - name: Terraform plan
    if: github.ref == 'refs/heads/master'
    run: |
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform workspace select apps
    docker run \
    $KBST_DOCKER_ARGS \
    $KBST_DOCKER_IMAGE \
    terraform plan --out=tfplan --input=false
    EOF

Follow the GitOps process

  1. Add, commit and push the pipeline.

    git add .
    git commit -m "Add Github Actions pipeline"
    git push origin ghactions
  2. Open a pull request.

    Github open pull request

  3. Check the pipeline run.

    The pipeline run for the ghactions branch does not apply changes. It only provides the output of terraform plan against the ops workspace to determine what changes will be applied once merged.

    Github feature branch pipeline run

    Since we already bootstrapped the clusters, the pipeline at this point has no planned changes.

  4. Merge the pull request to apply changes to ops.

    Merge the pull request into master. The pipeline applies changes against the ops workspace on every commit to master.

    Github master branch pipeline run

  5. Finally, set a tag to apply changes to apps.

    # checkout the master branch
    git checkout master
    # pull changes from origin
    git pull
    # tag the merge commit
    git tag apps-deploy-0
    # push the tag to origin to trigger the pipeline
    git push origin apps-deploy-0

    Github tag pipeline run

Compare the output of the three pipeline runs. You will see how when triggered from a feature branch, triggered from the master branch or triggered from a tag the pipeline behaves differently.

For more details refer to the GitOps Flow making changes section.

Recap

To recap:

  • You bootstrapped a local repository.
  • Created prerequisites like the Terraform remote state and the identity for the automation runs.
  • You provisioned both the ops and apps infrastructure environments.
  • Setup DNS and Ingress to be able to expose workloads.
  • You linked your repository to trigger automated pipeline runs.

Congratulations, you now have a fully automated GitOps infrastructure.

Professional Services: For organizations interested in accelerating their GitOps journey.

  • POC Workshops
  • Enterprise Integration
  • Training and Support

Learn more about available services >>