Provision Nginx Ingress to expose applications outside the cluster

TL;DR:

Nginx ingress controller dynamically configures Nginx to proxy requests to Kubernetes services
This guide provisions the ingress controller, a cloud load balancer and sets up DNS
Once completed, you can expose services outside the cluster by creating Ingress resources

Introduction

Nginx is one of the most widely used reverse proxies. The Nginx ingress controller integrates Nginx with Kubernetes and dynamically configures Nginx to proxy requests to applications on Kubernetes.

Following this guide you will:

provision Nginx ingress controller
expose it outside the cluster using a cloud load balancer and
configure the DNS zones provisioned by the Kubestack cluster modules to resolve to the load balancer and your ingress controller

This guide assumes that you have set up DNS following the DNS set up guide.

Before we can provision the Nginx ingress controller, we need a Kubestack repository. If you do not have a Kubestack repository yet, follow the Kubestack tutorial first. This guide assumes you have a Kubestack framework repository.

Nginx Ingress Installation

To install the Nginx ingress controller, ask your agent that has learned the Kubestack skill to add it as a platform feature.

# add nginx feature to every cluster
Add the nginx platform feature to all clusters

# to only add to a single cluster
Add the nginx platform feature to the eks_gc0_eu-west-1 cluster

Your agent scaffolds the feature module modules/nginx/ and one binding file per cluster. To render the upstream ingress-nginx chart into modules/nginx/manifests/upstream.yaml, your agent asks you to run the helm template command it provides:

helm template ingress-nginx oci://registry.k8s.io/ingress-nginx \
  --version <chart-version> \
  --values modules/nginx/values.yaml \
  --namespace ingress-nginx \
  --create-namespace \
  --include-crds \
  > modules/nginx/manifests/upstream.yaml

The exact command and the chart version are recorded in modules/nginx/README.md. If you are migrating an existing Nginx catalog module, the migration guide walks you through the transition.

Patch load balancer IP/CNAME

The Nginx ingress upstream manifests include a service type load balancer to route the traffic to the ingress pods. But to allow requests to hit the cloud load balancer the Kubernetes control plane provisions we have to make sure DNS resolves to the LBs IP/CNAME.

AmazonAzureGoogle

For EKS, the provisioned load balancer will return a CNAME. We have to add this CNAME to the DNS zone that has been provisioned by the Kubestack cluster module. To do so, open the <cluster_name>_feature_nginx.tf binding files for all EKS clusters and add a second module. The first module, scaffolded by your agent, provisions the upstream manifests including a service type loadbalancer. We now add a second module to read the CNAME of the created ELB and add it to the DNS zone created by the cluster module.

module "eks_gc0_eu-west-1_feature_nginx" {
  providers = {
    kustomization = kustomization.eks_gc0_eu-west-1
  }
  source = "./modules/nginx"
}

+ module "eks_gc0_eu-west-1_dns_zone" {
+   providers = {
+     aws        = aws.eks_gc0_eu-west-1
+     kubernetes = kubernetes.eks_gc0_eu-west-1
+   }
+
+   # make sure to match your cluster module's version
+   source = "github.com/kbst/terraform-kubestack//aws/cluster/elb-dns?ref=<framework version>"
+
+   ingress_service_name      = "ingress-nginx-controller"
+   ingress_service_namespace = "ingress-nginx"
+
+   metadata = module.eks_gc0_eu-west-1.current_metadata
+ }

Reading the CNAME

For the data source in the DNS module to be able to read the CNAME from the service's status, the ELB created from the service type loadbalancer resource has to exist first. There are two ways to achieve this.

Either, run tofu apply --target module.eks_gc0_eu-west-1_feature_nginx to deploy the Nginx manifests. Then you can let the pipeline handle the rest.
Or, if you prefer to avoid the manual apply, split this change up into two commits, and merge and promote them one after the other.

A note about depends_on on the DNS zone module. Generally, you could make the dependency explicit by using depends_on = [module.eks_gc0_eu-west-1_feature_nginx]. This works initially, but occasionally the data source will be pushed into apply phase, and as a result the DNS entries will get recreated. To avoid this recreation, and the DNS entries being temporarily unresolvable as a result, the guide here avoids setting depends_on and offers the two alternatives above.

Apply Changes

As with every change, we now follow the GitOps process. First, commit and push to start the peer review, then merge when the plan looks good. After the changes have been validated in the internal environment, promote the changes to the external environment.

The full workflow is documented on the GitOps process page.

But here's a short summary for convenience:

# create a new feature branch
git checkout -b add-nginx-ingress

# add the changes and commit them
git add .
git commit -m "Install nginx ingress, cloud loadbalancer and DNS"

# push the changes to trigger the pipeline
git push origin add-nginx-ingress

Then follow the link in the output, to create a new pull request. Review the pipeline run. And merge the pull request, when everything is green.

Last but not least, promote the changes once you validated them in ops by setting a tag.

# make sure you're on the merge commit
git checkout main
git pull

# then tag the commit 
git tag apps-deploy-$(git rev-parse --short HEAD)

# finally push the tag, to trigger the pipeline to promote
git push origin apps-deploy-$(git rev-parse --short HEAD)

Next Steps

With Nginx ingress controller deployed, a common next step is to also deploy Cert-Manager and configure it to issue Let's Encrypt certificates.

The Cert-Manager and Let's Encrypt guide will walk you through setting this up.

If you haven't yet, you must set up DNS for both, applications be reachable from the internet, as well as Let's Encrypt to be able to issue certificates.