---
name: kubestack
description: Scaffold and maintain Kubestack framework repositories — OpenTofu/Terraform code for Kubernetes platform infrastructure on AWS, Google Cloud, Azure, and Scaleway. Use when initializing a new Kubestack repository or when changing clusters, node pools, environments, or platform features in an existing one.
license: Apache-2.0
compatibility: Requires OpenTofu or Terraform to validate configurations, git for the GitOps workflow, and network access to fetch the framework version from kubestack.com and upstream manifests or Helm charts. Helm is required for chart-based platform features.
metadata:
  author: kbst
  source: https://github.com/kbst/terraform-kubestack
---

# Kubestack Agent Skill — Framework Usage

> This skill teaches an AI coding agent to scaffold and maintain Kubestack framework repositories — no CLI needed. For contributing to the framework itself, see the `AGENTS.md` in the framework repository root.

## Start Here

This skill is published at https://www.kubestack.com/SKILL.md. Read it end to end before starting any task. If the repository you are working in contains its own `AGENTS.md`, use the local copy — it is this same skill, kept in the repository so every agent session has it available. Refresh it from the published URL occasionally, see [Other hard rules](#other-hard-rules).

For a brand-new repository, follow [Scaffolding a New Repository](#scaffolding-a-new-repository) first — it creates the `README.md` and other shared files the rest of this skill reads from. On an existing repository:

**Before doing anything else, read the Platform section of `README.md`.**
Use every value recorded there as-is — never ask the user for a value that is already recorded. If a required value is missing, ask the user, record the answer in `README.md`, then write the files.

---

## Constraints

These rules are absolute. They apply to every task without exception.

### Commands you must never run

- `terraform apply` / `tofu apply`
- `terraform destroy` / `tofu destroy`
- `terraform init` / `tofu init`
- `terraform plan` / `tofu plan`
- Any `git` command that's not read-only — no commits, pushes, branch creation, or tagging

These commands make irreversible changes to live infrastructure or repository state. A human who has reviewed what will happen must always run them. If the user asks you to run any of them, explain that you cannot and give them the exact command to run themselves.

### The one command you must run

After writing or editing any `.tf` file, run `tofu validate` (or `terraform validate`). It is safe — no cloud API calls, changes nothing.

### Other hard rules

- Never modify `.terraform.lock.hcl` by hand.
- Never hardcode secrets, credentials, keys, or certificates in any `.tf` file.
- Never edit `modules/<feature_name>/manifests/upstream.yaml` directly for Helm-based features — it is a generated file. To change what is rendered, edit `values.yaml` and re-run the `helm template` command recorded in `modules/<feature_name>/README.md`.
- Refresh this skill occasionally: fetch the published version at https://www.kubestack.com/SKILL.md — at the latest whenever unsure whether the local copy is still current — and compare it to the repository's `AGENTS.md`. If it differs, tell the user and offer to update the local copy.
- Always update the Platform section of `README.md` after any task that adds, changes, or removes a platform component or setting.
- Always end every task with the GitOps follow-up instructions from [After Every Task](#after-every-task).
- In staged processes, prepare the file changes for the current stage only, hand off with the GitOps follow-up instructions, and wait for the user to confirm the pipeline has successfully applied before starting the next stage. Never run any git or terraform command beyond `validate` and read-only git.

---

## Update Obligations

Before considering a task complete, apply every obligation that matches what was done. When updating `README.md`, replace existing values — never append duplicates or leave stale entries.

| What was done | Obligation |
|---|---|
| First-time repository scaffold | Fill the Configuration table in `README.md` (framework version, environment names, base environment, `base_domain`). Add the first cluster to the Clusters table. |
| New cluster scaffolded (Stage 1) | Add a row to the Clusters table. Give the GitOps follow-up instructions; remind that node pools and platform features must follow in separate stages. |
| Cluster removed | Remove its row from the Clusters table and all its node pool and feature rows. |
| New node pool scaffolded | Add a row to the Node Pools table. |
| Node pool removed | Remove its row from the Node Pools table. |
| New platform feature scaffolded | Add a row to the Platform Features table. |
| Catalog module migrated to a platform feature | Follow [Migrating a Catalog Module to a Platform Feature](#migrating-a-catalog-module-to-a-platform-feature). Add a `moved` block per cluster binding file and preserve the namespace. Update or add the feature's row in the Platform Features table. |
| Platform feature removed | Remove its row from the Platform Features table. |
| Framework version changed | Update the version in the Configuration table, the `?ref=` value in every Kubestack framework module `source` reference, and the image tag in `Dockerfile`. |
| Framework version upgraded from v0.19.2-beta.0 or earlier | Apply every step of [Upgrading from v0.19.2-beta.0 or earlier](#upgrading-from-v0192-beta0-or-earlier) in addition to the version-reference updates. |
| Environment added | Add the name to the Configuration table and the new key to the `configuration` map in every module in the repository. |
| Environment removed | **Destructive.** Do not make any file changes immediately — follow [Removing an environment](#removing-an-environment). |
| Environment renamed | **Destructive.** Do not make any file changes immediately — follow [Renaming an environment](#renaming-an-environment). |
| Any task completed | Give the user the full GitOps follow-up instructions from [After Every Task](#after-every-task). |

---

## Framework Version

The framework version governs all release assets in the repository. These must always be identical — upstream releases bundle modules and the OCI image together:

- The `?ref=<version>` parameter in every Kubestack framework module `source` reference
- The container image tag in `Dockerfile`

Read the current version from the Configuration table in `README.md`. When adding a module to an existing repository, match the version already in use. When initialising a new repository or updating the version, fetch the latest from https://www.kubestack.com/cli.json. On a version update of an existing repository, apply the matching upgrade instructions from [Framework Upgrades](#framework-upgrades), if one exists for the version boundary being crossed.

All framework module sources follow the same GitHub format:

```
# cluster module
source = "github.com/kbst/terraform-kubestack//aws/cluster?ref=v0.18.1-beta.0"
# node-pool module
source = "github.com/kbst/terraform-kubestack//aws/cluster/node-pool?ref=v0.18.1-beta.0"
# platform feature module
source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=v0.18.1-beta.0"
```

### After a framework version update

Releases may update required provider versions. After changing any `?ref=` or `Dockerfile` image tag, remind the user to run the following before committing:

```
tofu init -upgrade
```

This refreshes `.terraform.lock.hcl` to the provider versions required by the new release. The updated lock file must be included in the same commit as the version change.

---

## Framework Upgrades

Framework releases occasionally contain breaking changes to the module interface. The upgrade instructions below describe how to change an existing repository to match the breaking changes of a newer framework version; each lists every change that requires action and the steps to perform it. Resource renames and moves inside the modules are always expressed as `moved` blocks in the released module code — never run `tofu state mv` or any other manual state operation; the moved blocks apply automatically when the user runs the next plan.

### Upgrading from v0.19.2-beta.0 or earlier

The release that follows v0.19.2-beta.0 changes the cluster and node-pool module interface on every provider. Apply all steps below in order when upgrading a repository on v0.19.2-beta.0 or earlier. The switch to OpenTofu and the new Scaleway modules require no configuration changes in existing repositories, with two exceptions: the Kubestack container image now ships `tofu` instead of `terraform`, so any pipeline or script that invokes `terraform` inside the image must call `tofu` instead, and the `kind` image flavor no longer exists.

**Step 1 — Update version references and providers.** Update the version in the Configuration table in `README.md`, every module `source` `?ref=` value, and the `Dockerfile` image tag — see [Framework Version](#framework-version). The AWS cluster module now requires the `tls` provider; the Azure cluster module now requires the `azuread` provider. Remind the user to run `tofu init -upgrade` before committing and to include the refreshed `.terraform.lock.hcl` in the same commit.

**Step 2 — Convert configuration values to native types.** The `configuration` variable of every cluster and node-pool module changed from loosely typed `map(map(string))` to typed objects. Apply these conversions to every value in every environment key of every module:

| Old value format | New value format |
|---|---|
| Numbers as strings: `min_size = "1"` | Numbers: `min_size = 1` |
| Booleans as strings: `cluster_vpc_legacy_node_subnets = "false"` | Booleans: `cluster_vpc_legacy_node_subnets = false` |
| Comma-separated lists: `cluster_availability_zones = "eu-west-1a,eu-west-1b"` | HCL lists: `cluster_availability_zones = ["eu-west-1a", "eu-west-1b"]` |
| `key=value` pairs: `additional_metadata_labels = "team=platform,costcenter=cc1"` | Maps: `additional_metadata_labels = { team = "platform", costcenter = "cc1" }` |

Attributes not mentioned in the tables below keep their names — only their value types change.

**Step 3 — Move the default node pool into `default_node_pool`.** Default node pool settings move from flat top-level attributes into a nested `default_node_pool` object. Apply the provider-specific renames below.

EKS (`aws/cluster`):

| v0.19.2-beta.0 | New |
|---|---|
| `cluster_instance_type` or `cluster_instance_types` (comma-separated string) | `default_node_pool.instance_types` (list) |
| `cluster_desired_capacity`, `cluster_min_size`, `cluster_max_size` (strings) | `default_node_pool.desired_capacity`, `.min_size`, `.max_size` (numbers) |
| `metadata_options_http_endpoint`, `metadata_options_http_tokens`, `metadata_options_http_put_response_hop_limit`, `metadata_options_http_protocol_ipv6`, `metadata_options_instance_metadata_tags` | `default_node_pool.metadata_options` object |
| `cluster_additional_node_tags` (`key=value` string) | `default_node_pool.additional_node_tags` (map) |
| `worker_root_device_volume_size`, `worker_root_device_encrypted` | `default_node_pool.root_device_volume_size` (number), `.root_device_encrypted` (bool) |
| `worker_ami_release_version` | `default_node_pool.ami_release_version` |

GKE (`google/cluster`):

| v0.19.2-beta.0 | New |
|---|---|
| `cluster_machine_type` | `default_node_pool.machine_type` |
| `cluster_initial_node_count`, `cluster_min_node_count`, `cluster_max_node_count` (strings) | `default_node_pool.initial_node_count`, `.min_node_count`, `.max_node_count` (numbers) |
| `cluster_node_location_policy` | `default_node_pool.node_location_policy` |
| `cluster_extra_oauth_scopes` (comma-separated) | `default_node_pool.extra_oauth_scopes` (list) |
| `cluster_disk_size_gb` (string), `cluster_disk_type`, `cluster_image_type` | `default_node_pool.disk_size_gb` (number), `.disk_type`, `.image_type` |
| `cluster_preemptible`, `cluster_auto_repair`, `cluster_auto_upgrade` | `default_node_pool.preemptible`, `.auto_repair`, `.auto_upgrade` (bools) |
| `remove_default_node_pool` | `default_node_pool.remove_default_node_pool` |
| `cluster_maintenance_exclusion_start_time`, `cluster_maintenance_exclusion_end_time`, `cluster_maintenance_exclusion_name`, `cluster_maintenance_exclusion_scope` | `cluster_maintenance_exclusion` object with `start_time`, `end_time`, `name`, `scope` |
| `router_advertise_config_groups`, `router_advertise_config_ip_ranges`, `router_advertise_config_mode` | `router_advertise_config` object with `groups` (set), `ip_ranges` (map), `mode` |
| `logging_config_enable_components` (comma-separated) | `logging_config` object with `enable_components` (list) |
| `monitoring_config_enable_components` (comma-separated) | `monitoring_config` object with `enable_components` (list) |

AKS (`azurerm/cluster`):

| v0.19.2-beta.0 | New |
|---|---|
| `default_node_pool_name`, `default_node_pool_type` | `default_node_pool.name`, `default_node_pool.type` |
| `default_node_pool_enable_auto_scaling` | `default_node_pool.enable_auto_scaling` (bool) |
| `default_node_pool_min_count`, `default_node_pool_max_count`, `default_node_pool_node_count` (strings) | `default_node_pool.min_count`, `.max_count`, `.node_count` (numbers) |
| `default_node_pool_vm_size` | `default_node_pool.vm_size` |
| `default_node_pool_only_critical_addons` | `default_node_pool.only_critical_addons` (bool) |
| `default_node_pool_os_disk_size_gb` (string) | `default_node_pool.os_disk_size_gb` (number) |
| `upgade_settings_max_surge`, `upgade_settings_drain_timeout_in_minutes`, `upgade_settings_node_soak_duration_in_minutes` (the old names contain a typo) | `default_node_pool.upgrade_settings` object with `max_surge`, `drain_timeout_in_minutes` (number), `node_soak_duration_in_minutes` (number) |

**Step 4 — Set newly required attributes.** Required configuration attributes are now enforced by `lifecycle` preconditions on the cluster and node-pool resources; a missing attribute fails planning with `missing required configuration attribute: <name>`. Add any that are missing:

| Provider | Required cluster attributes | Required default node pool attributes |
|---|---|---|
| EKS | `cluster_availability_zones` (list) | `instance_types`, `min_size`, `max_size` |
| GKE | `project_id`, `region` | `machine_type`, `min_node_count`, `max_node_count` |
| AKS | `resource_group`, `availability_zones` (list of zone numbers as strings) | `vm_size`, plus `min_count` and `max_count` unless `enable_auto_scaling = false` |

**Step 5 — Rewire node-pool module blocks.** The node-pool submodules no longer look up their cluster by name; they receive the cluster resource object directly from the cluster module. Change the cluster identity inputs of every additional node pool module block:

| Provider | v0.19.2-beta.0 inputs | New inputs |
|---|---|---|
| EKS | `cluster_name = module.<cluster>.current_metadata["name"]` | `cluster = module.<cluster>.cluster` and `cluster_metadata = module.<cluster>.current_metadata` |
| GKE | `cluster_metadata = module.<cluster>.current_metadata` | `cluster = module.<cluster>.cluster` and keep `cluster_metadata` |
| AKS | `cluster_name = module.<cluster>.current_metadata["name"]` and `resource_group = module.<cluster>.current_config["resource_group"]` | `cluster = module.<cluster>.cluster` |

Then apply the Step 2 type conversions to the node-pool `configuration` maps as well: `instance_types`, `min_size`, `max_size`, `desired_capacity`, and `disk_size` become numbers or lists, and `availability_zones` and `vpc_subnet_ids` become lists.

**Step 6 — Update direct `elb-dns` references (EKS only).** The `aws/cluster/elb-dns` module replaces its `metadata_fqdn` string input with a `metadata` object. Pass `metadata = module.<cluster>.current_metadata` in any direct reference. Skip if the repository does not reference `elb-dns` directly — the cluster module calls it internally.

**Step 7 — Remove references to deleted modules.** The experimental `aws/cluster-local` and `google/cluster-local` modules were removed without a replacement. Repositories that use them must remove those module blocks; there is no in-place upgrade path.

**Step 8 — Expect an EKS node AMI change.** EKS node pools now default to Amazon Linux 2023 AMI types (`AL2023_x86_64_STANDARD`, `AL2023_ARM_64_STANDARD`, `AL2023_x86_64_NVIDIA`), and node pools that use a launch template resolve an AL2023 AMI that cannot be pinned to Amazon Linux 2. The first apply after the upgrade therefore replaces existing Amazon Linux 2-based nodes. To keep a plain node group on Amazon Linux 2, set its `ami_type` explicitly (e.g. `AL2_x86_64`). Tell the user about this before hand-off so the node replacement is expected.

**Step 9 — Validate and hand off.** Run `tofu validate`. Do not run `tofu plan` or `tofu apply` — see [Constraints](#constraints). Hand off with the GitOps follow-up instructions from [After Every Task](#after-every-task) and remind the user that the moved blocks in the modules report the resource moves on the first plan of each environment.

---

## About Kubestack

Kubestack is an OpenTofu/Terraform framework for platform engineering teams:

- **Cluster modules** — provision a managed Kubernetes cluster (EKS on AWS, GKE on Google Cloud, AKS on Azure, Kapsule on Scaleway) and all required cloud infrastructure (VPC, IAM, subnets, etc.).
- **Node-pool modules** — add extra node pools to an existing cluster.
- **Platform feature modules** — deploy Kubernetes resources to a cluster (nginx ingress, Prometheus, cert-manager, etc.).

All modules implement a shared configuration inheritance model so that all environments derive from a single base configuration, preventing drift. Core concepts:

- **Environments = Terraform workspaces.** Each environment (e.g. `ops`, `apps`) is a completely independent copy of the entire platform in the cloud. Nothing is shared between environments.
- **Inheritance.** The base environment (`apps`, or whichever key is set as `configuration_base_key`) is inherited by all other environments, which may override individual values. It is not possible to unset an inherited value by setting it to `null`.
- **Cluster naming.** Cluster module names follow `<provider>_<name_prefix>_<region>`, e.g. `eks_gc0_eu-west-1`. The name is the Terraform module identifier, the file prefix, and the provider aliases.
- **File-per-cluster convention.** Each cluster has its own `<cluster_name>_cluster.tf` and `<cluster_name>_providers.tf`. Node-pool and feature files are prefixed with the cluster name.

---

## Naming Rules

This section covers decisions that may not yet be recorded. Once a decision is made, record it in `README.md` and read it from there on every subsequent task — never re-derive it.

### Environment names

The default environment names are `ops` (internal, validates changes first) and `apps` (external, runs user workloads). These become Terraform workspace names.

| Pattern | When to use |
|---|---|
| `ops`, `apps` | Default. All application environments (dev, stage, prod) run inside the single `apps` infrastructure environment. |
| `ops`, `apps`, `apps-prod` | When non-prod and prod application workloads need blast-radius isolation at the infrastructure level. |

Environment names must be consistent across every module in the repository. With three environments, set `configuration_base_key = "apps-prod"` so `apps` and `ops` both inherit from the production baseline. Order `configuration` map keys from most critical to least critical (e.g. `apps-prod`, then `apps`, then `ops`) — see [Configuration Inheritance](#configuration-inheritance).

Choosing the right separation mechanism:

| Goal | Use |
|---|---|
| Separate non-prod and prod workloads | Node pools (low overhead) or a third environment (maximum blast-radius protection). Not clusters. |
| Separate tenants or workload types | Node pools (low overhead) or additional clusters (maximum blast-radius protection). Not environments. |

### Changing environments on an existing platform

Environment names are recorded once at platform initialisation; changing them on a running platform is not a routine operation. **Adding** an environment is the least disruptive change — it provisions a new independent copy of the entire platform, and existing environments are unaffected: add the new key to the `configuration` map in every module and update the Configuration table in `README.md`. **Removing or renaming** follows the staged processes below. These are multi-stage operations spanning weeks or months — not single commits. At each stage, prepare the code changes for that stage only, hand off with the GitOps follow-up instructions and explicit guidance on what must happen operationally before the next stage can begin.

### Removing an environment

Removing an environment destroys every resource in it — all clusters, node pools, and platform features it contains. Any application relying on that infrastructure loses it permanently. Explain this to the user and get explicit confirmation before making any file change. Then follow three stages:

**Stage 1 — Announce deprecation.** Help the user communicate clearly to all teams using the environment that it will be shut down, giving them enough time to migrate their workloads elsewhere. You cannot send announcements, but you can help draft the message. No file changes are made. The stage is complete when the user confirms the deprecation period has ended and all teams have migrated.

**Stage 2 — Remove platform features and node pools.** Follow the same staged process as [Removing a Cluster](#removing-a-cluster) across every cluster in the environment: first remove all platform features (Stage 1 of that process), then all additional node pools (its Stage 2). Each sub-stage is its own GitOps cycle — prepare the file changes for whichever sub-stage the user is performing, hand off, and wait for confirmation before proceeding.

**Stage 3 — Remove the environment key.** Remove the environment's key from the `configuration` map in every module across the repository, and from the Configuration table in `README.md`. Hand off with the GitOps follow-up instructions and a clear warning that the Terraform plan will show all remaining infrastructure in that environment being destroyed — the expected signal for the reviewer to approve.

### Renaming an environment

There is no in-place rename. Terraform workspaces are identified by name; renaming means provisioning all resources from scratch under the new name and then destroying all resources under the old name. Applications must migrate between the two environments while both run in parallel — a process spanning multiple weeks or months. Explain this to the user and get explicit confirmation before making any file change. Then follow three stages:

**Stage 1 — Add the new environment.** Add the new environment key to the `configuration` map in every module and update the Configuration table in `README.md`. Follow the "Environment added" obligation and hand off with the GitOps follow-up instructions. The new environment provisions from scratch when the pipeline runs; the old one continues unchanged. The stage is complete when the user confirms the new environment is healthy and teams are ready to begin migrating.

**Stage 2 — Migrate applications.** You cannot perform application migration. Remind the user that all teams must move their workloads from the old environment to the new one before the old environment can be removed. No file changes are made. The stage is complete when the user confirms all applications have migrated and the old environment is no longer in use.

**Stage 3 — Remove the old environment.** Follow all three stages of [Removing an environment](#removing-an-environment), starting from its Stage 2 (the deprecation announcement was effectively the migration period in Stage 2 above). Hand off after each sub-stage and wait for confirmation before proceeding.

### `name_prefix`

A short label identifying the cluster's purpose, tenant, or compute type. The cluster name is `<name_prefix>-<workspace>-<region>`, e.g. `gc0-apps-eu-west-1`. The numeric suffix makes the prefix a stable, incrementable identity: `gc0` — first general-compute cluster (add `gc1` when `gc0` approaches capacity, before decommissioning `gc0`); `ml0` — first machine-learning cluster; `data0`, `platform0` — departmental or workload-type clusters.

Derive the `name_prefix` for a new cluster from the discovery questions in [Adding a Cluster](#adding-a-cluster):
- Same cloud, same region, same purpose → increment the counter on the existing prefix (e.g. `gc0` → `gc1`).
- Same cloud, same region, different purpose → new prefix reflecting the new purpose (e.g. `ml0`).
- New region or new cloud, same purpose → reuse the same prefix as the equivalent cluster in the other region or cloud (e.g. `gc0` again — the region in the cluster identifier already distinguishes it).

### `base_domain`

The domain suffix used to construct each cluster's FQDN: `<cluster_name>.<provider_short>.<base_domain>`, e.g. `gc0-apps-eu-west-1.aws.infra.example.com`. The user must own and control this domain. It is used for DNS and TLS configuration by platform features.

### Cluster identifier

The Terraform module name, file prefix, and provider alias all share the identifier `<provider_prefix>_<name_prefix>_<region_slug>`:

| Provider | Prefix | Region slug |
|---|---|---|
| AWS EKS | `eks` | `eu-west-1` (as-is) |
| Google GKE | `gke` | `europe-west1` (as-is) |
| Azure AKS | `aks` | `westeurope` (as-is) |
| Scaleway | `scw` | `fr-par` (as-is) |

Examples: `eks_gc0_eu-west-1`, `gke_gc0_europe-west1`, `aks_gc0_westeurope`, `scw_gc0_fr-par`.

---

## Repository Layout

```
.
├── Dockerfile                               # pins the Kubestack container image version
├── .gitignore                               # excludes .user/ and .terraform/
├── versions.tf                              # required_providers and version constraints
├── state.tf                                 # remote state backend (created during bootstrap)
│
├── <cluster_name>_cluster.tf                # one file per cluster
├── <cluster_name>_providers.tf              # one file per cluster
│
├── <cluster_name>_node_pool_<name>.tf       # one file per additional node pool
├── <cluster_name>_feature_<feature_name>.tf # one file per platform feature, per cluster
│
├── modules/                                 # one subdirectory per platform feature
│   └── <feature_name>/
│       ├── README.md                        # upstream source, version, and regeneration command
│       ├── versions.tf                      # required_providers for the module
│       ├── variables.tf                     # only when the module has input variables
│       ├── outputs.tf                       # only when the module exposes output values
│       ├── main.tf                          # calls the kustomization/overlay framework module
│       ├── values.yaml                      # only when upstream is a Helm chart
│       └── manifests/
│           └── upstream.yaml               # rendered Kubernetes resources for this feature
│
├── README.md
├── AGENTS.md                                # this file
└── .user/                                   # excluded from Git; persists cloud CLI auth
```

---

## Scaffolding a New Repository

No CLI and no pre-existing files are required. Create the shared files below in the order given, then scaffold the first cluster. `README.md` comes first because every other part of this skill reads platform decisions from it. Before writing any file, fetch the latest framework version from https://www.kubestack.com/cli.json and confirm the platform decisions from [Naming Rules](#naming-rules) with the user.

### `README.md`

Create `README.md` with the Platform section as its core — the shared source of truth for how the platform is configured, read and updated by every later task. Use the template below:

```
# Infrastructure Automation

This repository uses the [Kubestack](https://www.kubestack.com) framework to define and manage its Kubernetes platform.
Full [framework documentation](https://www.kubestack.com/framework/documentation/) is available online.

## Platform

This section is the shared source of truth for how this platform is configured.
It is kept up to date by both human contributors and agents as the platform evolves.
When making any change that adds, removes, or modifies a platform component, update the relevant table here as part of the same commit.

### Configuration

| Setting | Value |
|---|---|
| Framework version | |
| Environment names | |
| Base environment | |
| `base_domain` | |
| State backend | |

### Clusters

| Cluster name | Provider | Region | `name_prefix` |
|---|---|---|---|

### Node Pools

Additional node pools only — the default node pool for each cluster is configured inside the cluster module.

| Module name | Cluster | Purpose / notes |
|---|---|---|

### Platform Features

| Module name | Cluster | Notes |
|---|---|---|
```

As decisions are confirmed, record them in the Configuration table: framework version, environment names, base environment (`apps` for two environments, `apps-prod` for three), and `base_domain`. The `State backend` row is filled in when `state.tf` is created.

### `.gitignore`

```
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# .user home directory
.user/
```

### `versions.tf`

Always include `kbst/kustomization` and `hashicorp/kubernetes`, plus one entry per cloud provider the repository uses:

```
terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }

    kustomization = {
      source = "kbst/kustomization"
    }

    # Add per cloud provider in use:
    # aws      = { source = "hashicorp/aws" }
    # azurerm  = { source = "hashicorp/azurerm" }
    # google   = { source = "hashicorp/google" }
    # scaleway = { source = "scaleway/scaleway" }
  }
}
```

### `state.tf`

Defer this file until the user has provisioned remote state storage and all required values are known — it is not needed to complete the initial scaffold. If the user has not decided on a backend, ask before writing this file; once decided, record it in the Configuration table under "State backend". The backend need not match the cluster's cloud provider — any backend OpenTofu/Terraform supports may be used.

#### Bucket naming

Object storage bucket and container names must be globally unique. Append the short git hash of the repository's initial commit as a suffix: `terraform-state-kubestack-<short-hash>`, e.g. `terraform-state-kubestack-a1b2c3d`. The short hash ties the bucket to this repository and is stable. Obtain it after the first commit by running inside the repository:

```
git rev-parse --short HEAD
```

If the repository has no commits yet, ask the user to make an initial commit first. This pattern applies to the bucket name for AWS S3, GCS, and Scaleway, and the container name for Azure Blob Storage (the storage account name is pre-existing and supplied by the user separately).

**AWS S3:**
```
terraform {
  backend "s3" {
    bucket = "<bucket-name>"
    region = "<region>"
    key    = "tfstate"
  }
}
```

**Google Cloud Storage:**
```
terraform {
  backend "gcs" {
    bucket = "<bucket-name>"
  }
}
```

**Azure Blob Storage:**
```
terraform {
  backend "azurerm" {
    storage_account_name = "<storage-account>"
    container_name       = "<container-name>"
    key                  = "tfstate"
  }
}
```

**Scaleway Object Storage** (S3-compatible; bucket must already exist):
```
terraform {
  backend "s3" {
    bucket = "<bucket-name>"
    key    = "tfstate"

    # Replace <region> with your bucket region, e.g. fr-par, nl-ams, pl-waw.
    region   = "<region>"
    endpoint = "https://s3.<region>.scw.cloud"

    # Credentials: set SCW_ACCESS_KEY and SCW_SECRET_KEY environment variables.
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    use_path_style              = true
  }
}
```

For any other supported backend, refer to the [OpenTofu backend documentation](https://opentofu.org/docs/language/settings/backends/).

### `Dockerfile`

Pins the Kubestack container image used for CI/CD and local operations. The image version must always match the `?ref=` version in all module sources — see [Framework Version](#framework-version). Read the version from `README.md`, or fetch it from https://www.kubestack.com/cli.json when initialising a new repository.

```
FROM kubestack/framework:<version>-<flavor>
```

The flavor suffix matches the repository's primary cloud provider: `eks`, `gke`, `aks`, or `scw`. Multi-cloud repositories use the flavor-less tag: `FROM kubestack/framework:<version>`.

### `AGENTS.md`

Save this skill as `AGENTS.md` in the repository root so every future agent session loads it automatically. With network access, fetch the exact published copy instead of transcribing from memory:

```
curl -L https://www.kubestack.com/SKILL.md -o AGENTS.md
```

If the skill was learned from a local file, copy that file to the new repository.

### Scaffold the first cluster

With the shared files in place, scaffold the first cluster following [Scaffolding a Cluster](#scaffolding-a-cluster), then:

1. Run `tofu validate` (or `terraform validate`) to verify syntax.
2. Complete the first-time scaffold obligations from [Update Obligations](#update-obligations).
3. Summarize the bootstrap steps the user must still perform before the platform is live:
   - Make the initial commit — the short git hash is needed for the state bucket name when creating `state.tf`.
   - Provision the remote state backend, then have you write `state.tf` and record it in `README.md`.
   - Create one Terraform workspace per environment.
   - Bootstrap the environments and set up the automation pipeline.

---

## Configuration Inheritance

Every module takes a `configuration` map keyed by environment name and an optional `configuration_base_key`.

- `configuration_base_key` defaults to `apps`. All other environment keys inherit every attribute from the base.
- An attribute set in a non-base environment overrides the inherited value; an attribute absent from it is transparently inherited.
- It is **not** possible to unset an inherited value by setting it to `null`.
- **Order `configuration` map keys from most critical to least critical environment** (e.g. `apps-prod`, then `apps`, then `ops`) — most-important-first, making the blast-radius priority immediately visible when reading the file.

### Default two-environment layout

```
configuration = {
  apps = {
    # All required and optional attributes go here.
    # ops inherits everything and may override individual values.
  }

  ops = {}
}
```

### Three-environment layout

Use `apps-prod` as the base when the user wants a separate production environment:

```
configuration_base_key = "apps-prod"

configuration = {
  apps-prod = {
    # Production values — this is the base all others inherit from.
  }

  apps = {
    # Non-prod overrides, e.g. smaller instance type or replica count.
  }

  ops = {
    # Internal validation environment overrides.
  }
}
```

### Scaling down non-production environments

Set full production-scale values in the base environment and override only cost-sensitive attributes in non-production environments. `ops` `default_node_pool` overrides by provider:

| Provider | Attributes (example) |
|---|---|
| EKS | `instance_types = ["t3.medium"]`, `min_size = 1`, `max_size = 3`, `desired_capacity = 1` |
| GKE | `machine_type = "e2-medium"`, `min_node_count = 1`, `max_node_count = 1` (counts are per zone) |
| AKS | `vm_size = "Standard_B2s"`, `min_count = 1`, `max_count = 3` |

---

## Scaffolding a Cluster

Each cluster requires exactly two files: `<cluster_name>_cluster.tf` and `<cluster_name>_providers.tf`. Before writing either:

1. Read `base_domain`, environment names, and framework version from the Platform section of `README.md`.
2. Read existing cluster entries to confirm the `name_prefix` convention already in use.

### Provider aliasing

Every provider in a cluster's providers file carries an `alias` equal to the cluster identifier — uniformly for the cloud provider (`aws`, `google`, `azurerm`, `scaleway`), `kustomization`, and `kubernetes`, with no per-provider exceptions. Each module type receives only the providers it requires:

| Module type | Providers to pass |
|---|---|
| Cluster module | Cloud provider only |
| Node-pool module | Cloud provider only |
| Platform feature module | `kustomization` only |

The `kubernetes` provider is defined in the providers file for direct resource use if needed, but is not passed into cluster or node-pool modules.

Every providers file also carries the following identical blocks — only the cluster identifier changes:

```
provider "kustomization" {
  alias          = "<cluster_name>"
  kubeconfig_raw = module.<cluster_name>.kubeconfig
}

locals {
  <cluster_name>_kubeconfig = yamldecode(module.<cluster_name>.kubeconfig)
}

provider "kubernetes" {
  alias = "<cluster_name>"

  host                   = local.<cluster_name>_kubeconfig["clusters"][0]["cluster"]["server"]
  cluster_ca_certificate = base64decode(local.<cluster_name>_kubeconfig["clusters"][0]["cluster"]["certificate-authority-data"])
  token                  = local.<cluster_name>_kubeconfig["users"][0]["user"]["token"]
}
```

### Required values

If any of the following are missing from the Platform section of `README.md`, ask the user, record the answers, then write the files:

| Value | Where it goes |
|---|---|
| `name_prefix` | Inside `configuration.apps` |
| `base_domain` | Inside `configuration.apps` |
| Region | Provider block or `configuration.apps`, depending on provider |
| Availability zones | Inside `configuration.apps` (commented-out placeholder if unknown) |
| Instance / node type | Inside `configuration.apps.default_node_pool` |
| `min_size` and `max_size` (or equivalent) | Inside `configuration.apps.default_node_pool` |

The `configuration` map must include a key for every environment already in use in the repository — an empty `ops = {}` entry inherits from the base with no overrides.

### EKS (AWS)

**`eks_<name_prefix>_<region>_cluster.tf`**

```
module "eks_gc0_eu-west-1" {
  providers = {
    aws = aws.eks_gc0_eu-west-1
  }

  source = "github.com/kbst/terraform-kubestack//aws/cluster?ref=<version>"

  configuration = {
    apps = {
      name_prefix = "gc0"
      base_domain = "infra.example.com"

      default_node_pool = {
        instance_types   = ["t3.xlarge"]
        min_size         = 3
        max_size         = 9
        desired_capacity = 3
      }

      # EKS requires a minimum of 2 availability zones. Must match the region in the provider.
      # cluster_availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
    }

    ops = {}
  }
}
```

**`eks_<name_prefix>_<region>_providers.tf`** — the cloud provider block, plus the shared blocks above:

```
provider "aws" {
  alias  = "eks_gc0_eu-west-1"
  region = "eu-west-1"
}
```

### GKE (Google Cloud)

**`gke_<name_prefix>_<region>_cluster.tf`**

```
module "gke_gc0_europe-west1" {
  providers = {
    google = google.gke_gc0_europe-west1
  }

  source = "github.com/kbst/terraform-kubestack//google/cluster?ref=<version>"

  configuration = {
    apps = {
      name_prefix = "gc0"
      base_domain = "infra.example.com"

      # project_id = "my-gcp-project"
      # region     = "europe-west1"

      cluster_min_master_version = "1.30"

      default_node_pool = {
        # machine_type   = "e2-standard-4"
        # min_node_count = 1
        # max_node_count = 3
      }

      # Zones must be within the region above.
      # cluster_node_locations = ["europe-west1-b", "europe-west1-c", "europe-west1-d"]
    }

    ops = {}
  }
}
```

**`gke_<name_prefix>_<region>_providers.tf`** — plus the shared blocks above:

```
provider "google" {
  alias = "gke_gc0_europe-west1"
  # project and region are passed inside the cluster module configuration.
  # Set them here only if you want a provider-level default.
  # project = "my-gcp-project"
  # region  = "europe-west1"
}
```

### AKS (Azure)

**`aks_<name_prefix>_<region>_cluster.tf`**

```
module "aks_gc0_westeurope" {
  providers = {
    azurerm = azurerm.aks_gc0_westeurope
  }

  source = "github.com/kbst/terraform-kubestack//azurerm/cluster?ref=<version>"

  configuration = {
    apps = {
      name_prefix = "gc0"
      base_domain = "infra.example.com"

      resource_group = "my-resource-group"

      # Azure zone numbers are strings.
      availability_zones = ["1", "2", "3"]

      default_node_pool = {
        # vm_size   = "Standard_D4s_v3"
        min_count = 3
        max_count = 9
      }
    }

    ops = {}
  }
}
```

**`aks_<name_prefix>_<region>_providers.tf`** — plus the shared blocks above:

```
provider "azurerm" {
  alias    = "aks_gc0_westeurope"
  features {}
}
```

### Scaleway (Kapsule)

**`scw_<name_prefix>_<region>_cluster.tf`**

```
module "scw_gc0_fr-par" {
  providers = {
    scaleway = scaleway.scw_gc0_fr-par
  }

  source = "github.com/kbst/terraform-kubestack//scaleway/cluster?ref=<version>"

  configuration = {
    apps = {
      name_prefix = "gc0"
      base_domain = "infra.example.com"

      region = "fr-par"

      # Minor version string — patch upgrades are applied automatically.
      cluster_version = "1.35"

      default_node_pool = {
        node_type = "PRO2-S"

        zones = ["fr-par-1", "fr-par-2", "fr-par-3"]

        min_size = 1
        max_size = 3
      }
    }

    ops = {}
  }
}
```

**`scw_<name_prefix>_<region>_providers.tf`** — plus the shared blocks above:

```
provider "scaleway" {
  alias = "scw_gc0_fr-par"

  # Must be set explicitly — the target scope must never be inferred
  # from the CLI config or environment variables.
  organization_id = "my-organization-id"
  project_id      = "my-project-id"

  region = "fr-par"
}
```

---

## Adding a Cluster

Three stages — cluster first, then node pools, then platform features — each in its own GitOps cycle. This mirrors removal in reverse order and for the same reason: the Terraform dependency graph does not track cross-stage dependencies, and applying everything at once risks partial failures that are difficult to recover from. Prepare the file changes for one stage only, hand off with the GitOps follow-up instructions, and wait for confirmation that the pipeline has successfully applied before preparing the next stage.

### Stage 1 — Discovery and cluster scaffold

If not clear from the user's prompt already, hold a short discovery conversation. Record each answer in the Clusters table in `README.md` as it is confirmed — do not re-ask on subsequent stages.

**Question 1 — Same cloud and region, or new?**

| Scenario | Implication |
|---|---|
| Same cloud, same region, same purpose | Adding capacity. Mirror the existing cluster of that type exactly — same instance types, node pool shape, and features. Only the `name_prefix` counter increments. |
| Same cloud, same region, different purpose | New workload type. Mirror the existing cluster's structural decisions (environment layout, autoscaling approach, features); adapt instance types and purpose-specific configuration. |
| Same cloud, new region | Geographic expansion. Mirror the same-purpose cluster from the original region as closely as possible; adapt only what the new region requires (availability zone names, instance type availability). |
| New cloud provider | Multi-cloud addition. Mirror the same-purpose cluster across the provider boundary; adapt provider-specific attributes (instance type naming, zone format, auth model) but keep the configuration structure and scale identical. |

**Question 2 — What is the purpose of the new cluster?** Use the answer to confirm or choose the `name_prefix` following [Naming Rules → `name_prefix`](#name_prefix). Present the derived name and confirm before proceeding.

**Question 3 — Which existing cluster is the closest match to mirror?** From the Clusters table, identify the best match — same cloud provider first, then same purpose. Present your choice and confirm before reading that cluster's `.tf` files.

Then proceed with the scaffold:

1. Read environment names, `base_domain`, and framework version from the Platform section of `README.md`.
2. Derive the `<cluster_name>` from the confirmed `name_prefix`, cloud provider, and region.
3. Read the matched cluster's `<cluster_name>_cluster.tf` and `<cluster_name>_providers.tf` to extract the configuration to mirror.
4. Create `<cluster_name>_cluster.tf`, mirroring the matched cluster. Adapt only what differs: region, availability zones, and provider-specific attributes the new region or cloud requires.
5. Create `<cluster_name>_providers.tf`.
6. If the new cluster uses a cloud provider not yet declared in `versions.tf`, add it.
7. Add a row for the new cluster to the Clusters table in `README.md`.
8. Give the user the GitOps follow-up instructions. Remind them that node pools and platform features must follow in separate stages once this pipeline has successfully applied.

### Stage 2 — Add node pools

Once the user confirms the cluster pipeline applied successfully, mirror the node pools from the matched cluster: for each additional node pool, scaffold a corresponding `<new_cluster_name>_node_pool_<name>.tf` following [Scaffolding an Additional Node Pool](#scaffolding-an-additional-node-pool), adapting region-specific attributes (availability zones, instance type names). Give the GitOps follow-up instructions; remind that platform features follow in a separate stage.

### Stage 3 — Add platform features

Once the user confirms the node pool pipeline applied successfully, mirror the platform features from the matched cluster: for each feature, create the corresponding `<new_cluster_name>_feature_<feature_name>.tf` binding file following [Scaffolding a Platform Feature](#scaffolding-a-platform-feature). The `modules/<feature_name>/` directory already exists — only the cluster-binding file is new. Give the GitOps follow-up instructions.

### Multi-cloud and multi-region notes

When the new cluster uses a cloud provider not already in the repository, `versions.tf` must declare it. Each cluster's pair of files is completely independent — there are no cross-cluster dependencies in Kubestack. The `modules/<feature_name>/` directories are shared across all clusters; no changes to them are needed when adding a cluster-binding file for an existing feature.

---

## Scaffolding an Additional Node Pool

Every cluster includes a default node pool configured inside the cluster module; additional node pools are defined in separate files named `<cluster_name>_node_pool_<pool_name>.tf`, e.g. `eks_gc0_eu-west-1_node_pool_gpu.tf`. The pool name reflects the purpose: `extra`, `gpu`, `spot`, `arm`, etc. Pass cluster identity via the `cluster` and `cluster_metadata` inputs, taken from the cluster module's outputs — never hardcode the cluster name string.

Required values: pool name (short identifier reflecting purpose), instance/node type (provider-specific VM size), `min_size`/`max_size` or equivalent (autoscaling bounds), zones (within the cluster's region), and taints and labels if the pool is dedicated to a workload type.

### EKS node pool

```
module "eks_gc0_eu-west-1_node_pool_extra" {
  providers = {
    aws = aws.eks_gc0_eu-west-1
  }

  source = "github.com/kbst/terraform-kubestack//aws/cluster/node-pool?ref=<version>"

  cluster          = module.eks_gc0_eu-west-1.cluster
  cluster_metadata = module.eks_gc0_eu-west-1.current_metadata

  configuration = {
    apps = {
      name = "extra"

      instance_types   = ["t3a.xlarge"]
      min_size         = 3
      max_size         = 9
      desired_capacity = 3

      availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
    }

    ops = {}
  }
}
```

### GKE node pool

```
module "gke_gc0_europe-west1_node_pool_extra" {
  providers = {
    google = google.gke_gc0_europe-west1
  }

  source = "github.com/kbst/terraform-kubestack//google/cluster/node-pool?ref=<version>"

  cluster          = module.gke_gc0_europe-west1.cluster
  cluster_metadata = module.gke_gc0_europe-west1.current_metadata

  configuration = {
    apps = {
      name = "extra"

      project_id = module.gke_gc0_europe-west1.current_config["project_id"]
      location   = module.gke_gc0_europe-west1.current_config["region"]

      machine_type       = "e2-standard-4"
      initial_node_count = 1
      min_node_count     = 1
      max_node_count     = 3
    }

    ops = {}
  }
}
```

Always pull `project_id` and `location` from `module.<cluster_name>.current_config`.

### AKS node pool

```
module "aks_gc0_westeurope_node_pool_extra" {
  providers = {
    azurerm = azurerm.aks_gc0_westeurope
  }

  source = "github.com/kbst/terraform-kubestack//azurerm/cluster/node-pool?ref=<version>"

  cluster = module.aks_gc0_westeurope.cluster

  configuration = {
    apps = {
      node_pool_name = "extra"

      vm_size   = "Standard_D4s_v3"
      min_count = 3
      max_count = 9

      availability_zones = ["1", "2", "3"]
    }

    ops = {}
  }
}
```

AKS node pools identify their cluster via the `cluster` input alone. Azure zone numbers are strings. The node count is owned by the autoscaler — configure only `min_count` and `max_count`.

### Scaleway node pool

```
module "scw_gc0_fr-par_node_pool_extra" {
  providers = {
    scaleway = scaleway.scw_gc0_fr-par
  }

  source = "github.com/kbst/terraform-kubestack//scaleway/cluster/node-pool?ref=<version>"

  cluster          = module.scw_gc0_fr-par.cluster
  cluster_metadata = module.scw_gc0_fr-par.current_metadata

  configuration = {
    apps = {
      name = "extra"

      node_type = "PRO2-S"
      min_size  = 3
      max_size  = 9

      zones = ["fr-par-1", "fr-par-2", "fr-par-3"]
    }

    ops = {}
  }
}
```

One node pool is created per zone listed in `zones`.

### Taints and labels

Use taints when the pool is dedicated to a specific workload type (GPU, spot, machine-learning); use labels for soft scheduling preferences. EKS example:

```
apps = {
  name             = "gpu"
  instance_types   = ["g4dn.xlarge"]
  min_size         = 0
  max_size         = 4
  desired_capacity = 0

  taints = [
    {
      key    = "nvidia.com/gpu"
      value  = "true"
      effect = "NO_SCHEDULE"
    }
  ]

  labels = {
    "nvidia.com/gpu" = "true"
  }
}
```

---

## Removing a Node Pool

1. Delete `<cluster_name>_node_pool_<name>.tf`.
2. Remove the corresponding row from the Node Pools table in `README.md`.

Give the user the GitOps follow-up instructions. The Terraform plan on the feature branch will show the node pool resources being destroyed — the expected signal for the reviewer to approve the deletion.

---

## Scaffolding a Platform Feature

A platform feature is a set of Kubernetes resources applied to one or more clusters. Each feature is a local Terraform module under `modules/<feature_name>/` that wraps the Kubestack `kustomization/overlay` framework module. A separate root-level file per cluster calls that local module and wires in the cluster's `kustomization` provider alias.

`<feature_name>` is a short snake_case identifier, e.g. `gateway_api`, `service_mesh`, `cert_manager` — used as the module directory name, the local module name, and the suffix of the cluster-binding file.

### Files

| File | Purpose |
|---|---|
| `modules/<feature_name>/README.md` | Upstream source, version, and regeneration command — the source of truth for updates |
| `modules/<feature_name>/manifests/upstream.yaml` | Rendered Kubernetes resources for this feature, separated by `---` |
| `modules/<feature_name>/versions.tf` | `required_providers` for the module — pins the correct local provider name `kbst/kustomization` and prevents an invalid fallback to `hashicorp/kustomization` |
| `modules/<feature_name>/main.tf` | Calls the `kustomization/overlay` framework module |
| `modules/<feature_name>/variables.tf` | Only when the module has input variables |
| `modules/<feature_name>/outputs.tf` | Only when the module exposes output values |
| `modules/<feature_name>/values.yaml` | Only when upstream is a Helm chart — Helm values configured for the most critical environment |
| `<cluster_name>_feature_<feature_name>.tf` | One per target cluster — calls the feature module with the cluster's `kustomization` provider alias |

### Step 1 — Discover the upstream source

Before writing any files, determine the best upstream source for the requested feature. Do **not** open with a question — lead with a concrete recommendation.

#### 1a — Research known sources

Using your training knowledge, look up the feature by name across these registries and distribution channels (preference order):

| Source | What it hosts | Prefer when |
|---|---|---|
| [Artifact Hub](https://artifacthub.io) | Helm charts, plain YAML operators, OLM bundles, and more | Always check here first — it indexes the canonical chart/manifest for almost every popular project |
| GitHub Releases | Versioned plain YAML install manifests | Project publishes a single `install.yaml` or `deploy.yaml` as a release asset |
| OCI Helm registries (`oci://ghcr.io/…`, `oci://registry-1.docker.io/…`) | Immutable, digest-pinned Helm charts | Project publishes to GHCR or Docker Hub OCI; prefer over a traditional Helm repo |
| Traditional Helm repos (`https://charts.<project>.io/`) | Helm charts via `helm repo add` | No OCI distribution exists |
| Operator Hub / OLM | Kubernetes operators | Feature is operator-based and targets OpenShift or OLM-enabled clusters |

Known sources to draw from (non-exhaustive; check Artifact Hub for anything not listed — each project is also at `https://artifacthub.io/packages/helm/<org>/<chart>`):

| Feature | Source(s) |
|---|---|
| cert-manager | chart `oci://quay.io/jetstack/cert-manager`; YAML `<releases>/cert-manager.yaml` |
| Prometheus / kube-prometheus-stack | chart `oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack` |
| Argo CD | YAML `https://raw.githubusercontent.com/argoproj/argo-cd/<version>/manifests/install.yaml`; chart via `https://argoproj.github.io/argo-helm` |
| Flux CD | YAML via `flux install --export`; chart `oci://ghcr.io/fluxcd-community/charts/flux2` |
| Crossplane | chart `oci://xpkg.upbound.io/crossplane-contrib/chart/crossplane` |
| External Secrets Operator | chart `oci://ghcr.io/external-secrets/charts/external-secrets` |
| Sealed Secrets | chart `oci://ghcr.io/bitnami-labs/sealed-secrets-chart`; YAML `<releases>/controller.yaml` |
| Velero | chart via `https://vmware-tanzu.github.io/helm-charts` (`velero/velero`) |
| Karpenter | chart `oci://public.ecr.aws/karpenter/karpenter` |
| Metrics Server | chart `https://kubernetes-sigs.github.io/metrics-server/`; YAML `<releases>/components.yaml` |
| Gateway API (CRDs) | YAML `https://github.com/kubernetes-sigs/gateway-api/releases/download/<version>/standard-install.yaml` |

#### 1b — Present a recommendation and get confirmation

Present to the user: (1) the canonical upstream source — Artifact Hub URL if one exists, plus the actual chart or manifest URL; (2) the latest stable version you are aware of, noting your training data has a cutoff and the user should verify; (3) whether it is a Helm chart or plain YAML, and the recommended format (OCI Helm > traditional Helm repo > plain YAML, all else being equal); (4) a brief reason, e.g. "OCI distribution is available and preferred because chart versions are immutable". Example:

> I found **cert-manager** on Artifact Hub: https://artifacthub.io/packages/helm/cert-manager/cert-manager
>
> Recommended source: Helm chart via OCI at `oci://quay.io/jetstack/cert-manager`, latest stable version `v1.16.2` (verify at https://github.com/cert-manager/cert-manager/releases).
> OCI is preferred over the traditional Helm repo because the reference is immutable and version-pinned by digest.
>
> Shall I proceed with this source, or would you like to use a different one?

If no matching source could be found, or the user wants a different source: ask for the upstream source URL, confirm whether it is a plain YAML manifest or a Helm chart, then proceed with what they supply.

#### 1c — Choose the path based on format

- Helm chart (OCI or traditional repo) → follow the [Helm chart path](#helm-chart-path).
- Plain YAML manifest → follow the [Plain YAML path](#plain-yaml-path).

### Plain YAML path

Upstream publishes a versioned YAML manifest (e.g. a GitHub release asset).

1. Fetch the upstream manifest and write its full contents into `modules/<feature_name>/manifests/upstream.yaml`.
2. Create `modules/<feature_name>/README.md` recording the upstream project name and URL, the exact URL or release tag fetched, the version installed, and how to update (fetch the new release URL, overwrite `upstream.yaml`, re-run `tofu validate`). Template:

```
# <feature_name>

## Upstream source

- Project: <upstream project URL>
- Manifest URL: <exact URL fetched>
- Version: <version>

## Updating

1. Find the new release at <upstream releases URL>.
2. Fetch the new manifest: `curl -L <new manifest URL> -o modules/<feature_name>/manifests/upstream.yaml`
3. Review the diff for breaking changes.
4. Commit `manifests/upstream.yaml` and follow the GitOps workflow.
```

### Helm chart path

Prefer the OCI registry distribution over a traditional Helm repository — OCI references are immutable and version-pinned by digest.

**1. Create `modules/<feature_name>/values.yaml`** configured for the most critical environment (the base environment recorded in `README.md`). This file is the production-faithful baseline — do not add special-casing or reduced configuration here; those belong in the per-environment overrides in `main.tf`. Populate it with the values appropriate for production scale and configuration.

**2. Render `upstream.yaml` using `helm template`.** `upstream.yaml` is a generated file — never edit it directly; to change what is rendered, edit `values.yaml` and re-run the command. Always include `--include-crds` and `--create-namespace` so the namespace, any CRDs, and all other resources are emitted into a single file — the kustomization provider handles creation order automatically. You cannot run these commands: provide the exact command to the user, ask them to run it, and confirm the output was written to `upstream.yaml` before proceeding.

OCI chart:
```
helm template <release-name> oci://<registry>/<chart> \
  --version <chart-version> \
  --values modules/<feature_name>/values.yaml \
  --namespace <namespace> \
  --create-namespace \
  --include-crds \
  > modules/<feature_name>/manifests/upstream.yaml
```

Helm repo:
```
helm repo add <repo-name> <repo-url>
helm template <release-name> <repo-name>/<chart-name> \
  --version <chart-version> \
  --values modules/<feature_name>/values.yaml \
  --namespace <namespace> \
  --create-namespace \
  --include-crds \
  > modules/<feature_name>/manifests/upstream.yaml
```

The resulting `upstream.yaml` contains the Namespace, all CRDs (if any), and every other resource the chart installs — intentionally a complete, self-contained description of what the feature installs.

**3. Create `modules/<feature_name>/README.md`** recording the upstream project name and URL, the chart source (OCI reference or repo URL and chart name), the chart version installed, the exact `helm template` command used to regenerate `upstream.yaml`, and how to update. Template:

```
# <feature_name>

## Upstream source

- Project: <upstream project URL>
- Chart: oci://<registry>/<chart>  (or <repo-name>/<chart-name> from <repo-url>)
- Version: <chart-version>

## Regenerating upstream.yaml

Run the following command from the repository root, then commit the result:

    helm template <release-name> oci://<registry>/<chart> \
      --version <chart-version> \
      --values modules/<feature_name>/values.yaml \
      --namespace <namespace> \
      --create-namespace \
      --include-crds \
      > modules/<feature_name>/manifests/upstream.yaml

## Updating

1. Find the new chart version at <upstream releases URL>.
2. Update the `--version` flag in the command above.
3. Re-run the command, review the diff for breaking changes.
4. Update the Version line in this file.
5. Commit `manifests/upstream.yaml` and `README.md` and follow the GitOps workflow.
```

### kustomization/overlay configuration attributes reference

Every key inside a `configuration` environment block maps directly to a Kustomize attribute exposed by the `kustomization/overlay` module. All attributes are optional and default to `null` unless noted; use the full set below when writing or patching `main.tf`.

| Attribute | Purpose |
|---|---|
| `resources` (required in the base environment) | List of paths to YAML files or Kustomization directories to deploy. Always reference `upstream.yaml` via `${path.module}` so the path resolves correctly regardless of where OpenTofu is invoked from. |
| `namespace` | Sets `metadata.namespace` on every resource that does not already have one — use when the chart does not set namespaces on all resources, or to force all resources into a single namespace. |
| `common_labels` | Adds or overrides labels on every resource and every pod template, e.g. `{ "env" = terraform.workspace }`. |
| `common_annotations` | Adds or overrides annotations on every resource. |
| `name_prefix` / `name_suffix` | Prepends or appends a string to the `metadata.name` of every resource. |
| `images` | Patches image names, tags, or digests on pod specs without touching the YAML: `[{ name = <container name in pod.spec.containers>, new_name = ..., new_tag = ..., digest = "sha256:..." }]` (`digest` replaces `new_tag` when set). |
| `replicas` | Sets the replica count on Deployments, StatefulSets, or similar by name: `[{ name = ..., count = ... }]`. Prefer this over a JSON patch for simple replica changes. |
| `patches` | Strategic merge or RFC 6902 JSON patches. Use `patch = <<-EOF ... EOF` for inline patches or `path = "${path.module}/manifests/patch.yaml"` for file-based; `target` (with `group`, `version`, `kind`, `name`, `namespace`, `label_selector`, `annotation_selector`) narrows which resources the patch applies to. |
| `config_map_generator` | Generates ConfigMaps: `[{ name, namespace, behavior ("create"\|"replace"\|"merge"), literals = ["KEY=VALUE"], envs = [...], files = [...], options = { labels, annotations, disable_name_suffix_hash } }]`. |
| `secret_generator` | Same shape as `config_map_generator` plus `type`. Do **not** inline sensitive values — use literals only for non-secret bootstrapping data, or reference env files that are never committed. |
| `generator_options` | Global `{ labels, annotations, disable_name_suffix_hash }` applied to all `config_map_generator` and `secret_generator` entries; per-generator `options` take precedence where both are set. |
| Low-level pass-throughs | `components`, `crds`, `generators`, `transformers`, `vars` — lists of paths/objects mapping to Kustomize internals; rarely needed, include only when specifically required. |

### Environment patching philosophy

`values.yaml` is configured for the most critical environment and must not be diluted. Overrides for less critical environments (`ops`, and `apps` in a three-environment layout) are applied through the `configuration` map in `main.tf` using the attributes above — never through additional values files or conditional logic in `values.yaml`. This keeps the base honest: what is committed in `values.yaml` and `upstream.yaml` is exactly what runs in production, and lower-environment differences are explicit and reviewable in `main.tf`, reducing the risk of incidents in the critical environment.

Example — one replica in `ops` via `replicas` (use `images` for a debug image tag in `ops`, `common_labels` to label all resources with `terraform.workspace`, or `patches` when `replicas` is not sufficient):

```
configuration = {
  apps = {
    resources = [
      "${path.module}/manifests/upstream.yaml",
    ]
  }

  ops = {
    replicas = [
      {
        name  = "<deployment-name>"
        count = 1
      }
    ]
  }
}
```

### Step 2 — Create the Terraform module files

**`modules/<feature_name>/versions.tf`**

```
terraform {
  required_providers {
    kustomization = {
      source = "kbst/kustomization"
    }
  }
}
```

**`modules/<feature_name>/main.tf`**

```
module "<feature_name>" {
  providers = {
    kustomization = kustomization
  }

  source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=<version>"

  configuration = {
    apps = {
      resources = [
        "${path.module}/manifests/upstream.yaml",
      ]
    }

    ops = {}
  }
}
```

`${path.module}` always resolves to the directory of `main.tf`. Use the same `?ref=<version>` value as all other modules in the repository — read it from the Configuration table in `README.md`. Add per-environment `patches` to the `configuration` map as needed, following the environment patching philosophy above.

### Step 3 — Create the cluster binding

Create one file per cluster this feature is deployed to, **`<cluster_name>_feature_<feature_name>.tf`**; each cluster uses its own `kustomization` alias:

```
module "<cluster_name>_feature_<feature_name>" {
  providers = {
    kustomization = kustomization.<cluster_name>
  }

  source = "./modules/<feature_name>"
}
```

### Checklist

1. Choose a snake_case `<feature_name>`.
2. Research the upstream source (Artifact Hub first); present the recommendation and get confirmation before writing any files. Fall back to asking the user only if no known source is found or the user prefers a different one.
3. Create `modules/<feature_name>/README.md` (source, version, regeneration command), `manifests/upstream.yaml`, `versions.tf`, and `main.tf` with appropriate per-environment patches; `values.yaml` only for Helm charts; `variables.tf`/`outputs.tf` only if the module needs them.
4. For each target cluster, create `<cluster_name>_feature_<feature_name>.tf`.
5. Add a row for the feature to the Platform Features table in `README.md`.
6. Run `tofu validate` and give the user the GitOps follow-up instructions.

---

## Migrating a Catalog Module to a Platform Feature

The Kubestack catalog is deprecated. Migrating an existing catalog module (a module block with `source = "kbst.xyz/catalog/..."`) to a platform feature module must not destroy and recreate the feature's resources: state is preserved with a `moved` block, resource identity by keeping the same namespace. If multiple catalog modules are being migrated, migrate them one at a time — one feature branch and pull request each.

### Step 1 — Read the existing catalog module block

For every cluster the catalog module is deployed to, read the module block and record:

1. The exact module block address, e.g. `module.eks_gc0_eu-west-1_service_nginx` — the `moved` block's `from` address must match it exactly.
2. The `configuration` map, including all per-environment attributes — these carry over unchanged.
3. Any attribute that affects resource identity: `namespace`, `name_prefix`, `name_suffix` — these must be carried over unchanged.

### Step 2 — Scaffold the feature module

Follow [Scaffolding a Platform Feature](#scaffolding-a-platform-feature) steps 1 and 2 for the same feature, with one constraint: the upstream resources must be rendered into the same namespace the catalog module used — pass it in the `helm template --namespace` flag (or record it for the plain YAML path). If the catalog module's `configuration` set `namespace`, `name_prefix`, `name_suffix`, or similar attributes, apply the same values in the `configuration` map in `modules/<feature_name>/main.tf`, together with all other carried-over attributes.

Resource identity is namespace and name — kustomization resources are keyed by group, kind, namespace, and name. If the rendered `upstream.yaml` puts a resource in a different namespace or under a different name than the catalog module did, Terraform must destroy and recreate it; the `moved` block cannot prevent that.

### Step 3 — Replace the catalog module block with a binding and a moved block

In each cluster's binding file, call the new feature module and add a `moved` block adopting the catalog module's state:

**`<cluster_name>_feature_<feature_name>.tf`**

```
module "<cluster_name>_feature_<feature_name>" {
  providers = {
    kustomization = kustomization.<cluster_name>
  }

  source = "./modules/<feature_name>"
}

moved {
  from = module.<cluster_name>_service_<feature_name>
  to   = module.<cluster_name>_feature_<feature_name>
}
```

Use the actual catalog module block address recorded in Step 1 as the `from` value — do not assume the `service` naming. Both catalog modules and platform feature modules manage the same `kustomization_resource` resources, so the module-level move adopts the existing state one to one. Then delete the old catalog module block file for that cluster. Keep the `moved` block in the binding file — it is harmless after the state has been updated and documents the migration.

### Step 4 — Verify and hand off

1. Run `tofu validate`.
2. Update or add the feature's row in the Platform Features table in `README.md`.
3. Give the user the GitOps follow-up instructions and tell them what to look for in the plan: the module move, and no destroy and create pairs for the feature's resources. In-place updates are expected if the rendered upstream version differs from the catalog module version. Destroy and create pairs mean a resource changed namespace or name — fix the `values.yaml`, the `namespace` attribute, or the render command before the user applies.

---

## Updating a Platform Feature

When the user asks to update a platform feature (one feature by name, or all features), read `modules/<feature_name>/README.md` for each feature being updated — the source of truth recording where `upstream.yaml` came from and the exact command needed to regenerate it. The same procedure covers both formats:

1. Read the module README to find the upstream source (releases URL or chart source), the current version, and the exact fetch or `helm template` command.
2. Identify the new version to install. If the user has not specified one, ask them or check the upstream releases page together.
3. Provide the user with the updated command (new version URL or new `--version` flag) and ask them to run it to overwrite `upstream.yaml`.
4. Once the user confirms `upstream.yaml` has been updated, review the diff together for breaking changes. For Helm charts, update `values.yaml` or `main.tf` patches if the new version requires it.
5. Update the version recorded in the module README.
6. Run `tofu validate` to verify syntax.
7. Give the user the GitOps follow-up instructions.

### Updating all platform features

Read `modules/*/README.md` for every feature module and apply the procedure per feature. Each feature update is a separate task — do not batch multiple feature updates into a single commit, as this makes the Terraform plan harder to review and a single breaking change harder to isolate.

---

## Removing a Platform Feature

A platform feature may be removed from a single cluster or from all clusters.

1. Delete the `<cluster_name>_feature_<feature_name>.tf` file for each cluster the feature is being removed from, and remove the corresponding rows from the Platform Features table in `README.md`.
2. Check whether any binding files for this feature remain in the repository. **If binding files remain**, other clusters still use this feature — keep `modules/<feature_name>/` in place; your work ends here. **If no binding files remain**, the feature module is unused — ask the user: *"No clusters reference this feature any more. Should I delete the `modules/<feature_name>/` directory as well, or keep it for future use?"* Delete the directory only if the user confirms; otherwise leave it in place.

Give the user the GitOps follow-up instructions. The Terraform plan on the feature branch will show the feature resources being destroyed — the expected signal for the reviewer to approve the deletion.

---

## Removing a Cluster

Three separate stages, each an independent unit of work: make the file changes, hand off with the GitOps follow-up instructions, and wait for the user to confirm the pipeline has successfully applied before beginning the next stage. This minimises disruption and avoids interdependencies during shutdown that the Terraform dependency graph does not track.

**Stage 1 — Remove all platform features.** For each platform feature bound to this cluster, follow the full [Removing a Platform Feature](#removing-a-platform-feature) process, including the module usage check — a feature module must be deleted if this cluster was its last binding, kept if other clusters still reference it. The plan will show the platform feature resources being destroyed — the expected signal for the reviewer to approve.

**Stage 2 — Remove all additional node pools.** Delete every `<cluster_name>_node_pool_<name>.tf` file for this cluster and remove the corresponding rows from the Node Pools table in `README.md`. The plan will show the node pool resources being destroyed — the expected signal for the reviewer to approve.

**Stage 3 — Remove the cluster and its providers.** Delete `<cluster_name>_cluster.tf`, `<cluster_name>_providers.tf`, and the cluster's row from the Clusters table in `README.md`. The plan will show all remaining cluster infrastructure being destroyed — the expected signal for the reviewer to approve.

---

## GitOps Workflow

All infrastructure changes follow this four-step process. Your role is to produce correct Terraform files and hand off to the user. Applying changes is always the responsibility of the GitOps pipeline — a human must approve before anything executes against real infrastructure.

1. **Change** — Make changes in a feature branch. Push the branch. The pipeline runs `tofu plan` against the `ops` workspace.
2. **Review** — A peer reviews the diff and the plan. Make additional commits if changes are requested.
3. **Merge** — Merge to main. The pipeline applies to `ops`, then plans against `apps`.
4. **Promote** — Tag the merge commit with an `apps-deploy-*` tag. The pipeline applies to `apps`.

---

## After Every Task

At the end of every task that writes or deletes files, give the user the instructions below. Adapt the branch name and file list to what was actually changed. If the task was a framework version update, include the lock file refresh step (marked below) between staging and committing.

---

> The Terraform files are ready. Here is what to do next to get these changes applied through the GitOps pipeline.
>
> **1. Create a feature branch and stage the changes:**
> ```
> git checkout -b <descriptive-branch-name>
> git add <list of changed files>
> ```
>
> *(Framework version update only)* **Refresh the provider lock file** before committing:
> ```
> tofu init -upgrade
> git add .terraform.lock.hcl
> ```
> This locks the provider versions required by the new framework release. Include the updated lock file in the same commit.
>
> **2. Commit and push the branch:**
> ```
> git commit -m "<short description of what changed>"
> git push -u origin <descriptive-branch-name>
> ```
> The pipeline will now run `tofu plan` against the `ops` workspace. Review the plan output before proceeding.
>
> **3. Open a pull request / merge request** and have a peer review both the file diff and the Terraform plan.
>
> **4. Merge to main** once approved. The pipeline applies to `ops`, then plans against `apps`.
>
> **5. Promote to `apps`** by tagging the merge commit:
> ```
> git tag apps-deploy-$(date +%Y%m%d%H%M%S)
> git push origin --tags
> ```
> The pipeline will apply the changes to `apps`.