Skip to main content

EKS Kubernetes Upgrade Runbook


Step 1: Bump the version in VersionConfig

Kube version for a cluster is set via spec.kubernetes.version on the VersionConfig it (or an ancestor it extends via spec.extendRef) resolves to, in the platform-versions repo.

One minor version at a time

Like any Kubernetes upgrade, you cannot skip minor versions — 1.31 → 1.35 must go 1.31 → 1.32 → 1.33 → 1.34 → 1.35, one PR/rollout per step.

Check for sibling VersionConfigs

If other clusters extend the same parent VersionConfig and don't set their own version, they inherit whatever you set on the parent. Pin siblings you don't want to move yet with an explicit version override before merging.

Open a PR bumping spec.kubernetes.version by one minor version, get it reviewed, and merge.

Customer access

platform-versions is not customer-accessible — this step is not yet self-service. Contact Ybor support to request the version bump.

Once the new version is live, both the FargateProfile and the Karpenter autoscaler will provision new nodes on the new kube version. Existing nodes are not destroyed automatically — that's why the manual steps below are needed.

Step 2: Manually restart Karpenter on Fargate

Fargate nodes in this setup only host the Karpenter controller pods, and Fargate nodes are never deleted on their own. To get Karpenter itself running on a node with the correct kube version, manually restart its deployment:

kubectl rollout restart deploy/karpenter --namespace karpenter

Step 3: Karpenter-managed nodes roll over on their own — usually

Karpenter-managed nodes have a 7-day TTL and roll over to the new version automatically within a week. No manual intervention is needed if you're only doing a single minor version bump and can wait out the week.

Caveat: upgrading across multiple minor versions

Because each version step requires the previous step's nodes to be fully on the new version before you can move on (see the "one minor version at a time" rule above), waiting a full week per step to upgrade e.g. 1.31 → 1.35 (4 steps) is impractical.

To skip the wait, manually cordon, drain, and delete every Karpenter-managed node that hasn't rolled over yet, forcing them to be replaced with new-version nodes immediately:

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>

Repeat for every old-version Karpenter node before proceeding to the next VersionConfig bump.

Verification

  • kubectl get nodes -o wide — confirm all nodes report the expected KUBELET-VERSION
  • Confirm no old-version Karpenter nodes remain before starting the next minor version bump