EKS Kubernetes Upgrade Runbook
eks-kube-upgrade-rollover.sh codifies Steps 2-3 and Verification below — Karpenter restart, cordon/drain of old-version nodes (including PDB-stall remediation prompts), and a post-rollover check. Defaults to dry-run; requires --apply plus an interactive confirmation to mutate a cluster. Requires kubectl and jq.
./eks-kube-upgrade-rollover.sh --target-version 1.32 # dry-run
./eks-kube-upgrade-rollover.sh --target-version 1.32 --apply # execute
Step 1: Bump the version in VersionConfig
Kube version for a cluster is set via spec.kubernetes.version on the VersionConfig it (or an ancestor it extends via spec.extendRef) resolves to, in the platform-versions repo.
Like any Kubernetes upgrade, you cannot skip minor versions — 1.31 → 1.35 must go 1.31 → 1.32 → 1.33 → 1.34 → 1.35, one PR/rollout per step.
If other clusters extend the same parent VersionConfig and don't set their own version, they inherit whatever you set on the parent. Pin siblings you don't want to move yet with an explicit version override before merging.
Open a PR bumping spec.kubernetes.version by one minor version, get it reviewed, and merge.
platform-versions is not customer-accessible — this step is not yet self-service. Contact Ybor support to request the version bump.
Once the new version is live, both the FargateProfile and the Karpenter autoscaler will provision new nodes on the new kube version. Existing nodes are not destroyed automatically — that's why the manual steps below are needed.
Step 2: Manually restart Karpenter on Fargate
Fargate nodes in this setup only host the Karpenter controller pods, and Fargate nodes are never deleted on their own. To get Karpenter itself running on a node with the correct kube version, manually restart its deployment:
kubectl rollout restart deploy/karpenter --namespace karpenter
Step 3: Karpenter-managed nodes roll over on their own — usually
Karpenter-managed nodes have a 7-day TTL and roll over to the new version automatically within a week. No manual intervention is needed if you're only doing a single minor version bump and can wait out the week.
Caveat: upgrading across multiple minor versions
Because each version step requires the previous step's nodes to be fully on the new version before you can move on (see the "one minor version at a time" rule above), waiting a full week per step to upgrade e.g. 1.31 → 1.35 (4 steps) is impractical.
To skip the wait, manually cordon and drain every Karpenter-managed node that hasn't rolled over yet, forcing them to be replaced with new-version nodes immediately.
Cordon every old-version node first, then start draining. If you drain node A while node B (same old version) is still uncordoned, a pod evicted from A can get rescheduled onto B — right back on the old version.
# 1. Cordon every old-version node first
kubectl cordon <node-name-1>
kubectl cordon <node-name-2>
# ... repeat for all old-version nodes
# 2. Only then start draining
kubectl drain <node-name-1> --ignore-daemonsets --delete-emptydir-data
kubectl drain <node-name-2> --ignore-daemonsets --delete-emptydir-data
There's no need to kubectl delete node afterward — Kubernetes deletes the underlying node automatically once it's drained.
If drain hangs on a PDB
A drain can stall reporting PodDisruptionBudget violations. Since all old-version nodes are already cordoned (per above), it's safe to force the pod off by restarting its owning workload:
kubectl rollout restart deployment/<name> -n <namespace>
# or
kubectl rollout restart statefulset/<name> -n <namespace>
This schedules a new pod and only deletes the old one once the new one is ready. Because every old-version node is cordoned, the new pod can't land back on one — avoiding another drain hang.
If the app itself is failing to run (no pods are ready), rollout restart will hang forever — the new pod will never become ready, so the old pod is never deleted. In that case, delete all of that app's pods directly instead.
Repeat for every old-version Karpenter node before proceeding to the next VersionConfig bump.
Verification
kubectl get nodes -o wide— confirm all nodes report the expectedKUBELET-VERSION- Confirm no old-version Karpenter nodes remain before starting the next minor version bump