Skip to main content

GPUs

GPU nodes are provisioned on demand. Ask for a GPU and the platform launches a node to run your pod, then reclaims that node once nothing needs it. Nothing sits pre-allocated, so the first GPU pod on an idle cluster waits a few minutes for a node to boot and its drivers to come up — that early Pending is normal.

Request a GPU​

Ask for GPUs the way you ask for CPU and memory: a request for nvidia.com/gpu. Beyond the request itself, a GPU pod needs two things:

  • runtimeClassName: nvidia — nvidia isn't the default container runtime. Without it the pod schedules but can't see the GPU.
  • A toleration for the GPU taint. GPU nodes are tainted nvidia.com/gpu so ordinary work stays off expensive hardware. A pod without the toleration is excluded from every GPU node and stays Pending.
runtimeClassName: nvidia
tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

LLMKube derives the toleration for you whenever a service requests a GPU, so that one you can leave out. runtimeClassName you still set yourself on the InferenceService — it passes through to the pod and is not defaulted. If you are writing a plain manifest, include both.

Platform Applications

A Platform Application can express the GPU request (resources.gpu) and node selection (workloadSelectors), but the spec has no field for runtimeClassName or tolerations. Talk to the platform team before designing a Platform Application around a GPU, rather than discovering the gap at deploy time.

Exclusive and shared GPUs​

A GPU workload normally gets exclusive use of a GPU — the right choice for latency-sensitive serving: predictable latency and the full VRAM budget. Where your platform team has enabled it, a time-sliced pool lets several workloads share one physical GPU, for dev or experimentation. Time-slicing gives no memory or compute isolation between co-tenants, so keep latency-sensitive work on exclusive GPUs and reserve sharing for low-stakes, bin-packed jobs.

Choose between them with a nodeSelector:

PoolnodeSelectorWhat you get
Exclusivep6m.dev/node-type: gpuOne workload per physical GPU
Sharedp6m.dev/node-type: gpu-sharedSeveral workloads per physical GPU — how many is a platform setting, commonly 4
Pin the pool you want

Where both pools exist, exclusive is not automatic. They carry the same GPU taint and neither requires a selector, so a workload that sets none can land on whichever GPU node the scheduler finds — including a shared one, where it gets a slice rather than a whole GPU for the same request.

spec:
  nodeSelector:
    p6m.dev/node-type: gpu          # or gpu-shared to opt into sharing

A Platform Application expresses the same thing through workloadSelectors.

Request one GPU per pod on the shared pool and scale with replicas. A larger request buys nothing a second pod would not, since co-tenants get no isolation from each other anyway.

A burst of shared-pool pods packs onto one node. Node provisioning happens before the GPU is split, so the platform tells the scheduler how many slices a node will end up advertising (a Karpenter NodeOverlay). Without that, ten pods asking for one GPU each would justify ten nodes — the exact spend sharing exists to avoid. This is a platform-side setting and needs a recent Karpenter Azure provider; on a cluster that predates it, a cold burst briefly provisions extra nodes that are reclaimed once the pods pack onto the first one.

Drivers and CUDA​

You don't install a driver, and you don't choose one. Every GPU pool is set up with a specific NVIDIA driver version by the platform team, and that version is declared on the pool — so you can read it, and you should, before picking a container image. The nvidia RuntimeClass is likewise created for you on cloud GPU nodes; on a self-managed or off-cloud cluster it comes from your own NVIDIA container-runtime install.

What this means for your workload: a container image built against a newer CUDA major version than the node's driver supports will not start. The process dies at initialization with "the NVIDIA driver on your system is too old" — the pod is scheduled and holding a GPU, so this reads as a crash-loop, not a scheduling problem. The constraint is the major version, not an exact match: a cu12x build runs on any CUDA 12.x driver. Pick an image inside the pool's major rather than reaching back many releases, which costs you upstream fixes for nothing.

Read the pool before you pick an image​

Two labels answer the question, and they are deliberately different things:

LabelWritten byMeans
p6m.dev/cuda.driver-version.majorThe platform, on the poolThe driver major this pool is declared to run — readable with no GPU node running
nvidia.com/cuda.driver-version.majorGPU Feature Discovery, on the nodeWhat is actually loaded on a node that exists right now
# what each GPU pool declares — works on an idle cluster, no node needed
kubectl get nodepools.karpenter.sh -o custom-columns=\
'POOL:.metadata.name,DRIVER:.spec.template.metadata.labels.p6m\.dev/cuda\.driver-version\.major'

# what a running GPU node actually has
kubectl get node <node> --show-labels | tr ',' '\n' | grep -i nvidia
Don't assume "the cluster's driver"

Pools on the same cluster can run different driver versions. Read the pool you are actually pinned to — see Exclusive and shared GPUs for how that pinning works. If the two labels above disagree on a node you are debugging, stop and tell your platform team: the pool's declaration is wrong, and any conclusion you draw from either number is unreliable until it is fixed.

Driver versions change, and the change replaces your node​

On cloud GPU pools the driver version is a platform-side decision. It isn't upgraded underneath a running node — the pool gets a new node at the new version and the old one is taken away, so your pod is rescheduled, not upgraded in place. Three consequences for how you build:

  • A GPU workload must survive being restarted and moved. Checkpoint long jobs; don't hold state only in the pod or on its node.

  • The move only happens if your pod can go somewhere. Kubernetes will not take the old node away while a pod on it has nowhere else to run — which, on a small GPU pool where your pod requires that pool, means nowhere. The change then quietly does not happen: the pool is configured for the new driver, your node keeps the old one, and nothing errors. So don't assume an announced driver change reached you. Read the node:

    kubectl get node $(kubectl get pod $POD -n $NS -o jsonpath='{.spec.nodeName}') \
      --show-labels | tr ',' '\n' | grep cuda.driver-version

  • An image that works today can break on a newer driver, even though newer is normally backward-compatible. Some CUDA images run on an older driver only through forward compatibility, which the image supports for a specific list of driver branches — move to a branch outside that list and it stops working for no reason but its absence. If you pin an image, say so when your platform team announces a driver change, and re-test rather than assuming a newer driver is strictly safer.

If current images have outgrown the pool's driver, the fix is on the platform side — raise it rather than pinning an ancient image permanently. This is a driver constraint and not a GPU-model one: the same wall appears on any card behind the same driver.

On AKS this path is validated end to end — the platform GPU pool provisions a node on demand, the driver is installed on it, and the device plugin advertises nvidia.com/gpu. GPU pools on other clouds are configured differently, so confirm pool names and driver expectations with your platform team rather than carrying the AKS shape across.

What you never need to do​

Every one of these is handled for you, and doing it yourself makes things worse rather than better:

  • Install or upgrade a driver, in an init container, a privileged pod, or a DaemonSet of your own. GPU nodes get their driver from the platform; a second installer fights the first.
  • Label or annotate a node. The driver labels above are re-stamped from the pool every time a node registers, so a hand-edit lasts until the next node and silences the platform's drift audit in the meantime.
  • Add a node affinity on a driver label. Pin the pool with p6m.dev/node-type instead. An affinity on a driver label that no pool declares makes the autoscaler create no node at all, and the pod pends forever with nothing to explain it.
  • Ask for nvidia.com/gpu in the limits but not the requests (or a fractional GPU). GPUs are integer, non-overcommittable resources; request whole ones.
  • Set NVIDIA_DISABLE_REQUIRE to get past a driver check. It removes the check, not the incompatibility — the container starts and then fails somewhere less legible.
  • Mount /dev/nvidia* or the host's driver libraries yourself. runtimeClassName: nvidia does that, correctly, for the driver that is actually there.

When a GPU pod stays Pending​

Several minutes is normal, and longer than you would expect. The node has to launch, and on pools where the platform installs the driver rather than baking it into the image, the node then spends several more minutes building the driver before it advertises nvidia.com/gpu at all — measured at over five minutes end to end. A pod that is Pending while a node exists but shows no GPU capacity is that build, not a fault. Check before concluding anything:

kubectl get nodes -l p6m.dev/node-type \
  -o custom-columns='NODE:.metadata.name,AGE:.metadata.creationTimestamp,GPU:.status.capacity.nvidia\.com/gpu'

Past that, work down this list:

  • No GPU quota in the cloud subscription. Nodes never launch, so the pod waits forever. Quota is per region and per GPU family, and raising it is a cloud-provider request — ask the platform team.
  • A nodeSelector that matches nothing. GPU nodes are labeled p6m.dev/node-type; a bare node-type key, or a value other than gpu / gpu-shared, matches no node and the scheduler will never place the pod.
  • gpu-shared on a cluster without the shared pool. The pool is opt-in per cluster. If your platform team hasn't enabled it, that selector matches nothing.
  • No toleration for the GPU taint. The pod is schedulable in principle but excluded from every GPU node.
  • A node affinity on a driver label. If something added a required affinity on nvidia.com/cuda.driver-version.major or p6m.dev/cuda.driver-version.major, and no pool declares a value satisfying it, the autoscaler creates no node at all — so this looks like an autoscaler failure rather than a placement one. Pin the pool with p6m.dev/node-type instead.
  • Still nothing? If no node appears at all and quota is fine, ask the platform team — the GPU pools and their provisioning behavior are platform-managed.
Placement you didn't write

Some of the pod's placement is derived rather than authored, so the manifest you wrote is not the whole story. LLMKube adds the GPU toleration for you, and in shared mode pins the pod to p6m.dev/node-type: gpu-shared. Read the pod, not the InferenceService or Deployment, when a placement decision surprises you:

kubectl get pod $POD -n $NS -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}'
kubectl get pod $POD -n $NS -o jsonpath='{.spec.affinity}{"\n"}'

For the general case — pods pending for reasons that have nothing to do with GPUs — see Pod Pending.