Skip to main content

GPUs

GPU nodes are provisioned on demand. Ask for a GPU and the platform launches a node to run your pod, then reclaims that node once nothing needs it. Nothing sits pre-allocated, so the first GPU pod on an idle cluster waits a few minutes for a node to boot and its drivers to come up — that early Pending is normal.

Request a GPU​

Ask for GPUs the way you ask for CPU and memory: a request for nvidia.com/gpu. Beyond the request itself, a GPU pod needs two things:

  • runtimeClassName: nvidia — nvidia isn't the default container runtime. Without it the pod schedules but can't see the GPU.
  • A toleration for the GPU taint. GPU nodes are tainted nvidia.com/gpu so ordinary work stays off expensive hardware. A pod without the toleration is excluded from every GPU node and stays Pending.
runtimeClassName: nvidia
tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

LLMKube sets both for you — declare an InferenceService and the operator builds the pod spec. If you are writing the manifest yourself, include them.

Platform Applications

A Platform Application can express the GPU request (resources.gpu) and node selection (workloadSelectors), but the spec has no field for runtimeClassName or tolerations. Talk to the platform team before designing a Platform Application around a GPU, rather than discovering the gap at deploy time.

Exclusive and shared GPUs​

A GPU workload normally gets exclusive use of a GPU — the right choice for latency-sensitive serving: predictable latency and the full VRAM budget. Where your platform team has enabled it, a time-sliced pool lets several workloads share one physical GPU, for dev or experimentation. Time-slicing gives no memory or compute isolation between co-tenants, so keep latency-sensitive work on exclusive GPUs and reserve sharing for low-stakes, bin-packed jobs.

Choose between them with a nodeSelector:

PoolnodeSelectorWhat you get
Exclusivep6m.dev/node-type: gpuOne workload per physical GPU
Sharedp6m.dev/node-type: gpu-sharedSeveral workloads per physical GPU — how many is a platform setting, commonly 4
Pin the pool you want

Where both pools exist, exclusive is not automatic. They carry the same GPU taint and neither requires a selector, so a workload that sets none can land on whichever GPU node the scheduler finds — including a shared one, where it gets a slice rather than a whole GPU for the same request.

spec:
  nodeSelector:
    p6m.dev/node-type: gpu          # or gpu-shared to opt into sharing

A Platform Application expresses the same thing through workloadSelectors.

Request one GPU per pod on the shared pool and scale with replicas. A larger request buys nothing a second pod would not, since co-tenants get no isolation from each other anyway.

A burst of shared-pool pods packs onto one node. Node provisioning happens before the GPU is split, so the platform tells the scheduler how many slices a node will end up advertising (a Karpenter NodeOverlay). Without that, ten pods asking for one GPU each would justify ten nodes — the exact spend sharing exists to avoid. This is a platform-side setting and needs a recent Karpenter Azure provider; on a cluster that predates it, a cold burst briefly provisions extra nodes that are reclaimed once the pods pack onto the first one.

When a GPU pod stays Pending​

A few minutes is normal while a node boots. Past that, work down this list:

  • No GPU quota in the cloud subscription. Nodes never launch, so the pod waits forever. Quota is per region and per GPU family, and raising it is a cloud-provider request — ask the platform team.
  • A nodeSelector that matches nothing. GPU nodes are labeled p6m.dev/node-type; a bare node-type key, or a value other than gpu / gpu-shared, matches no node and the scheduler will never place the pod.
  • gpu-shared on a cluster without the shared pool. The pool is opt-in per cluster. If your platform team hasn't enabled it, that selector matches nothing.
  • No toleration for the GPU taint. The pod is schedulable in principle but excluded from every GPU node.
  • Still nothing? If no node appears at all and quota is fine, ask the platform team — the GPU pools and their provisioning behavior are platform-managed.

For the general case — pods pending for reasons that have nothing to do with GPUs — see Pod Pending.