Skip to main content

LLMKube (LLM Serving)

LLMKube is the platform's paved-road runtime for self-hosted LLM inference on GPU nodes. It's a Kubernetes operator: you declare a Model (where the weights come from) and an InferenceService (how to serve them), and it handles GPU scheduling, health checks, and an OpenAI-compatible API. This page takes a pilot team from "operator is installed" to "my model is answering requests."

For what an Installable / Installation does at the platform layer, see Installation. This page assumes the operator is already installed on your cluster (below).

Upstream

The platform packages the open-source defilantech/llmkube chart as the llmkube Installable, pinned to a specific version. That project owns the CRD schema, the serving runtimes (SGLang, vLLM, llama.cpp), and the model-source behavior. The platform's value-add is the version-pinned Installable, the model-cache defaults, and the per-cluster Installation wiring. CRD-field and runtime questions belong upstream — see llmkube's config/samples/ and docs/.

Pick your path​

There are several serving runtimes with different strengths. Pick by model format and what you're optimizing for — all are viable for production; this isn't a prod-vs-dev split:

vLLM (safetensors)llama.cpp (GGUF)
Strong atHigh-throughput concurrent serving (continuous batching, PagedAttention), tensor-parallel multi-GPU, broad safetensors/AWQ/GPTQ support — for latency-sensitive interactive serving, prefer SGLang (see tip below)Aggressive & flexible quantization (custom GGUF quants and KV-cache types, e.g. TurboQuant), low memory footprint, strong single-GPU / CPU / mixed serving
Weights viaOperator download from a pinned HF repo (hf://…@rev + files), or pvc:// pre-stagedOperator download from a single-file GGUF URL → SHA256 model cache
Model cache (modelCache)Used — the operator stages the repo's files into the cache, reused across restartsUsed — download-once, reused across restarts
Safetensors repos need a files list

For a multi-file safetensors repo, point source at a pinned HF repo (hf://<org>/<repo>@<rev>) and list the artifacts in spec.files (weights + config + tokenizer). The files list is what tells the operator this is a repo to stage — omit it and source is treated as a single file. See the vLLM + hf:// tab below for the full shape. pvc:// (pre-staged) and skipModelInit (vLLM pulls at startup) are the alternatives.

Choosing a runtime for interactive serving

LLMKube also ships SGLang (runtime: sglang) alongside vLLM and llama.cpp. For latency-sensitive, concurrent workloads — agentic coding, chat, anything where a user is waiting on each token — SGLang is the recommended engine: under concurrent load it keeps per-token latency low and stable. Choose vLLM when you're optimizing aggregate throughput (total tokens/sec across many requests) and can tolerate a looser latency tail. vLLM and SGLang serve the same safetensors checkpoints, so trying the other is a one-line runtime: change. As with vLLM, pin the SGLang image to match your GPU driver (see Gotchas).

Prerequisites​

  • Operator installed. The llmkube Installable is enabled on your cluster (see Enable the operator). Confirm: kubectl -n llmkube-system get pods (controller-manager 1/1).
  • A GPU node pool on the cluster. Two pod-level requirements, and they are handled differently. You set runtimeClassName: nvidia on the InferenceService — it passes straight through to the pod and is not defaulted, so omitting it gets you a pod that schedules onto the GPU node and then fails at runtime with "no CUDA-capable device is detected." The GPU taint toleration is automatic: whenever a service requests a GPU the operator derives the toleration from the resolved GPU resource name, so no per-service tolerations block is needed. resources.gpu maps to nvidia.com/gpu, and the pod lands on one of the cluster's GPU pools: pin the one you want, and check what driver the nodes carry before choosing a runtime image, on GPUs. A cluster without GPU quota leaves serving pods Pending until quota is granted.
  • A HuggingFace token for gated models / the staging step. Create it directly — kubectl create secret generic hf-token --from-literal=HF_TOKEN=<token> -n <ns>. For anything long-lived, prefer syncing it from your cloud secret store rather than a hand-created secret — see External Secrets for the general pattern (the same Secret name is what the Model / staging step consumes).

Enable the operator​

Cluster enablement happens in the org's .platform repo. Create installations/llmkube.yaml:

apiVersion: p6m.dev/v1alpha1
kind: Installation
metadata:
  name: llmkube
spec:
  cd:
    enabled: true
    autoPromote: true
  installableRef:
    kind: Installable
    name: llmkube
    namespace: installables
  destinations:
    - clusterRef:
        name: <your-cluster-name>

Find the cluster's clusterRef.name via grep clusterRef installations/*.yaml in the same repo. Push and merge; ArgoCD reconciles the operator into llmkube-system.

Serve a model (copy-paste → running)​

vLLM image ↔ GPU driver

A vLLM or SGLang image can require a newer CUDA driver than the GPU nodes carry. If the pod crash-loops at startup with "the NVIDIA driver on your system is too old," pin spec.image to a build inside the node's CUDA major version — see Drivers and CUDA. llama.cpp/GGUF is unaffected.

Point the Model at a pinned HuggingFace safetensors repo and list the artifacts in spec.files — the operator stages them into the model cache (download-once, reused across restarts). This is the default for vLLM.

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: my-model               # FILL ME
  namespace: <your-namespace>  # FILL ME
spec:
  source: hf://<org>/<repo>@<rev>   # FILL ME — pin an IMMUTABLE ref: a commit SHA or release tag, not a moving branch like main
  format: safetensors
  files:                            # FILL ME — the repo artifacts to stage
    - model-00001-of-00002.safetensors   # weight shards
    - model-00002-of-00002.safetensors
    - model.safetensors.index.json
    - config.json                        # + config / tokenizer files
    - generation_config.json
    - tokenizer.json
    - tokenizer_config.json
    - vocab.json
    - merges.txt
  hardware:
    accelerator: cuda
    gpu: { enabled: true, count: 1, vendor: nvidia }
---
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: my-model-svc           # FILL ME
  namespace: <your-namespace>  # FILL ME
spec:
  modelRef: my-model
  replicas: 1
  runtime: vllm
  runtimeClassName: nvidia     # REQUIRED — nvidia is not the default container runtime
  # image omitted → the chart's default vLLM image (override with spec.image)
  vllmConfig:                  # vLLM tunables live HERE, not top-level
    maxModelLen: 8192          # FILL ME — fit to VRAM
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true
    # gated repos: hfTokenSecretRef: { name: hf-token, key: HF_TOKEN }
  extraArgs:
    - "--served-model-name"    # else the served id defaults to the staged weights path
    - "my-model"               #   clients then pass "my-model" as the request "model"
  endpoint: { port: 8000, type: ClusterIP }
  resources: { gpu: 1, cpu: "2", memory: "8Gi" }

The files list is required for a repo source — without it the operator treats source as a single file (and a raw hf:// URL will fail to download). The operator stages the listed files into the cache and starts vLLM against the staged directory.

Apply the Model first, then the InferenceService:

kubectl apply -f model.yaml -n <your-namespace>
kubectl apply -f inferenceservice.yaml -n <your-namespace>

Verify​

Wait for both to reach Ready:

kubectl -n <your-namespace> get model my-model -w
kubectl -n <your-namespace> get inferenceservice my-model-svc -w

Port-forward and hit the OpenAI-compatible endpoint (vLLM on 8000, llama.cpp on 8080). The model field must match the served model name (the --served-model-name you set, or the HF id):

kubectl -n <your-namespace> port-forward svc/my-model-svc 8000:8000
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "messages": [{"role": "user", "content": "What is 2+2?"}], "max_tokens": 64}'

Metrics & observability​

Each serving pod exposes Prometheus metrics on its API port at /metrics — throughput (tokens/s), per-token latency, and request load (llamacpp:* for llama.cpp, vllm:* for vLLM).

You don't need to do anything to get them scraped. The operator annotates serving pods for the platform's metrics agent itself, resolving the real API port per runtime — whatever your endpoint.port or containerPort works out to. That port resolution is the part a hand-written annotation gets wrong, because metrics live on the API container port, whose port name (http) isn't one the agent matches by name. The metrics land in the platform's metrics store, and the LLMKube Inference Grafana dashboard visualizes them next to the GPU / DCGM hardware dashboard.

Confirm the annotations are on your pod:

kubectl -n <your-namespace> get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.prometheus\.io/port}{"\n"}{end}'

spec.podAnnotations is now only for overriding what the operator emits — a value you set always wins on the same key. The one case that needs it is metrics served somewhere other than the API port:

spec:
  podAnnotations:
    prometheus.io/port: "9090"   # only if /metrics is NOT on your API port
Enabled per cluster

Operator-emitted annotations are a platform-side setting, enabled in the platform's shared catalog. If the command above shows no port, your cluster may predate it — ask your platform team rather than adding the annotations by hand, so every service on the cluster gets fixed at once.

Service-level objectives​

An InferenceService can declare a service-level objective in spec.slo. The operator hands it to the cluster's SLO tooling (Pyrra), which generates the recording rules, the error-budget arithmetic, and multi-window burn-rate alerts — none of which you write yourself. A few lines of YAML turn "is it fast enough?" into a number you can hold a decision against, and it costs no extra hardware: everything is computed from metrics already being scraped. The LLMKube SLO Grafana dashboard reads the result. If the alerting model is new to you, Google's SRE workbook chapter on alerting on SLOs is the pattern Pyrra implements.

Availability — the fraction of scrapes in which the serving pod was up:

spec:
  slo:
    objective: "99.5"          # percent, between 50 and 99.999
    window: 28d                # default

indicator defaults to availability; name defaults to <inferenceservice-name>-<indicator>. Note that objective is a string, not a number — an unquoted value is rejected.

Latency — the fraction of requests finishing under a bound. vLLM only today: llama.cpp exports no request-duration histogram, and rather than render an SLO that would read empty forever, the operator refuses and says so in a condition.

spec:
  runtime: vllm
  slo:
    indicator: latency         # availability | latency — there are no other values
    objective: "95"            # 95% of requests…
    latencyThreshold: "1.0"    # …complete within 1.0 second
    window: 28d

Read that literally: 95% of requests complete within 1.0s over 28 days. The percentile lives in objective — there is no separate p95/p99 field, and no way to express "p99 under 1s" directly. You pick the bound, and the fraction of requests that must beat it.

latencyThreshold must match a histogram bucket boundary, as a string

The threshold becomes an exact le= label match against the runtime's latency histogram, and exact means string comparison: "2" and "2.0" are different values, and only one of them exists in any given image. The formatting differs between vLLM images, so read the boundary off the image you actually run instead of copying a threshold out of a doc. A threshold matching no bucket records nothing — the SLO then reads empty forever, with no error raised anywhere.

kubectl -n <your-namespace> port-forward svc/<your-service> 8000:8000
# in another shell
curl -s http://localhost:8000/metrics \
  | grep vllm:e2e_request_latency_seconds_bucket \
  | grep -o 'le="[^"]*"' | sort -u

Check it landed​

spec.slo needs the SLO tooling installed on the cluster and the operator's SLO integration switched on — both platform-side, both per cluster. Where either is missing, the InferenceService still serves normally; only the SLO goes unrendered. The SLOReady condition says which case you are in:

kubectl -n <your-namespace> get inferenceservice <your-service> \
  -o jsonpath='{range .status.conditions[?(@.type=="SLOReady")]}{.status}{" "}{.reason}{"\n"}{end}'
SLOReady reasonWhat it means
SLOCreatedRendered. The rules are generated; burn-rate series appear once the cluster's rule evaluator has run over them.
IntegrationDisabledThe operator's SLO integration is off on this cluster — ask your platform team to enable it.
PyrraNotInstalledThe SLO tooling isn't installed on this cluster — ask your platform team.
IndicatorUnsupportedForRuntimeindicator: latency on a runtime with no latency histogram (anything other than vLLM).
ReconcileFailedThe render failed; the operator logs carry the reason.

A second condition, SLODataSourceAvailable, appears once SLOReady is True, and warns when the SLO is valid but has no metrics source to read from — the case today is a service on an off-cluster node, which the cluster's metrics agent cannot scrape.

Declaring an SLO does not mean someone gets paged

The rules Pyrra generates include burn-rate alerts, but which channel or rotation they reach is a platform-side routing decision, and no part of spec.slo. Ask your platform team where your service's alerts land, so you know what a burning error budget will actually reach.

Three lifecycle behaviors worth knowing: removing spec.slo deletes the rendered SLO and stops its alerts, renaming it replaces the old one instead of leaving a duplicate behind, and deleting the InferenceService cleans up after itself.

A brand-new SLO reads alarming, then settles

The error budget is computed across the whole window (28 days by default), so the first hours are dominated by whatever happened at startup — a couple of failed scrapes during a cold start can show a deeply negative budget. It self-corrects as data accumulates. Judge a fresh SLO by its burn-rate panels, which reflect the recent past; budget-remaining is only meaningful once the window holds real data.

GPU sharing​

An InferenceService normally gets exclusive use of a GPU — the right choice for latency-sensitive serving. Where your platform team has enabled a time-sliced pool, several workloads can instead share one physical GPU, which suits dev and experimentation but gives co-tenants no memory or compute isolation.

Set spec.nodeSelector to pick a pool — p6m.dev/node-type: gpu for exclusive, p6m.dev/node-type: gpu-shared to opt into sharing. Where both pools exist, exclusive is not automatic: a service that sets no selector can land on either. See GPUs for the full picture, including what happens to a burst of shared-pool services.

Two config layers​

Keep two kinds of setting separate — it decides where a value belongs:

  • Layer 1 — platform-wide (the Installable). Operator version, default vLLM image, modelCache mode/size, DCGM / observability. Platform-managed defaults that every deployment on the cluster inherits — not touched per model.
  • Layer 2 — per-model serving knobs (this template). Everything under vllmConfig (maxModelLen, gpuMemoryUtilization, enablePrefixCaching, speculative, dtype, kvCacheDtype), plus extraArgs, runtimeClassName, resources. These vary per model/workload — a pilot picks them per InferenceService.

Rule of thumb: same for every model on the platform → Layer 1; varies per model → Layer 2 (here).

Starter profiles​

Two tuned Layer-2 starting points — copy the vllmConfig block matching your workload. These are sensible starting values; treat the magnitudes as directional and re-measure on your hardware — the shape transfers.

Chatbot — many short requests sharing a system prompt:

  vllmConfig:
    maxModelLen: 8192
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true     # ON — shared system prompts reuse KV → lower TTFT
    enableChunkedPrefill: true    # on by default in vLLM V1; keeps prefill from stalling decode

Coding agent — longer contexts, tool-call / multi-turn traffic:

  vllmConfig:
    maxModelLen: 16384            # room for code + tool schemas
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true
    # n-gram speculative decoding helps at LOW concurrency but goes net-NEGATIVE
    # once the batch saturates — enable only for bursty / low-QPS traffic, and see
    # upstream for the exact vllmConfig.speculative shape. Measure before shipping.

Do not ship --enforce-eager in a production profile — it's a dev/iteration convenience (fast restarts) at a real throughput cost; leave CUDA-graph capture on for serving.

Gotchas​

  • runtimeClassName: nvidia is mandatory — nvidia isn't the default container runtime. Omit it and the pod schedules but can't see the GPU.
  • No nodeSelector means no guarantee of an exclusive GPU. Where a shared pool exists, a service without a pool selector can land on it and quietly get a slice — same resources.gpu: 1, a fraction of the card. Symptom: unexplained latency, or less VRAM than the GPU should have. See GPUs.
  • The model cache backs both operator-download paths — GGUF (single file) and vLLM hf:// (multi-file staging) both land in it and are reused across restarts. Only pvc:// (pre-staged) and skipModelInit (ephemeral, re-downloads every restart) bypass the cache.
  • skipModelInit re-downloads the full weights on every restart into ephemeral storage. Prefer pvc:// for anything you'll restart.
  • The default StorageClass differs by environment (e.g. AKS managed-csi vs. a local local-path). The perService cache logic is the same across them (RWO, WaitForFirstConsumer); only the class name and backing differ.
  • Pending GGUF-cache PVC is normal — perService + WaitForFirstConsumer means it binds only once the serving pod schedules onto a GPU node.
  • VRAM sizing is soft. gpuMemoryUtilization is a fraction of total VRAM, and it's not a hard cap — vLLM can overshoot the target, and CUDA-graph capture allocates after the startup memory check, so a check that passes can still OOM. Check nvidia-smi for other GPU processes first: anything already resident eats into your budget. A clean GPU tolerates a high fraction (e.g. 0.90); leave more headroom when the GPU has other residents. Crash-loops can leave transient un-reclaimed VRAM that cascades the next OOM.
  • --enforce-eager gives ~1-minute restarts by skipping the torch.compile / CUDA-graph capture spike — handy while iterating, at a small peak-throughput cost. Leave it off for production.
  • --served-model-name with pvc://. The default model id becomes the mount path (/model-source/<dir>); set a friendly name so clients pass a sane "model".
  • flashAttention/contextSize/jinja are llama.cpp (top-level) fields. vLLM uses its own attention backend (default FlashAttention on sm_80+) via vllmConfig.attentionBackend — don't set the top-level flashAttention for a vLLM service.
  • flashAttention doesn't always surface in the args. Setting it true on llama.cpp may not visibly reach the rendered startup command (no error, just no flash_attn line). Check the llama.cpp command before relying on it.
  • The GGUF cache has no integrity check. "Already cached, skipping download" tests file existence, not size/hash — an interrupted download leaves a truncated GGUF that gets reused (model is corrupted or incomplete). Set Model.spec.sha256 to force verification, or clear the cache PVC after a failed pull.
  • Quantize on small cards. An 8B model in fp16 won't fit ~12 GB — use AWQ/GPTQ/GGUF. For reference, a 7B AWQ model (~5 GB) fits comfortably at 0.90 on a 12–16 GB GPU.
  • The first vLLM image pull is large (several GB) — one-time per node.
  • Pin spec.image to the pool's CUDA major. A recent vLLM or SGLang image can need a newer CUDA driver than the GPU pool runs, and the runtime dies at engine init with "the NVIDIA driver on your system is too old". Pin to a recent build inside that CUDA major — a cu12x build runs on any 12.x driver, so reaching further back than that costs you upstream fixes and metrics for nothing. Read the pool's declared driver rather than assuming a cluster-wide one; Drivers and CUDA has the two labels and what they mean. llama.cpp/GGUF is unaffected.
  • Leaving spec.image unset is a moving target, not a platform guarantee. With no image, you get whatever runtime image the operator build compiles in as its default. The llmkube Installable floats on a minor range and clusters commonly auto-promote, so those defaults — and their CUDA floors — can move with no diff in any repository you watch. That is usually fine and keeps you current. If you need the runtime to be reproducible, or you have already hit a driver mismatch once, pin spec.image; that pin is the opt-out, and it is the one field that stops a silent upstream change reaching your service.
  • A driver upgrade can break a pinned image that worked yesterday. Newer drivers are normally backward-compatible, but an image that ran on an older driver via CUDA forward compatibility supports only the driver branches it enumerates. Re-test pinned services when your platform team announces a driver change — see Driver versions change.
  • Older vLLM images bind IPv6 only. Some older vLLM builds start on [::] and the IPv4 readiness probe gets connection refused, so the pod never goes Ready even though vLLM is up. Add extraArgs: ["--host", "0.0.0.0"].
  • vLLM's served model name defaults to the weights path (e.g. /models/<hash>). Either pass that exact string as the request model, or set extraArgs: ["--served-model-name", "my-model"] for a friendly name.
  • GPUs — GPU pools, exclusive vs shared, and what to check when a GPU pod stays Pending.
  • CSI Addon — RWX StorageClass source if you move the GGUF cache to shared mode.
  • Storage — StorageClass reference (RWO vs RWX, per cloud).