Skip to main content

LLMKube (LLM Serving)

LLMKube is the platform's paved-road runtime for self-hosted LLM inference on GPU nodes. It's a Kubernetes operator: you declare a Model (where the weights come from) and an InferenceService (how to serve them), and it handles GPU scheduling, health checks, and an OpenAI-compatible API. This page takes a pilot team from "operator is installed" to "my model is answering requests."

For what an Installable / Installation does at the platform layer, see Installation. This page assumes the operator is already installed on your cluster (below).

Upstream

The platform packages the open-source defilantech/llmkube chart as the llmkube Installable, pinned to a specific version. That project owns the CRD schema, the serving runtimes (vLLM, llama.cpp), and the model-source behavior. The platform's value-add is the version-pinned Installable, the model-cache defaults, and the per-cluster Installation wiring. CRD-field and runtime questions belong upstream — see llmkube's config/samples/ and docs/.

Pick your path​

There are two serving runtimes with different strengths. Pick by model format and what you're optimizing for — both are viable for production; this isn't a prod-vs-dev split:

vLLM (safetensors)llama.cpp (GGUF)
Strong atHigh-concurrency serving (continuous batching, PagedAttention), tensor-parallel multi-GPU, broad safetensors/AWQ/GPTQ supportAggressive & flexible quantization (custom GGUF quants and KV-cache types, e.g. TurboQuant), low memory footprint, strong single-GPU / CPU / mixed serving
Weights viaOperator download from a pinned HF repo (hf://…@rev + files), or pvc:// pre-stagedOperator download from a single-file GGUF URL → SHA256 model cache
Model cache (modelCache)Used — the operator stages the repo's files into the cache, reused across restartsUsed — download-once, reused across restarts
Safetensors repos need a files list

For a multi-file safetensors repo, point source at a pinned HF repo (hf://<org>/<repo>@<rev>) and list the artifacts in spec.files (weights + config + tokenizer). The files list is what tells the operator this is a repo to stage — omit it and source is treated as a single file. See the vLLM + hf:// tab below for the full shape. pvc:// (pre-staged) and skipModelInit (vLLM pulls at startup) are the alternatives.

Prerequisites​

  • Operator installed. The llmkube Installable is enabled on your cluster (see Enable the operator). Confirm: kubectl -n llmkube-system get pods (controller-manager 1/1).
  • A GPU node with the nvidia RuntimeClass. Serving pods set runtimeClassName: nvidia — required, since nvidia isn't the default container runtime. The RuntimeClass is created for you on the cloud GPU node by the gpu-operator's container-toolkit; on a self-managed / off-cloud cluster it comes from your NVIDIA container-runtime install. Pods request resources.gpu, which maps to nvidia.com/gpu, and land on the cluster's Karpenter gpu NodePool. On Azure this is validated end-to-end: the platform gpu pool provisions a Standard_NC4as_T4_v3 (in-quota T4) on demand, the NVIDIA driver ships pre-installed in the Azure GPU node image (the gpu-operator detects it and stands its own driver DaemonSet down), and the device plugin advertises nvidia.com/gpu. A cluster without GPU quota leaves serving pods Pending until quota is granted.
  • A HuggingFace token for gated models / the staging step. Create it directly — kubectl create secret generic hf-token --from-literal=HF_TOKEN=<token> -n <ns>. For anything long-lived, prefer syncing it from your cloud secret store rather than a hand-created secret — see External Secrets for the general pattern (the same Secret name is what the Model / staging step consumes).

Enable the operator​

Cluster enablement happens in the org's .platform repo. Create installations/llmkube.yaml:

apiVersion: p6m.dev/v1alpha1
kind: Installation
metadata:
  name: llmkube
spec:
  cd:
    enabled: true
    autoPromote: true
  installableRef:
    kind: Installable
    name: llmkube
    namespace: installables
  destinations:
    - clusterRef:
        name: <your-cluster-name>

Find the cluster's clusterRef.name via grep clusterRef installations/*.yaml in the same repo. Push and merge; ArgoCD reconciles the operator into llmkube-system.

Serve a model (copy-paste → running)​

vLLM image ↔ GPU driver

The chart's default vLLM image can require a newer CUDA driver than your GPU nodes ship (e.g. the driver baked into the Azure GPU node image). If a vLLM pod crash-loops at startup with "the NVIDIA driver on your system is too old," pin spec.image to a build matching the node's CUDA — see Gotchas. llama.cpp/GGUF is unaffected.

Point the Model at a pinned HuggingFace safetensors repo and list the artifacts in spec.files — the operator stages them into the model cache (download-once, reused across restarts). This is the default for vLLM.

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: my-model               # FILL ME
  namespace: <your-namespace>  # FILL ME
spec:
  source: hf://<org>/<repo>@<rev>   # FILL ME — pin an IMMUTABLE ref: a commit SHA or release tag, not a moving branch like main
  format: safetensors
  files:                            # FILL ME — the repo artifacts to stage
    - model-00001-of-00002.safetensors   # weight shards
    - model-00002-of-00002.safetensors
    - model.safetensors.index.json
    - config.json                        # + config / tokenizer files
    - generation_config.json
    - tokenizer.json
    - tokenizer_config.json
    - vocab.json
    - merges.txt
  hardware:
    accelerator: cuda
    gpu: { enabled: true, count: 1, vendor: nvidia }
---
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: my-model-svc           # FILL ME
  namespace: <your-namespace>  # FILL ME
spec:
  modelRef: my-model
  replicas: 1
  runtime: vllm
  runtimeClassName: nvidia     # REQUIRED — nvidia is not the default container runtime
  # image omitted → the chart's default vLLM image (override with spec.image)
  vllmConfig:                  # vLLM tunables live HERE, not top-level
    maxModelLen: 8192          # FILL ME — fit to VRAM
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true
    # gated repos: hfTokenSecretRef: { name: hf-token, key: HF_TOKEN }
  extraArgs:
    - "--served-model-name"    # else the served id defaults to the staged weights path
    - "my-model"               #   clients then pass "my-model" as the request "model"
  endpoint: { port: 8000, type: ClusterIP }
  resources: { gpu: 1, cpu: "2", memory: "8Gi" }

The files list is required for a repo source — without it the operator treats source as a single file (and a raw hf:// URL will fail to download). The operator stages the listed files into the cache and starts vLLM against the staged directory.

Apply the Model first, then the InferenceService:

kubectl apply -f model.yaml -n <your-namespace>
kubectl apply -f inferenceservice.yaml -n <your-namespace>

Verify​

Wait for both to reach Ready:

kubectl -n <your-namespace> get model my-model -w
kubectl -n <your-namespace> get inferenceservice my-model-svc -w

Port-forward and hit the OpenAI-compatible endpoint (vLLM on 8000, llama.cpp on 8080). The model field must match the served model name (the --served-model-name you set, or the HF id):

kubectl -n <your-namespace> port-forward svc/my-model-svc 8000:8000
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "messages": [{"role": "user", "content": "What is 2+2?"}], "max_tokens": 64}'

Metrics & observability​

Each serving pod exposes Prometheus metrics on its API port at /metrics — throughput (tokens/s), per-token latency, and request load (llamacpp:* for llama.cpp, vllm:* for vLLM). To have the platform's metrics agent scrape them, add scrape annotations to the pods via the InferenceService:

spec:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8000"   # MUST equal your endpoint.port (8080 for the llama.cpp example)

Set prometheus.io/port explicitly and match it to your endpoint.port (8000 in the vLLM examples, 8080 in the llama.cpp example): the metrics live on the API container port, whose port name (http) isn't one the agent matches by name. Once scraped, the metrics land in the platform's metrics store and the LLMKube Inference Grafana dashboard visualizes them next to the GPU / DCGM hardware dashboard.

note

Emitting these annotations automatically from the operator (so no per-service block is needed) is planned. Until then, add the block above to each InferenceService you want scraped.

GPU sharing​

By default each InferenceService gets exclusive use of a GPU — the right choice for latency-sensitive serving: predictable latency and the full VRAM budget. For dev or experimentation where you want to pack several small workloads onto one GPU, the platform offers an opt-in time-sliced GPU pool. Time-slicing gives no memory or compute isolation between co-tenants, so keep latency-sensitive serving on exclusive GPUs and reserve sharing for low-stakes, bin-packed work.

Two config layers​

Keep two kinds of setting separate — it decides where a value belongs:

  • Layer 1 — platform-wide (the Installable). Operator version, default vLLM image, modelCache mode/size, DCGM / observability. Platform-managed defaults that every deployment on the cluster inherits — not touched per model.
  • Layer 2 — per-model serving knobs (this template). Everything under vllmConfig (maxModelLen, gpuMemoryUtilization, enablePrefixCaching, speculative, dtype, kvCacheDtype), plus extraArgs, runtimeClassName, resources. These vary per model/workload — a pilot picks them per InferenceService.

Rule of thumb: same for every model on the platform → Layer 1; varies per model → Layer 2 (here).

Starter profiles​

Two tuned Layer-2 starting points — copy the vllmConfig block matching your workload. These are sensible starting values; treat the magnitudes as directional and re-measure on your hardware — the shape transfers.

Chatbot — many short requests sharing a system prompt:

  vllmConfig:
    maxModelLen: 8192
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true     # ON — shared system prompts reuse KV → lower TTFT
    enableChunkedPrefill: true    # on by default in vLLM V1; keeps prefill from stalling decode

Coding agent — longer contexts, tool-call / multi-turn traffic:

  vllmConfig:
    maxModelLen: 16384            # room for code + tool schemas
    gpuMemoryUtilization: 0.90
    enablePrefixCaching: true
    # n-gram speculative decoding helps at LOW concurrency but goes net-NEGATIVE
    # once the batch saturates — enable only for bursty / low-QPS traffic, and see
    # upstream for the exact vllmConfig.speculative shape. Measure before shipping.

Do not ship --enforce-eager in a production profile — it's a dev/iteration convenience (fast restarts) at a real throughput cost; leave CUDA-graph capture on for serving.

Gotchas​

  • runtimeClassName: nvidia is mandatory — nvidia isn't the default container runtime. Omit it and the pod schedules but can't see the GPU.
  • The model cache backs both operator-download paths — GGUF (single file) and vLLM hf:// (multi-file staging) both land in it and are reused across restarts. Only pvc:// (pre-staged) and skipModelInit (ephemeral, re-downloads every restart) bypass the cache.
  • skipModelInit re-downloads the full weights on every restart into ephemeral storage. Prefer pvc:// for anything you'll restart.
  • The default StorageClass differs by environment (e.g. AKS managed-csi vs. a local local-path). The perService cache logic is the same across them (RWO, WaitForFirstConsumer); only the class name and backing differ.
  • Pending GGUF-cache PVC is normal — perService + WaitForFirstConsumer means it binds only once the serving pod schedules onto a GPU node.
  • VRAM sizing is soft. gpuMemoryUtilization is a fraction of total VRAM, and it's not a hard cap — vLLM can overshoot the target, and CUDA-graph capture allocates after the startup memory check, so a check that passes can still OOM. Check nvidia-smi for other GPU processes first: anything already resident eats into your budget. A clean GPU tolerates a high fraction (e.g. 0.90); leave more headroom when the GPU has other residents. Crash-loops can leave transient un-reclaimed VRAM that cascades the next OOM.
  • --enforce-eager gives ~1-minute restarts by skipping the torch.compile / CUDA-graph capture spike — handy while iterating, at a small peak-throughput cost. Leave it off for production.
  • --served-model-name with pvc://. The default model id becomes the mount path (/model-source/<dir>); set a friendly name so clients pass a sane "model".
  • flashAttention/contextSize/jinja are llama.cpp (top-level) fields. vLLM uses its own attention backend (default FlashAttention on sm_80+) via vllmConfig.attentionBackend — don't set the top-level flashAttention for a vLLM service.
  • flashAttention doesn't always surface in the args. Setting it true on llama.cpp may not visibly reach the rendered startup command (no error, just no flash_attn line). Check the llama.cpp command before relying on it.
  • The GGUF cache has no integrity check. "Already cached, skipping download" tests file existence, not size/hash — an interrupted download leaves a truncated GGUF that gets reused (model is corrupted or incomplete). Set Model.spec.sha256 to force verification, or clear the cache PVC after a failed pull.
  • Quantize on small cards. An 8B model in fp16 won't fit ~12 GB — use AWQ/GPTQ/GGUF. For reference, a 7B AWQ model (~5 GB) fits comfortably at 0.90 on a 12–16 GB GPU.
  • The first vLLM image pull is large (several GB) — one-time per node.
  • vLLM image ↔ GPU driver CUDA compatibility. A recent vLLM image can require a newer CUDA driver than the cloud GPU node ships — on Azure the GPU node image bakes in an older driver, and vLLM then dies at engine init with "the NVIDIA driver on your system is too old". If you hit this, pin spec.image to a vLLM build matching the node's CUDA (e.g. a CUDA-12.4 image on a 12.4 driver). llama.cpp/GGUF is unaffected. (The platform-side fix is a driverless GPU node so the operator installs a current driver — ask the platform team.)
  • Older vLLM images bind IPv6 only. Some older vLLM builds start on [::] and the IPv4 readiness probe gets connection refused, so the pod never goes Ready even though vLLM is up. Add extraArgs: ["--host", "0.0.0.0"].
  • vLLM's served model name defaults to the weights path (e.g. /models/<hash>). Either pass that exact string as the request model, or set extraArgs: ["--served-model-name", "my-model"] for a friendly name.
  • CSI Addon — RWX StorageClass source if you move the GGUF cache to shared mode.
  • Storage — StorageClass reference (RWO vs RWX, per cloud).