Monitor the application operator

The platform application operator turns each PlatformApplication resource into the Kubernetes objects that run your service: Deployments, Services, Istio routing, external secrets, autoscalers, and disruption budgets. It runs in your cluster, in the platform-operator namespace.

When the operator stops working, nothing breaks loudly. Your running workloads keep serving traffic, because they are already deployed. What stops is change: a new commit deploys and never appears, a scaled replica count never takes effect, a rotated secret never reaches the pod. The queries on this page exist to surface that gap before someone reports it as a mysterious deployment that did nothing.

What you'll learn

• Which metrics the operator exposes, and under which labels
• How to tell "nothing is failing" apart from "nothing is being measured"
• PromQL for reachability, reconciliation progress, throughput, failures, and resource headroom
• LogQL for the operator's log stream, and what it does not tell you

What the operator exposes

The operator serves Prometheus metrics on port 9090. Every metric it emits is prefixed platform_application_operator_. The platform's metrics stack scrapes it, so these selectors are fixed for every cluster:

Selector	Value
Namespace	platform-operator
Container	platform-operator
Scrape job	platform-operator/platform-operator

The examples on this page use those literal values and omit a cluster selector. If your Prometheus holds more than one cluster, add cluster="CLUSTER_NAME" to each selector. Replace CLUSTER_NAME with the name of the cluster you want to inspect.

The reconciliation metrics require application operator version 0.1.419 or later. Against an earlier version, the operator registers the counters but never records them, so they read a flat 0 while it reconciles normally. Check the running version with the following query:

kube_pod_container_info{namespace="platform-operator", container="platform-operator"}

The rule that governs every query

An empty query result and a result of zero mean different things, and confusing them is the most common way a monitoring setup lies to you:

• Zero means the thing was measured and did not happen. No reconcile failed.
• Empty means the thing was not measured. The operator might be gone, the scrape might be broken, or the metrics pipeline might have stopped.

Grafana renders both as an unremarkable panel unless you make it do otherwise. Treat an empty result as Unknown and show it as such, rather than letting it render as a reassuring zero.

One metric family inverts this rule, and it is covered in the section on reconcile failures.

Check that the operator is reachable

up is the scrape result for the operator's metrics endpoint. It is 1 when the last scrape succeeded and 0 when it failed, and it disappears entirely when the target is no longer discovered:

up{job="platform-operator/platform-operator"}

An empty result here is the Unknown case, and it is worth distinguishing explicitly. The following query returns a value only when the operator has no scrape target at all:

absent(up{job="platform-operator/platform-operator"})

Scrape freshness catches a stalled pipeline that has not yet aged out. The result is the age of the most recent scrape, in seconds:

time() - timestamp(up{job="platform-operator/platform-operator"})

A value far above your scrape interval means the metrics are stale even though up still reads 1.

Restarts distinguish a process that is crash-looping from one that is wedged:

increase(kube_pod_container_status_restarts_total{namespace="platform-operator", container="platform-operator"}[1h])

Check that the operator is making progress

Reachability is not progress. An operator can answer every scrape while no longer acting on anything, and this is the failure mode that reachability and resource metrics both read as fine.

The operator runs a scanner that counts objects whose last reconcile is older than its staleness threshold. The scanner writes 0 explicitly on every pass, so a zero here means "scanned, nothing stale" rather than "no data":

sum by (kind) (platform_application_operator_scanner_stale_objects{namespace="platform-operator"})

Because the scanner writes zero explicitly, an empty result is meaningful: it means the scanner itself is not running. Guard for it:

absent(platform_application_operator_scanner_stale_objects{namespace="platform-operator"})

Anything above zero for a sustained period means the operator is running but no longer finishing work on those objects.

Measure reconcile throughput and duration

Reconcile rate covers both controllers the operator runs, for PlatformApplication and PlatformTask resources. The counter carries no controller label, so the two are indistinguishable in this result:

sum(rate(platform_application_operator_reconcile_runs_total{namespace="platform-operator"}[15m]))

The 95th percentile reconcile duration shows whether work is finishing promptly:

histogram_quantile(0.95, sum by (le) (rate(platform_application_operator_reconcile_duration_seconds_bucket{namespace="platform-operator"}[15m])))

histogram_quantile returns NaN, not zero, when the window contains no observations. Grafana draws that as a gap in the line. Read a gap as "nothing reconciled in this window", never as "every reconcile was instant".

Track reconcile failures

This is the one query that must render zero rather than Unknown, and it needs care.

platform_application_operator_reconcile_failures_total is a labeled counter family. It creates no series until the first failure, so a naive query returns nothing while the operator is running without failures, and a panel built on it looks identical to a dead metrics pipeline. The error label holds the resource kind whose reconcile failed, such as Deployment or AuthorizationPolicy, rather than an error message.

Supply a zero, but gate it twice:

sum by (error) (rate(platform_application_operator_reconcile_failures_total{namespace="platform-operator"}[15m]))
  or (label_replace(vector(0), "error", "none", "", "")
      and on() (sum(up{job="platform-operator/platform-operator"}) > 0)
      and on() absent(platform_application_operator_reconcile_failures_total{namespace="platform-operator"}))

Both gates earn their place:

• up > 0 stops the synthetic zero from standing in for an operator that is gone. Without it, a dead operator renders as a confident zero failures.
• absent(...) keeps the synthetic series out of your legend once real failure kinds exist. or is a set operator: it adds any right-hand series whose labels do not collide with the left, so without this gate a permanent none series sits beside the real kinds and reads as a fourth failure kind.

The query then resolves to three distinct states:

State	Result
Failures are occurring	One series per failing resource kind
Operator is running, nothing failing	A single series labeled none, at zero
Operator is unreachable	No series, which your panel should render as Unknown

Watch resource headroom

The operator's pods carry CPU and memory limits. These queries express usage as a percentage of the limit, which is the allocation the operator actually has:

sum(rate(container_cpu_usage_seconds_total{namespace="platform-operator", container="platform-operator"}[5m]))
  / sum(kube_pod_container_resource_limits{namespace="platform-operator", container="platform-operator", resource="cpu"})
  * 100

sum(container_memory_working_set_bytes{namespace="platform-operator", container="platform-operator"})
  / sum(kube_pod_container_resource_limits{namespace="platform-operator", container="platform-operator", resource="memory"})
  * 100

Both queries divide by a limit, so both return nothing when no limit is set. Guard each division separately rather than reading a missing panel as zero usage. A cluster can set a CPU limit and no memory limit, and one guard covering both would leave the unguarded panel just as unexplained:

absent(kube_pod_container_resource_limits{namespace="platform-operator", container="platform-operator", resource="cpu"})

absent(kube_pod_container_resource_limits{namespace="platform-operator", container="platform-operator", resource="memory"})

CPU throttling is the symptom that matters more than the percentage, because it means the operator is already waiting on CPU:

sum(rate(container_cpu_cfs_throttled_periods_total{namespace="platform-operator", container="platform-operator"}[5m]))
  / sum(rate(container_cpu_cfs_periods_total{namespace="platform-operator", container="platform-operator"}[5m]))
  * 100

Query the operator's logs

The operator writes structured logs to stdout, which the platform ships to Loki. There is no level label on the log stream: the level is part of the line, so every level-based query parses it out. The same expression extracts the Rust module path, which is the most useful grouping the operator offers.

Log volume by level:

sum by (level) (rate({namespace="platform-operator", container="platform-operator"}
  | regexp `\s+(?P<level>TRACE|DEBUG|INFO|WARN|ERROR)\s+(?P<module>[A-Za-z0-9_:]+):` [5m]))

The operator's own view of the Kubernetes API, which is the half of connectivity that up cannot see. kube_client::client::builder writes a line only when a call fails, so the or vector(0) tail turns "no matching line" into a zero rather than a gap:

sum(count_over_time({namespace="platform-operator", container="platform-operator"}
  |~ `kube_client::client::builder` [5m])) or vector(0)

This is the one zero on the page that does not validate itself. An absent log stream floors to the same zero as a live stream carrying no failures, so this result cannot on its own tell "no API errors" from "no logs at all". Read it beside the log volume query above: a non-zero log volume is what makes this zero mean the first rather than the second.

Recent warnings and errors:

{namespace="platform-operator", container="platform-operator"}
  | regexp `\s+(?P<level>TRACE|DEBUG|INFO|WARN|ERROR)\s+(?P<module>[A-Za-z0-9_:]+):`
  | level=~"WARN|ERROR"

On Azure clusters, that last query is dominated by a single recurring notice, Cloud azure CDN not supported. Skipping deployment annotations additions., which the operator emits on every Deployment reconcile and which needs no action. Exclude it so the lines that matter are visible:

{namespace="platform-operator", container="platform-operator"}
  | regexp `\s+(?P<level>TRACE|DEBUG|INFO|WARN|ERROR)\s+(?P<module>[A-Za-z0-9_:]+):`
  | level=~"WARN|ERROR" != `Cloud azure CDN not supported`

What the logs do not tell you

A failed reconcile writes no ERROR line. The operator increments platform_application_operator_reconcile_failures_total and continues, so the failure counter is the only place a failed reconcile surfaces.

ERROR lines do appear for other reasons, such as Kubernetes API errors. Neither direction is safe to infer: a quiet log stream is not evidence that reconciliation is succeeding, and an ERROR line is usually about something other than a failed reconcile. Use the failure counter for reconcile outcomes and the logs for everything else.

Metric reference

The operator exposes the following metrics. All of them carry the standard scrape labels for cluster, namespace, pod, and job.

Metric	Type	Notes
platform_application_operator_reconcile_runs_total	Counter	All reconciles, both controllers combined. No controller label. Requires 0.1.419 or later.
platform_application_operator_reconcile_duration_seconds	Histogram	Reconcile wall time. Query the _bucket series with histogram_quantile. Requires 0.1.419 or later.
platform_application_operator_reconcile_failures_total	Counter	Labeled by error, which holds the resource kind that failed. No series exists before the first failure. Requires 0.1.419 or later.
platform_application_operator_scanner_stale_objects	Gauge	Objects past the staleness threshold, labeled by kind. Written as 0 on every scan.

Related

• Observability covers instrumenting your own applications with logs, metrics, traces, and health checks.
• Metrics explains the Prometheus metric types and naming conventions the operator follows.