The obs layer installs kube-prometheus-stack, which is three things people conflate: Prometheus itself, the Prometheus Operator that configures it through CRDs, and a bundle of exporters and dashboards. On the exam you work through the operator's CRDs, so that is the layer to be fluent in.

needsmake up obs

Orientation

competency 4.1 · monitoring solutions

Two skills, both testable: get a target scraped, and write a query that answers a question. Everything else in domain 4 (alerting, dashboards, DORA metrics, incident diagnosis) is built on those two.

The pull model in one paragraph

Prometheus scrapes HTTP endpoints on an interval and stores samples in a local TSDB. Each series is a metric name plus a set of labels, and every distinct label combination is a separate series, which is why cardinality is the resource you actually manage. Service discovery (in Kubernetes: the API server) produces the target list; relabeling rewrites and filters it; the scrape either succeeds (up = 1) or does not (up = 0). Nothing is pushed, so a workload that only exists for ten seconds is invisible unless it pushes to a Pushgateway or emits an event elsewhere.

The operator model

ServiceMonitor · PodMonitor · selectors

What to scrape is declared, not configured. A ServiceMonitor says "scrape the endpoints of Services matching these labels, on this port name, at this path, every N seconds". A PodMonitor does the same without a Service. A ScrapeConfig (newer) covers targets outside Kubernetes. A Probe drives blackbox-exporter checks. The operator watches these CRDs and rewrites Prometheus's configuration live.

ServiceMonitor (selector: app=example, port: web, path: /metrics)
      │  matched by Prometheus.spec.serviceMonitorSelector   ← the filter people forget
      ▼
Service app=example, ports: [{name: web, port: 8080}]
      │  its EndpointSlice supplies the actual pod IPs
      ▼
pod:8080/metrics  ──scraped every 30s──▶  series {job="example", namespace=…, pod=…}
The hour everyone loses once

Prometheus only picks up ServiceMonitors matching its own serviceMonitorSelector, which on a stock kube-prometheus-stack matches the Helm release label (release: prometheus). A perfectly correct monitor without that label is silently ignored: no error, no event, no target. This lab deliberately disables that filter (serviceMonitorSelectorNilUsesHelmValues=false), so its selector is {} and every monitor everywhere is picked up. Both facts matter, and the durable habit is: read the selector first, then write the monitor.

kubectl -n monitoring get prometheus -o jsonpath='{.items[0].spec.serviceMonitorSelector}'; echo
kubectl -n monitoring get prometheus -o jsonpath='{.items[0].spec.serviceMonitorNamespaceSelector}'; echo

Prints {} on this cluster and the release-label selector on a stock install. That one command is the answer to "my target does not appear" everywhere, including the exam's cluster, whose selector you do not get to assume. The namespace selector is the second half of the same trap: a monitor in a namespace Prometheus does not watch is equally invisible.

The other three ways a target goes missing

  1. Port name, not number. endpoints[].port is the Service's port name. An unnamed Service port cannot be referenced, and the monitor matches while scraping nothing.
  2. No endpoints. The Service selector matches no ready pods (section 1.1). Same symptom, different layer.
  3. RBAC or network. Prometheus's ServiceAccount must be able to list endpoints in that namespace, and NetworkPolicy must allow monitoring → workload. In a default-deny tenant namespace, that allow is a thing someone has to write.

Diagnosis order: Status → Targets in the UI (it shows the reason for a failed scrape), then up{job="…"}, then the four causes above.

Relabeling, at exam depth

relabelings act on discovered targets before the scrape (drop targets, rewrite the job or namespace labels); metricRelabelings act on samples after (drop expensive series). The one pattern worth remembering: dropping a high-cardinality metric at ingestion with action: drop on a __name__ regex is how you rescue a Prometheus drowning in one bad exporter.

PromQL, the working subset

90% of exam queries live here
TypeRead it withExample
Counter (only climbs, resets to 0)rate() · increase()rate(http_requests_total[5m])
Gauge (up and down)read it directlykube_pod_status_ready
Histogram (buckets)histogram_quantile()histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m])))
Summary (pre-computed quantiles)read the quantile labelx{quantile="0.99"}
  • Selectors: up{namespace="argocd"}, with =, !=, =~, !~. Regexes are fully anchored.
  • Ranges: [5m] turns an instant vector into a range vector: required by rate/increase, and forbidden everywhere else. "Expected type instant vector" almost always means a stray range selector.
  • Aggregation: sum by (label) (...) and sum without (pod) (...). by keeps only what you name; without keeps everything else. Aggregating before rate is wrong: rate first, then sum.
  • Arithmetic between vectors matches on labels; mismatched label sets produce an empty result, which is why on()/ignoring() and group_left exist. You used vector arithmetic for the cost gap in section 1.5.
  • Emptiness: absent(x) is 1 when x has no series (the way to alert on "the metric stopped existing"), and … or vector(0) is how you make a query return a number instead of nothing.
  • rate vs irate: rate averages over the window (use it for alerts and dashboards), irate uses the last two samples (spiky, for zooming in). increase is rate × window, useful for "how many in 24h".

Why the raw counter is useless, and what the window does to the answer:

up deserves special respect

1 per healthy target, 0 per failing one, which makes count(up == 0) the cluster's own health check. make validate runs exactly that query and demands zero; the lab even patched kubeadm's localhost-bound control-plane metrics to make them reachable (make fix-cp-metrics). Any "is monitoring healthy" question starts with this query.

What is already collected, for free

SourceGives youTypical series
kube-state-metricsobject state from the API serverkube_pod_info, kube_deployment_status_replicas_unavailable, kube_pod_container_status_restarts_total
cAdvisor (kubelet)container resource usagecontainer_cpu_usage_seconds_total, container_memory_working_set_bytes
node-exporterhost metricsnode_load1, node_filesystem_avail_bytes
kubelet / apiservercontrol-plane health and latencyapiserver_request_duration_seconds_bucket

Knowing these exist means you rarely need to instrument anything to answer an exam question about the cluster itself.

Exercises

tick the dot when its check passes

Prometheus's UI address comes from make urls.

In the UI under Status → Targets, or via API: count the scrape pools, find which ServiceMonitor each corresponds to (kubectl -n monitoring get servicemonitors), and confirm zero targets down.

verify: curl -s 'http://<prom>/api/v1/query?query=count(up==0)' | jq -r '.data.result[0].value[1] // "0"' prints 0. If not, the down target's job name points at the responsible ServiceMonitor.

Deploy an app that exposes metrics and monitor it, first wrong, then right:

kubectl create deploy example --image=quay.io/brancz/prometheus-example-app:v0.5.0 --port=8080
kubectl expose deploy example --port=8080 --name=example
kubectl apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: example, namespace: default }
spec:
  selector: { matchLabels: { app: example } }
  endpoints: [{ port: "8080" }]
EOF

Wait a minute, search Targets: nothing. The endpoint's port must be the Service's port name, and this Service has none, so the monitor matches the Service and then finds no endpoint. It fails silently, which is exactly why it is worth experiencing once. Fix it: patch the Service so the port is named (kubectl patch svc example --type=json -p '[{"op":"add","path":"/spec/ports/0/name","value":"web"}]') and set port: web in the monitor.

verify: up{job="example"} returns 1 in the UI. Then add the release: prometheus label to your monitor anyway and confirm nothing changes here, while being able to say why it would be the difference between working and ignored on a stock install.

Generate a little traffic (kubectl run curl --image=curlimages/curl:8.11.1 --restart=Never -- sh -c 'for i in $(seq 100); do curl -s example.default.svc:8080; done'), then write, without copying: the request rate (rate(http_requests_total{job="example"}[5m])), summed by status code, and the p95 duration from http_request_duration_seconds_bucket.

verify: rates are non-zero and the quantile returns a number, not NaN. If it is NaN, reason about why (too few buckets observed yet); that reasoning is itself testable.

Answer three questions using only metrics already collected: how many pods per namespace (kube_pod_info), which container restarts most (kube_pod_container_status_restarts_total), and each node's CPU pressure (node_load1 vs allocatable).

verify: each answer is one query in the UI. kube-state-metrics and node-exporter are pre-installed sources the exam expects you to know exist.

Self-check

answer before opening
Your ServiceMonitor is correct and no target appears. List the checks in order.

1) Prometheus's serviceMonitorSelector and serviceMonitorNamespaceSelector: does it even consider your monitor? 2) Does the monitor's selector match the Service's labels? 3) Is endpoints[].port the port name, and does the Service name it? 4) Does the Service have ready endpoints? 5) RBAC and NetworkPolicy between monitoring and the target namespace.

Why is sum(rate(x[5m])) right and rate(sum(x)[5m]) wrong?

Counters reset per series; rate knows how to handle resets within a single series. Summing first merges series (and their resets) into a meaningless line, and the syntax is invalid anyway without a subquery. Rate first, aggregate second, always.

How do you alert on "the metric disappeared entirely"?

absent(up{job="x"} == 1) or absent(metric): an ordinary comparison returns no series when there is nothing to compare, so it can never fire. absent() exists precisely to turn "no data" into a value of 1.

A p95 query returns NaN. Two plausible reasons?

No observations in the window (nothing has been recorded yet, so all buckets are empty), or you aggregated away the le label that histogram_quantile needs. The canonical form keeps it: histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m]))).

An exporter added a label with one value per request. What happens, and what is the fix?

Cardinality explosion: a new series per unique value, memory and TSDB growth, slow queries. Fix at ingestion with a metricRelabeling that drops the label or the metric, and upstream by removing it from the exporter. High-cardinality labels (user IDs, URLs with IDs, trace IDs) belong in logs and traces, not metric labels.

Docs to know your way around

study time, not exam time
  • prometheus.io: querying basics and the function reference (rate, increase, histogram_quantile, absent); metric types.
  • prometheus-operator.dev: the ServiceMonitor troubleshooting page, which is the release-label story in official form; the API reference for ServiceMonitor/PodMonitor/ScrapeConfig.
  • Offline: kubectl explain servicemonitor.spec.endpoints, the Prometheus UI's Status → Targets and Status → Configuration pages, and its built-in expression autocompletion.