Cost on Kubernetes is an allocation problem, not a billing problem. The cluster costs what it costs; the question is which namespace, workload or label is responsible for how much of it, and the answer is computed from requests, usage and time, out of the same Prometheus metrics you already collect.

needsmake up obs

Orientation

competency 1.2 · cost management for right-sizing and scaling

You already built the measuring half in section 1.2. This section adds the money view and the loop that connects them: measure usage, compare with requests, adjust requests, confirm nothing degraded. That loop is the competency; OpenCost is just the instrument.

The allocation formula, conceptually

For each container, for each time window: cost = max(request, usage) × unit_price × hours, summed over CPU, memory, GPU, storage and network, then attributed to the pod's namespace, labels or owner. Everything OpenCost shows is that expression sliced differently. Once you see it, the "why is my namespace expensive when nothing is running" question answers itself: because request won the max().

What drives spend, in the order it matters

requests · idle · sprawl
  1. Requests, not usage. Capacity planning and most internal pricing allocate by what you reserved. A deployment requesting 4 CPU and using 200m costs 4 CPU. The gap between the two is the "efficiency" number OpenCost reports, and closing it is right-sizing.
  2. Idle capacity. Nodes are bought whole; unrequested space is idle cost. OpenCost can either show it separately or spread it across tenants, and knowing that both accounting choices exist is the exam-relevant fact. Showing idle separately makes the platform team accountable for bin-packing; spreading it makes tenants feel the true cost of the cluster they share. Neither is wrong; they answer different questions.
  3. Object sprawl. LoadBalancers, PVs, snapshots, forgotten namespaces and orphaned PVCs bill whether or not traffic flows. This is why the team-a quota caps count/services.loadbalancers at 1, and why quota is a cost tool as much as a fairness tool.
  4. Node shape and lifecycle. The lever above the workload: instance families, spot/preemptible capacity, consolidation (Karpenter-style), and scale-to-zero for dev environments. You cannot demonstrate these in kind, but "right-size the pods, then right-size the nodes" is the correct order to say out loud.

The same gap, priced. Notional rates, real arithmetic; note how quickly trimming turns risky once usage approaches the request:

Vocabulary that shows up in scenario questions

Showback reports cost to teams; chargeback actually bills them. Allocation is dividing shared cost by a defensible key (namespace, label, owner). Efficiency is usage ÷ request. Unit economics is cost per business thing (per tenant, per request). A "reduce cost" scenario is usually asking for allocation first: you cannot cut what you cannot attribute.

How OpenCost actually works here

metrics in, allocation out

OpenCost is a small controller plus a query API. It reads pod and node inventory from the API server, joins it with usage series from Prometheus (container_cpu_usage_seconds_total, container_memory_working_set_bytes, kube_pod_container_resource_requests, node capacity) and applies a price list: cloud billing rates in a real cluster, default on-prem rates in this one. The prices in the lab are notional; the ratios and gaps are real, and those are what you are learning to read.

kube-state-metrics ─┐
cadvisor / kubelet  ─┼─▶ Prometheus ─▶ OpenCost ─▶ allocation API ─▶ kubectl cost / UI / Grafana
node inventory     ─┘                        ▲
                                            price list (cloud rates or defaults)

Two consequences worth predicting before you run anything. First, OpenCost is only as good as the metrics under it, so a broken scrape shows up as missing cost, not as an error, and diagnosing "why is this namespace zero" is a Prometheus problem wearing a cost hat. Second, allocation windows matter: a 1-hour window over a freshly created workload shows nothing useful, so use --window 1d when you want stable numbers.

One flag to burn in

kubectl cost assumes a Kubecost install by default. --opencost is what points it at this stack: different service name, port and API path, all bundled in the one flag. Forgetting it produces a connection error that looks like OpenCost is broken when it is simply not being asked.

The right-sizing loop, as an exam answer

  1. Measure: kubectl top, Prometheus, VPA recommendations (1.2).
  2. Compare: requests vs p95-ish usage, per container, not per namespace average.
  3. Adjust: kubectl set resources or the manifest in git; memory to roughly peak plus headroom, CPU to a sane request with a generous or absent limit.
  4. Confirm: nothing throttling, nothing OOMKilled, nothing Pending, and the efficiency number moved.

Step 4 is where people lose the mark. Right-sizing that breaks scheduling or introduces throttling is worse than the waste it removed, and saying so unprompted is what separates a platform engineer's answer from a dashboard reader's.

Exercises

tick the dot when its check passes

OpenCost's UI is on the LoadBalancer make urls prints, but the CLI is faster and exam-shaped:

kubectl cost --opencost namespace --show-all-resources
kubectl cost --opencost namespace --historical --window 1d
verify: a table where monitoring dominates (kube-prometheus-stack is the heaviest thing installed) and the team namespaces are near zero. If the command errors, diagnose the path: kubectl-cost talks to the opencost service, which talks to Prometheus; kubectl -n opencost get deploy,svc and the pod logs tell you which hop broke.

Two views of the same truth. From cost:

kubectl cost --opencost namespace --show-efficiency

From raw metrics, CPU requested minus used, per namespace (run in the Prometheus UI from make urls):

sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})
  - sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[10m]))

The label filters matter: cadvisor also emits pod-level and node-level aggregate series (empty container) and, on many versions, the pause container as container="POD". Without {container!="",container!="POD"} you count the same CPU two or three times and the "gap" you compute is fiction.

verify: the namespace at the top of the PromQL result matches the low efficiency scores in kubectl-cost. Name the top offender and what you would change.

Deploy the demo app with a deliberate oversize:

kubectl apply -k examples/demo-app/base
kubectl set resources deploy demo --requests=cpu=500m,memory=512Mi

Wait ten minutes for metrics to accumulate, get the VPA recommendation for it (section 1.2), then apply sane numbers with kubectl set resources again.

verify: kubectl cost --opencost namespace --show-efficiency for default improves between the two states, and the pods never restarted into a Pending state (right-sizing that breaks scheduling is worse than the waste).

Compute what team-a can maximally cost: its quota hard-caps requests at 2 CPU / 4Gi. That number is a budget expressed in Kubernetes objects. Write the one-sentence explanation of why a platform team sets quotas even when nobody fights over capacity.

verify: if your sentence does not mention predictable spend, read this section again.

Self-check

answer before opening
A namespace runs almost no traffic and still tops the cost report. Give the two most likely explanations.

Oversized requests (billed on reservation, not usage), or expensive attached objects: LoadBalancers, provisioned volumes, retained snapshots. Check efficiency first: very low efficiency points at requests; high efficiency with high cost points at the objects.

Why is container!="" in that PromQL query not optional?

cadvisor emits per-container series plus pod-level and node-level rollups with an empty container label, and often the pause container as container="POD". Summing everything counts the same CPU two or three times, so every aggregate over cadvisor series needs {container!="",container!="POD"} or an equivalent.

Should idle cost be spread across tenants or shown separately? Argue both.

Spread: tenants see the true cost of the cluster they share, which discourages hoarding. Separate: the platform team owns bin-packing and node choice, so making idle visible as their number is what drives consolidation. Mature setups show both: tenant cost at request-level, plus a platform-owned idle line.

You cut a deployment's CPU request from 500m to 50m and latency degrades. What happened?

Requests set the cgroup CPU weight, so under contention the container now gets a much smaller share; and if a limit exists near the old request, throttling is likelier. Right-sizing means matching observed usage with headroom for peaks, not matching the average. Confirm with throttling metrics, not with the cost graph.

Give the four-step right-sizing loop in one breath.

Measure usage → compare with requests → adjust requests → confirm no degradation (no throttling, no OOM, no Pending) and re-measure efficiency. The fourth step is the one that makes it engineering rather than cost-cutting.

Docs to know your way around

study time, not exam time
  • opencost.io: the allocation API and the efficiency definition; the "how cost is calculated" page is short and is exactly the formula above.
  • github.com/kubecost/kubectl-cost: flag reference; it has more views (controller, deployment, label) than the two used here.
  • finops.org: the FinOps framework vocabulary (inform, optimize, operate) if a scenario question uses the words.
  • Offline: kubectl cost --help, and the same Prometheus queries from section 4.1.