Requests and limits look like beginner material and then show up everywhere: quotas count requests, OpenCost bills requests, LimitRanges inject them, Kyverno policies demand them, HPAs divide by them, and the resources break drill corrupts them. Get the mechanics exact and four other sections get easier.
make upOrientation
Two numbers per container decide scheduling, cost, eviction order, throttling, and whether your workload survives a noisy neighbour. Almost nobody can state precisely what each one does at each layer. That precision is the section.
A request is a claim against the scheduler's arithmetic; a limit is a ceiling enforced by the kernel at runtime. Nothing reconciles the two after scheduling, which is how a node can be 30% used and 100% requested at the same time. That gap is the entire cost domain in one sentence, and section 1.5 turns it into money.
The mechanics, exactly
What the scheduler does with requests
The scheduler sums the requests of all non-terminated pods on a node and compares against the node's allocatable, not its capacity. Allocatable is capacity minus kube-reserved, system-reserved, and the eviction threshold. Live usage is never consulted. A node running at 5% CPU with every millicore requested is, to the scheduler, completely full.
node capacity ████████████████████████████████████████ 4000m
allocatable ██████████████████████████████████░░░░░░ 3800m (minus kubelet/system/eviction)
sum of requests ████████████████████████████████░░░░░░░░ 3600m ← what the scheduler sees
actual usage ██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1100m ← what you pay for and never use
└──────────── the right-sizing gap ────────────┘
The four numbers, live: drag the requests up until the node refuses another pod, then drag usage down and watch what you are paying for:
What the kernel does with limits
| Resource | request becomes | limit becomes | over the limit |
|---|---|---|---|
| cpu | cpu.weight (relative share) | cpu.max (quota per 100ms period) | throttled: silently, no event, no restart |
| memory | eviction ranking only | memory.max | OOMKilled: instant, by the kernel |
| ephemeral-storage | scheduling claim | kubelet-enforced | pod evicted |
| hugepages / devices | requests must equal limits; no overcommit possible | ||
CPU is compressible: exceed the limit and the CFS scheduler simply stops giving you cycles until the next 100ms period. Latency-sensitive services with a tight CPU limit get periodic stalls that look like network problems; container_cpu_cfs_throttled_periods_total is where the truth lives, and a nonzero rate on a healthy-looking service is a real finding. Memory is incompressible: there is no "slow down", only the OOM killer.
For a pod with --restart=Never, an OOM kill lands in .status.containerStatuses[0].state.terminated.reason. In a crash-looping Deployment, the container has already restarted, so the same evidence is in lastState.terminated.reason and state shows waiting: CrashLoopBackOff. Reading the wrong field and concluding "no OOM here" is a classic self-inflicted wound.
QoS, derived not memorised
| Class | Condition | Eviction order | oom_score_adj |
|---|---|---|---|
| Guaranteed | every container: requests == limits for both cpu and memory | last | -997 |
| Burstable | at least one request or limit set, but not Guaranteed | middle: those exceeding requests first | 2–999, scaled |
| BestEffort | nothing set at all | first | 1000 |
Derive it rather than recall it. Set the four numbers and read the class, the eviction rank and the two runtime behaviours:
Under node memory pressure the kubelet ranks victims in a specific order: pods whose usage exceeds their requests first, then by Pod Priority (lowest goes first), then by how far over the request they are. QoS class is a consequence of that first test rather than an input: a BestEffort pod requested nothing, so it always exceeds. The practical corollary: a high-priority Burstable pod outlives a low-priority BestEffort one, which is why PriorityClass protects platform components from eviction, not just from preemption. The same ranking is mirrored to the kernel via oom_score_adj so a system-level OOM picks the same victim. CPU pressure never evicts anything; it only throttles. If someone tells you a pod was "evicted for CPU", they are describing something else.
Placement: the levers, in the order you reach for them
| Lever | Direction | Hard or soft | Reach for it when |
|---|---|---|---|
| nodeSelector | pod → node labels | hard only | trivially simple pinning; blunt |
| nodeAffinity | pod → node labels | required… / preferred… with weights | expressive pinning, operators (In, NotIn, Exists, Gt, Lt) |
| podAffinity / podAntiAffinity | pod → other pods, within a topologyKey | both | co-locate with a cache; spread replicas across nodes |
| taints + tolerations | node repels pods | NoSchedule, PreferNoSchedule, NoExecute | dedicated pools, control planes, GPU nodes |
| topologySpreadConstraints | even distribution over a label domain | DoNotSchedule / ScheduleAnyway | zone/node spread with a bounded skew, the modern default |
| priorityClass | who wins when the node is full | preemption | platform components must outrank tenant workloads |
Three details that decide tasks. Taints repel pods, tolerations do not attract: a toleration only says "I can live here", never "put me here"; pairing a taint with a matching nodeAffinity is how you actually dedicate a pool. NoExecute evicts already-running pods that lack the toleration, and tolerationSeconds is what makes the node-not-ready eviction delay configurable. And anti-affinity is expensive to evaluate at scale, which is exactly why topologySpreadConstraints exists: same intent, bounded cost, plus maxSkew to say how even is even enough.
The kind config labels its workers into two zones (topology.kubernetes.io/zone) precisely so spread constraints do something observable. kubectl get nodes -L topology.kubernetes.io/zone shows you the domains before you write the constraint, and reading the domain labels first is the habit that stops you writing a constraint against a key no node carries, which fails open under ScheduleAnyway and closed under DoNotSchedule.
One neighbour worth naming because availability tasks touch it: a PodDisruptionBudget constrains voluntary disruptions (drains, node upgrades) with minAvailable or maxUnavailable. It does nothing about crashes or node failures. A PDB of minAvailable: 100% is the classic way to make a cluster upgrade hang forever, and recognising that stalled-drain symptom is worth a mark.
Autoscaling: three different scalers, three different jobs
| Scaler | Changes | Input | Gotcha |
|---|---|---|---|
| HPA | replica count | live metrics (Resource, Pods, Object, External) | needs metrics-server; percentage targets are relative to requests |
| VPA | the requests themselves | historical usage | fights an HPA on the same resource; updateMode: Off makes it advisory |
| Cluster Autoscaler / Karpenter | node count / node shape | unschedulable pods | reacts to Pending pods, so it is downstream of requests too |
| KEDA | replicas, including 0 | queue depth, topic lag, cron, any external source | a ScaledObject generates an HPA underneath; it is the thing that produces those "External" metrics, and the only way to scale to zero |
The HPA formula, which explains every surprise
desiredReplicas = ceil( currentReplicas × ( currentMetricValue / desiredMetricValue ) )With one caveat that explains most "why didn't it scale" moments: the HPA does nothing while the ratio sits inside its tolerance (10% by default), and pods that are unready or missing metrics are left out of the average entirely. With --cpu-percent=20 and containers requesting 25m, the target is 5m of actual usage per pod. That is why a demo app with a tiny request scales up under a trivial load: the percentage is a fraction of the request, not of the node. Change the request and you change the autoscaler's behaviour without touching the HPA, one of those couplings that looks like a bug the first time you meet it.
Scaling up is fast; scaling down waits out a stabilisation window (300s by default) so a brief dip cannot flap your fleet. Both directions are configurable per-HPA under spec.behavior with policies (pods or percent, per period) and selectPolicy. Knowing that downscale lag exists, and that it is deliberate, is worth a mark on its own.
VPA modes
Off: compute recommendations only. The exam-relevant one: a pure right-sizing oracle you can read without letting anything evict.Initial: apply recommendations at pod creation only.Auto/Recreate: evict and recreate pods to resize them. Disruptive by design, because changing a running pod's requests historically required a new pod.
Newer clusters can resize CPU and memory in place, which softens that trade-off. Do not test for it with kubectl explain pod.spec.containers.resizePolicy; that field has been in the schema since 1.27 whether or not the feature is on. The honest check is whether the subresource exists: kubectl get --raw /api/v1 | grep -o 'pods/resize', or simply try the patch and read the error.
kubectl top pod --containers for the instantaneous truth, kubectl describe node for the requests-vs-allocatable table at the bottom (the single most useful capacity view kubectl gives you), and kubectl get vpa -o jsonpath='{.items[*].status.recommendation…}' for what the numbers ought to be.
Exercises
Create three pods in default: one with requests==limits, one with only requests, one with nothing. Predict each class before checking:
kubectl get pod <name> -o jsonpath='{.status.qosClass}{"\n"}'Memory first. kubectl run lost its resource flags a while back, so this is also a rep for the --overrides escape hatch:
kubectl run oom --image=polinux/stress --restart=Never \
--overrides='{"spec":{"containers":[{"name":"oom","image":"polinux/stress","command":["stress","--vm","1","--vm-bytes","128M","--vm-hang","0"],"resources":{"requests":{"memory":"64Mi"},"limits":{"memory":"64Mi"}}}]}}'
kubectl get pod oom -w # until STATUS shows OOMKilledkubectl get pod oom -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}' prints OOMKilled (with --restart=Never nothing restarts, so the evidence sits in state, not lastState; in a crash-looping Deployment it is the other way round, and knowing which field to read is half the diagnosis). CPU, by contrast, would have throttled silently. Remember which is which; the exam loves the difference.Deploy 4 replicas of anything with:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector: { matchLabels: { app: spread-demo } }kubectl get pods -l app=spread-demo -o wide shows 2+2 across the two worker zones, and kubectl get nodes -L topology.kubernetes.io/zone confirms which zone each node carries.Deploy examples/demo-app/base into default (kubectl apply -k examples/demo-app/base), then:
kubectl autoscale deploy demo --min=2 --max=6 --cpu-percent=20
kubectl run load --image=busybox:1.37 --restart=Never -- \
sh -c 'while true; do wget -qO- http://demo.default.svc:80 >/dev/null; done'
kubectl get hpa demo -wload and watch it settle back after the stabilisation window (about 5 minutes; knowing that downscale lag exists is worth a mark). Note the demo container requests 25m CPU, which is why 20% is reachable at all.With demo still running:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: demo, namespace: default }
spec:
targetRef: { apiVersion: apps/v1, kind: Deployment, name: demo }
updatePolicy: { updateMode: "Off" }Give it a few minutes, then:
kubectl get vpa demo -o jsonpath='{.status.recommendation.containerRecommendations[0]}' | jqSelf-check
A node shows 25% CPU usage and refuses to schedule a pod requesting 200m. Explain, in one sentence.
The scheduler counts requests against allocatable, not usage: the node's requests are already at allocatable even though the processes are idle. Fix by right-sizing the existing requests (or adding capacity), not by adding CPU headroom that already exists.
Which pod does the kubelet evict first under memory pressure, and why?
BestEffort, because it declared no memory request at all and therefore ranks worst. Then Burstable pods exceeding their requests, ordered by how far over they are. Guaranteed last. The kernel's oom_score_adj mirrors that ranking so a system-level OOM picks the same victim.
Your service has p99 latency spikes every few seconds; CPU usage sits at 60% of the limit. First metric you check?
rate(container_cpu_cfs_throttled_periods_total[5m]). Average usage below the limit hides per-period throttling: the container burns its 100ms quota early and stalls until the next period. Average utilisation is the wrong lens for CPU limits.
Why can a VPA in Auto mode and an HPA on CPU not coexist on the same workload?
They form a loop: the HPA scales replicas based on usage-against-requests while the VPA rewrites those same requests, so each one keeps invalidating the other's denominator. Standard resolution: HPA on CPU with VPA in Off (advisory), or HPA on a custom/external metric while VPA owns the resources.
A drain hangs forever on one node. What are the two usual causes?
A PodDisruptionBudget that cannot be satisfied (often minAvailable equal to the replica count), or unmanaged pods: bare pods with no controller, which kubectl drain refuses to evict without --force. Both are voluntary-disruption mechanics; neither has anything to do with node health.
Docs to know your way around
- kubernetes.io: Resource Management for Pods and Containers; Pod Quality of Service Classes; Node-pressure Eviction; Assigning Pods to Nodes; Pod Topology Spread Constraints; the HorizontalPodAutoscaler walkthrough (the algorithm section especially).
- github.com/kubernetes/autoscaler: the VPA README, in particular the
updateModetable. - Offline:
kubectl explain pod.spec.containers.resources,kubectl explain hpa.spec.behavior --recursive, and the requests table at the bottom ofkubectl describe node.