4.6 Incident response: diagnosis under a clock

The highest-leverage section in the curriculum. Incident tasks are where exam time evaporates, and make break exists precisely because this skill cannot be read into existence. The concepts fit on half a page; the rest is reps.

needsmake core obs sec

Orientation

competency 4.3 · diagnosing and remediating incidents

Under a clock, the difference between five minutes and twenty is not knowledge; it is having a method you follow instead of improvising. Learn the method, then burn it in with drills until it runs without conscious effort.

A method, so you never freelance under pressure

four steps, in order

Blast radius first. What is broken and what still works. kubectl get pods -A | grep -v Running, make status, the top-level dashboards. One namespace or all? Data plane or control plane? This bounds everything that follows and stops you debugging a symptom two layers from the cause.
Timeline second. What changed. kubectl get events -A --sort-by=.lastTimestamp | tail -30, recent syncs in Argo CD, helm list -A for fresh revisions, kubectl rollout history. Most incidents are deployments wearing a costume.
Descend one resource at a time. describe → events → logs → previous logs (--previous, for crash loops) → exec/debug. Resist skipping levels; the fast-looking jump is where wrong theories come from.
Fix the cause, then prove recovery with the same signal that showed the failure. If you found it via a failing curl, the incident ends with that curl succeeding, not with a pod showing Running.

The signal-to-layer map, worth memorising

Symptom	Layer	First command
Pending	scheduling: resources, taints, PVC, quota, affinity	kubectl describe pod (events at the bottom)
ImagePullBackOff / ErrImagePull	registry, tag, pull secret, network	kubectl describe pod
CrashLoopBackOff	the app itself	kubectl logs --previous
OOMKilled	memory limit (1.2)	…state.terminated.reason / lastState
CreateContainerConfigError	missing ConfigMap/Secret key	kubectl describe pod
Running but broken	NetworkPolicy, DNS, config, RBAC	Hubble, `auth can-i`, app logs
Terminating forever	finalizer, dead controller (3.3)	kubectl get -o yaml \| grep finalizers
Forbidden in a controller log	RBAC, not the workload	kubectl auth can-i --as=system:serviceaccount:…
Everything failing to admit	a webhook backend is down (5.2)	kubectl get validatingwebhookconfigurations

"Running but broken" is the row that separates people: it is where kubectl get pods stops helping and network, identity and config tooling starts.

Node-level moves

When the fault is the node rather than the workload: kubectl cordon <node> stops new pods landing there, kubectl drain <node> --ignore-daemonsets --delete-emptydir-data evicts the rest (respecting PodDisruptionBudgets, which is why it hangs; see section 1.2), and kubectl uncordon puts it back. Draining is also the safe way to test whether a workload survives losing a node, which is a better use of a quiet afternoon than reading about it.

Two tools people forget exist

kubectl debug -it <pod> --image=busybox:1.37 --target=<container>   # ephemeral container into a distroless pod
kubectl debug node/<node> -it --image=busybox:1.37                  # a node shell without SSH

Both exist in the exam environment. The first is the answer to "the image has no shell"; the second to "I need to look at the node and there is no SSH".

Time-boxing is part of the skill

Seven minutes per task is the lab's clock because it is roughly the exam's. If you are four minutes in with no theory that fits the evidence, stop and re-run step 1: widen, do not deepen. And write your diagnosis down before you fix anything; a written theory is falsifiable, a mental one drifts to match whatever you just tried.

After the fix: the part the competency actually names

"structured incident response process"

Remediation is half the competency; the process around it is the other half, and it compresses to five lines you can write in five minutes:

Symptom: what a user or a check observed, with a timestamp.
Blast radius: who and what was affected, and who was not.
Root cause: the change or condition that produced it, stated as a mechanism, not a culprit.
Fix: what you did, including anything you did that turned out to be irrelevant.
Proof of recovery: the command or signal that confirms it, and the follow-up that prevents recurrence (an alert that would have caught it sooner, a policy, a default).

Blameless, mechanism-focused, and short enough that you will actually write it. Vocabulary that may appear in a scenario: MTTD (detect), MTTA (acknowledge), MTTR (restore), error budget (how much unreliability the SLO permits), toil (manual repetitive work that scales with load, the thing a platform is supposed to remove).

Remediation ≠ resolution

On the exam and in life: first restore service (roll back, scale, fail over, restore the policy), then fix the cause properly. Those are two different actions with two different urgencies, and saying which one you are doing is a mark of someone who has done this before.

Exercises

reps, not reading

One session per few days, not all at once:

make break            # random; 7-minute target
make break-answer     # only after you have committed to a diagnosis

Then drill the ones that hurt, by name: FAULT=image, probe, resources, rbac, quota, netpol, config. The rbac fault is invisible in pod listings and only surfaces under kubectl auth can-i --as=...; the netpol fault leaves everything Running while DNS is dead (the additive-allow-list lesson from 1.4).

verify: your diagnosis written down before make break-answer, and the fix confirmed by the failing behaviour now succeeding. make break-fix afterwards shows the evidence trail a systematic diagnosis would have followed; compare it with the path you actually took and note where you diverged.

Run FAULT=probe make break, but restrict yourself for the first three minutes to Grafana, Prometheus and Loki only: find the symptom in the namespace dashboard (restarts, readiness), the failing pod in kube_pod_container_status_ready, and the evidence in its logs via Explore. Then finish with kubectl.

verify: the telemetry told you which pod and roughly why before you ever described it. The exam's observability tasks assume tools-first diagnosis, and this inverts your kubectl habit deliberately.

After any drill, write five lines: symptom, blast radius, root cause, fix, the check that proves recovery.

verify: a stranger could re-run your incident from those lines. Five minutes, once per drill; this is what a "structured incident response process" reduces to in practice.

Course platforms end this module with a scripted "repair a broken stack" lab; the version here is meaner and unscripted: pick a layer, delete something load-bearing but subtle (a ServiceMonitor, the Loki datasource ConfigMap, kyverno's webhook… choose while not thinking about diagnosis), do something else for an hour, then come back and find it via make validate, whose failing checks are your incident ticket.

verify: make validate returns to PASS with FAIL 0. Self-inflicted incidents with a validation suite are the closest thing to a free exam simulator this repo has.

Self-check

say the first command out loud

Every pod in a namespace is Running and the app returns nothing. First three moves?

Make something try (a curl or nslookup from inside the namespace) so there is evidence to read; check endpoints/readiness for the Service; check policy verdicts (Hubble DROPPED) and DNS. "Running" says the kubelet is happy, nothing more.

A pod is CrashLoopBackOff and kubectl logs shows nothing useful.

kubectl logs --previous: you are reading the current, not-yet-started container. Then describe for the exit code and reason (137 with OOMKilled points at memory, 1 with app output points at config), then the previous container's lastState.terminated.

Nothing can be created cluster-wide; every apply times out with a webhook error. What happened and what is the lever?

An admission webhook's backend is down and its failurePolicy is Fail, so the API server refuses writes it cannot validate. Restore the backend, or (as a deliberate emergency step) remove/scope the webhook configuration. That is why failurePolicy is a governance decision, not a default to accept blindly (5.2).

You have four minutes left and no theory that fits the evidence. What do you do?

Go back to step 1 and widen: re-check blast radius and timeline rather than digging deeper into the current object. Most stuck diagnoses are the result of over-committing to the first plausible layer. On the exam, flag it and move on; a stuck task costs you two easy ones.

What proves an incident is over?

The same signal that showed the failure, now succeeding: the failing curl returns 200, the alert resolves, the validation check passes. A pod showing Running proves only that a container started.

Docs to know your way around

study time, not exam time

kubernetes.io: Debug Pods / Debug Running Pods (the ephemeral containers page), Troubleshooting Applications, and the Debug Cluster pages.
sre.google: the postmortem culture chapter, for the vocabulary a scenario question may borrow.
Your own postmortem notes: by the third drill they are the best incident doc you own.