The exam will not ask you to write a controller in Go. It will ask you to operate, integrate and above all diagnose operators, which means reading their outputs fluently: status conditions, events, owned resources, and logs, in that order.

needsmake up api

Orientation

competency 3.3 · operators for automation and integration

Every layer of this lab is operators consuming CRDs: Crossplane, Kyverno, Prometheus Operator, Trivy Operator, CloudNativePG, Argo's controllers, Flux's controllers. An unfamiliar operator on the exam is the same four questions every time: what kind does it watch, what does it create, what does its status say, and where do its logs live?

Level-based, not edge-based

A controller watches a kind and, for each object, runs the same function: observe actual state, compare with desired spec, take one step toward convergence, write status, requeue. It acts on the state it finds, not on the event that woke it, so missed events cost nothing, duplicate events are harmless, and the loop is safe to run at any time. That is why deleting an operator-owned pod is a non-event: the next reconcile recreates it without knowing or caring that you deleted it.

The conventions that make operators readable

status · generation · owners · finalizers

1 · Status conditions

Typed, with type, status (True/False/Unknown), reason (machine-speak, CamelCase), message (for you), lastTransitionTime and observedGeneration. Ready is the summary; the others tell you which phase is stuck. A well-behaved operator also emits Events, which carry the same story with timestamps and counts.

2 · observedGeneration versus metadata.generation

If they differ, the controller has not yet processed your latest edit, and whatever status says describes a previous spec. Checking this first avoids diagnosing stale information, and almost nobody does it. metadata.generation increments on spec changes only, provided the CRD has the status subresource. Without it, status writes bump generation too, and observedGeneration tells you nothing. That is one more reason the subresource is not optional for a real API (section 3.2).

3 · Owner references

Operator-created resources carry ownerReferences pointing at their parent. This answers "what keeps recreating this thing", drives cascade deletion (foreground, background, or orphan), and lets you reconstruct an operator's whole object tree without documentation. Note the rules: an owner must be in the same namespace (or be cluster-scoped), and a namespaced object cannot own a cluster-scoped one, the constraint that shapes Crossplane's design in 3.5.

4 · Finalizers

A deletion sticks in Terminating while a finalizer is present, because the controller is doing teardown work, or is dead and cannot. A stuck namespace or CR almost always means "find whose finalizer, and why its controller is not running". Removing the finalizer by hand is the last resort, and it leaks whatever the teardown was supposed to clean up (external volumes, cloud resources, DNS records). Say that trade out loud before you do it.

Cluster/pg (your CR)
 ├─ ownerRef ─▶ StatefulSet-ish pods  pg-1, pg-2         each with a PVC
 ├─ ownerRef ─▶ Services  pg-rw, pg-ro, pg-r
 ├─ ownerRef ─▶ Secrets  pg-app, pg-superuser        ← connection details
 └─ status.conditions[]  Ready / Initialized / ContinuousArchiving …
                         observedGeneration must equal metadata.generation
The four-command diagnosis 
kubectl get <kind> <name> -o jsonpath='{.status.conditions}' | jq
kubectl get events --field-selector involvedObject.name=<name> --sort-by=.lastTimestamp
kubectl get all,pvc,secret -l <the operator's label> -n <ns>
kubectl -n <operator-ns> logs deploy/<controller> --tail=100

In that order. Conditions tell you the phase, events tell you the attempts, owned objects tell you what exists, logs tell you why the controller gave up. Jumping to logs first is the most common time sink in this domain.

Vocabulary around operators

worth knowing as words
  • Operator capability levels (the OperatorHub model): 1 basic install → 2 seamless upgrades → 3 full lifecycle (backup, failover) → 4 deep insights (metrics, alerts) → 5 auto-pilot (auto-scaling, auto-tuning). A useful yardstick when a scenario asks whether to adopt an operator or run something yourself.
  • OLM (Operator Lifecycle Manager): installs and upgrades operators from catalogs. Not installed here; know the name.
  • Kubebuilder / Operator SDK / controller-runtime: the scaffolding. kubebuilder is on your PATH from make tools, and scaffolding a controller once (kubebuilder init, kubebuilder create api) is genuinely instructive for understanding what operators are made of. It is also beyond what the CNPE tests. Rainy-day material.
  • Reconcile loop hazards worth naming: hot loops (a controller that writes status on every pass and re-triggers itself), requeue backoff, leader election (why an operator's Deployment is usually 1 replica or uses a lease), and cache staleness right after a write.
  • Operator vs controller vs webhook: a controller reconciles; an operator is a controller plus domain knowledge shipped with its CRDs; a webhook intercepts writes synchronously. Different failure modes: a dead controller means nothing converges, a dead webhook (with failurePolicy: Fail) means nothing can be written at all.
Integration is a competency word

"Operators for platform automation and integration" means wiring an operator into the rest of the platform: its CRs live in git (GitOps), its metrics are scraped (ServiceMonitor), its secrets flow to consumers (connection Secrets), its resources are policed (policy engines see CRs like anything else), and its API is exposed to developers through a thinner abstraction (Crossplane, kro, or a Backstage template). Being able to list those five integrations is a complete answer.

Exercises

tick the dot when its check passes

In default (deliberately; the tenant-namespace variant comes last):

kubectl apply -f - <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: pg, namespace: default }
spec:
  instances: 2
  storage: { size: 1Gi }
EOF
kubectl get cluster pg -w

While it converges, watch like a diagnostician: kubectl get events --field-selector involvedObject.name=pg --sort-by=.lastTimestamp, kubectl get cluster pg -o jsonpath='{.status.conditions}' | jq, and kubectl get pods,pvc,svc -l cnpg.io/cluster=pg.

verify: Cluster reports Ready, two instance pods, each with its own PVC (section 1.3's semantics, live), and three Services (rw, ro, r). Name what created each object without guessing: ownerReferences.

kubectl delete pod pg-1 and time what happens. Then scale the honest way, kubectl patch cluster pg --type=merge -p '{"spec":{"instances":3}}', and check observedGeneration catches up to generation before you trust the new status.

verify: a replacement pod appears within seconds and the Cluster's conditions ripple through a degraded state and back to Ready; after the patch, status.observedGeneration == metadata.generation. Two probes, two core behaviours.

The lab ships a booby-trapped manifest: examples/crossplane/pg-cluster.yaml targets team-a, where the default-deny NetworkPolicy silently strangles the instance's access to the API server. Apply it, and diagnose from evidence only: conditions say Initialized True but Ready False, phase "Setting up primary" forever, and kubectl -n team-a logs job/pg-1-initdb ends in dial tcp 10.96.0.1:443: i/o timeout. The file's comments explain why the obvious ipBlock fix fails (ClusterIP is DNAT'd before policy evaluation) and give the Cilium toEntities: [kube-apiserver] answer; do not read them until you have formed your own theory.

verify: after applying the CiliumNetworkPolicy from the comments, the cluster converges Ready in team-a. This exercise is the single best half-hour in domain 3.

Pick two operators you have installed and, for each, name the watched kind, find one owned resource via ownerReferences, and locate the controller deployment.

verify: kubectl api-resources --api-group=<group> and a one-line answer each. The point is transferable fluency: an unfamiliar operator on the exam is the same four questions.

Self-check

answer before opening
You patch a CR and its status still reports the old problem. What do you check before believing it?

status.observedGeneration against metadata.generation. If they differ, the controller has not processed your edit yet and the status describes the previous spec. If they match and the status is still wrong, now it is a real finding.

A namespace has been Terminating for twenty minutes. Method?

Find what is left (kubectl api-resources --verbs=list --namespaced -o name | xargs -n1 kubectl get -n <ns>), find the finalizer on it, and find whether the owning controller is running. The fix is usually reviving the controller; stripping the finalizer is a last resort that leaks whatever it was cleaning up.

Why is deleting an operator-managed pod not a way to "reset" anything?

Because reconciliation is level-based: the controller observes that a pod is missing and creates one, with no memory of the event. If you need a different outcome, change the spec (or the underlying cause); the loop will faithfully restore whatever the spec says forever.

An operator's CRs apply fine and nothing happens at all. Two checks?

Is the controller running (and did it win leader election)? And does its RBAC cover your namespace and your kind? A Forbidden in the controller's logs is the classic silent failure. Only after both: is the CR shaped in a way the controller ignores (a selector, a class field, a paused annotation)?

Name the five ways an operator integrates with the rest of a platform.

Its CRs are delivered by GitOps; its metrics are scraped and alerted on; its connection Secrets are consumed by apps (or synced by ESO); its CRs are subject to policy at admission; and its API is fronted by a thinner developer-facing abstraction (Crossplane/kro/portal template). That list is the "integration" half of the competency.

Docs to know your way around

study time, not exam time
  • kubernetes.io: Operator pattern; Controllers; Owners and Dependents (cascade deletion); Finalizers.
  • github.com/kubernetes/community: the API conventions doc's conditions section, for what Reason/Message actually promise.
  • cloudnative-pg.io: the Cluster API reference, mostly to practise navigating a big operator's docs quickly.
  • Offline: kubectl explain cluster.status.conditions, kubectl get <res> -o jsonpath='{.metadata.ownerReferences}', kubectl api-resources --api-group=<g>.