Admission control is the API server's last word. Policy engines are well-organised admission webhooks plus reporting. The lab runs three engines, which sounds redundant and is exactly the point: the same rule in three dialects is the fastest way to learn what is engine-specific and what is admission-generic.
make up secOrientation
Get the request pipeline exact and most of this section is corollaries.
request ──▶ authn ──▶ authz (RBAC) ──▶ mutating admission ──▶ schema validation ──▶ validating admission ──▶ etcd
│ │
LimitRanger, sidecar and PSS, ValidatingAdmissionPolicy,
label injection, Kyverno Kyverno, Gatekeeper webhooks
MutatingPolicy (all see the mutated object)
Run a pod through it. The interesting cases are the ones where a mutation quietly satisfies a validation:
- Validating policies see the object after mutation, so a mutation can satisfy a validation the user never wrote (that is the LimitRange trap below).
- Admission applies at write time only. Tightening a policy never touches existing objects; you need an audit/report mechanism for those.
- Every webhook is an availability dependency of the API server, and
failurePolicychooses which way you fail.
The three dialects
| Kyverno | Gatekeeper (OPA) | ValidatingAdmissionPolicy | |
|---|---|---|---|
| Language | YAML + CEL | Rego | CEL |
| Objects | ClusterPolicy/Policy (classic), and ValidatingPolicy/MutatingPolicy/ImageValidatingPolicy (newer, CEL) | ConstraintTemplate → generated CRD → Constraint | ValidatingAdmissionPolicy + …Binding |
| Can mutate | yes | assign mutations (separate CRDs) | a Mutating counterpart exists in newer versions |
| Can generate | yes (create resources on events) | no | no |
| Reporting | PolicyReport CRs | violation counts on the Constraint, via audit sweeps | none built in |
| Runs as | a webhook (a Deployment you must keep alive) | a webhook | in-process in the API server |
| Enforcement dial | classic: spec.validationFailureAction: Audit|Enforce · newer: spec.validationActions: [Audit|Deny|Warn] | enforcementAction: deny|warn|dryrun | validationActions on the binding |
Kyverno speaks two dialects, and you need both. The classic one is ClusterPolicy/Policy: a list of typed rules (validate, mutate, generate, verifyImages) with match/exclude blocks and a JMESPath-flavoured pattern language, switched between reporting and enforcement by spec.validationFailureAction: Audit|Enforce. It is fully supported and it is what most installed Kyverno runs. The newer dialect (ValidatingPolicy, MutatingPolicy, ImageValidatingPolicy, CEL expressions, spec.validationActions: [Audit|Deny|Warn]) arrived in Kyverno 1.14/1.15 and is what this lab installs. You do not get to choose which one the exam's cluster has, so recognise both, and let kubectl api-resources | grep kyverno plus kubectl explain settle which is in front of you. The lab's examples/kyverno/ pair is the newer form: require-resources.yaml validates that every container declares requests (in Audit mode; flip to Deny to enforce), and add-tenant-label.yaml mutates a cost-centre label onto Deployments at admission.
Gatekeeper is two-step by design: a ConstraintTemplate defines a parameterised policy in Rego and generates a CRD; a Constraint is an instance of that CRD binding it to resources with parameters. That indirection is what makes a library of reusable policies possible, and the gatekeeper-library repo is that library. This is also where the exam's "OPA" lives in practice: Rego inside templates.
ValidatingAdmissionPolicy is native: CEL, no controller to install, no webhook to keep alive, paired with a Binding that says which namespaces and what action. Worth one rep because it is what the other two increasingly compile down to, and because a task could hand it to you.
The rollout choreography is a better exam answer than "apply the policy": ship in Audit, read the reports to find every existing offender, fix them (or exempt them explicitly), then flip to Deny. Kyverno's PolicyReports and Gatekeeper's audit violations exist for exactly that middle step, and the same staged posture appears in PSS (5.3) as warn/audit before enforce.
Operating policy safely
failurePolicy.Failmeans an unreachable webhook blocks the write: secure, and capable of freezing the entire cluster if the policy pods die.Ignoremeans writes proceed unchecked: available, and a bypass window during an outage. There is no correct universal answer; there is only the answer you chose deliberately, plustimeoutSecondsand a well-scopednamespaceSelectorto bound the risk.- Scope your webhooks.
rules(which resources and operations),namespaceSelectorandobjectSelectordecide what is intercepted. Exemptingkube-systemis near-universal, and it is itself a governance decision: it means a cluster-admin path exists that your policies do not see. - Latency. Every intercepted write pays a round trip. Broad wildcard rules on
*/*are how a policy engine becomes a cluster-wide performance problem. - Order. Mutating webhooks run in an order you do not fully control and may run more than once (re-invocation policy), so mutations must be idempotent.
- Where the message goes. A denial arrives in the apply error, verbatim. Write policy messages a stranger can act on: what is wrong, and what to change.
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get policyreports -A # kyverno findings, governance-as-data
kubectl get constraints # gatekeeper, with violation counts
kubectl -n kyverno logs deploy/kyverno-admission-controller | tail -50A cluster that rejects everything mysteriously usually has a policy engine's webhook with a dead backend; the first command finds it, and failurePolicy explains it.
Exercises
One trap to dodge first, and it is the best lesson in this section: the tenant namespaces cannot demonstrate this policy at all, because their LimitRanges inject requests and LimitRanger is a built-in mutating admission plugin that runs before validating webhooks. By the time Kyverno sees a team-a or team-b pod, it already has requests. So drill in a namespace with no LimitRange:
kubectl create ns policy-test
kubectl -n policy-test run naked --image=busybox:1.37 --restart=Never -- sleep 60 # succeeds: Audit
kubectl get policyreports -n policy-test # the violation is recorded instead
kubectl patch validatingpolicy require-resource-requests --type=merge \
-p '{"spec":{"validationActions":["Deny"]}}'
kubectl -n policy-test run naked2 --image=busybox:1.37 --restart=Never -- sleep 60["Audit"] when done.Create a Deployment in team-b, then read it back: kubectl -n team-b get deploy <name> -o jsonpath='{.metadata.labels.cost-centre}' prints the namespace name, put there at admission by add-cost-centre-label. Verify against a deployment created before the policy existed (none of the lab's have the label).
Same rule, Rego dialect, same LimitRange-free namespace. ConstraintTemplate k8srequireresources whose Rego denies containers missing resource requests, then a constraint targeting Pods in policy-test with enforcementAction: warn first. Gatekeeper's library (open-policy-agent/gatekeeper-library) has a containerlimits template to adapt; adapting library Rego rather than writing from scratch is the honest workflow.
kubectl get k8srequireresources -o yaml shows audit violations counted. You can now articulate the trade: CEL policies read like schemas, Rego like code; Kyverno mutates and generates, Gatekeeper's audit and library are mature.Express the same rule as a ValidatingAdmissionPolicy + binding (CEL: object.spec.containers.all(c, has(c.resources.requests)), match Pods, bound to policy-test).
policy-test namespace, so the engines' results stay interpretable.kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations and, for one Kyverno entry, read rules (which resources), namespaceSelector (what is exempt) and failurePolicy.
kube-system in the selector while you are there; policy engines exempting the control plane is itself a governance decision.)Self-check
A "require requests" policy never fires in a tenant namespace. Why?
Its LimitRange defaults the requests during mutating admission, before the validating webhook runs, so the object Kyverno inspects already complies. Admission order (mutate then validate) is the whole explanation, and it generalises to every defaulting mechanism.
failurePolicy: Fail versus Ignore: argue both, then say what bounds the risk.
Fail: no unvalidated writes, at the cost of the API server refusing writes when the webhook is down (a cluster-wide outage from a policy pod). Ignore: writes keep flowing, at the cost of a bypass window. Bound it with a tight namespaceSelector, low timeoutSeconds, exemptions for system namespaces, and running the engine with enough replicas that it is not a single point of failure.
You tighten a policy to Deny. What happens to the workloads that already violate it?
Nothing: admission is write-time. They keep running until their next write (a rollout, a scale, a controller re-create), and then they fail, possibly at 3am. That is why the Audit → report → fix → Enforce sequence exists, and why you check the reports before flipping the dial.
Which engine would you pick to also create a default NetworkPolicy in every new namespace?
Kyverno: generation is a first-class capability there (generate rules that create and keep resources in sync). Gatekeeper validates and can mutate but does not generate; ValidatingAdmissionPolicy only validates. Matching capability to requirement is the whole question.
Why might a platform team prefer ValidatingAdmissionPolicy for a simple rule?
No extra controller, no webhook, no availability dependency, no version skew; the API server evaluates CEL in-process. The trade is a narrower feature set (no mutation in older versions, no generation, no reporting), so it suits simple invariants rather than a governance programme.
Docs to know your way around
- kyverno.io: ValidatingPolicy/MutatingPolicy references and the policy library.
- open-policy-agent.github.io/gatekeeper: ConstraintTemplate walkthrough; the gatekeeper-library repo for adaptable Rego.
- kubernetes.io: Validating Admission Policy, and Dynamic Admission Control for the webhook plumbing (
failurePolicy, selectors, reinvocation). - Offline:
kubectl explain validatingpolicy.spec,kubectl explain validatingadmissionpolicy.spec.validations,kubectl get validatingwebhookconfigurations -o yaml.