Supply-chain security in one breath: scan what you build, sign what you ship, and refuse at admission anything unsigned or unscanned. The lab has all three stations, and the pipeline from 2.4 already contains the scan gate.

needsmake core cicd sec

Orientation

competency 5.5 · security scanning in deployment pipelines

Five controls sit between a git push and a running pod in this lab, and each catches a failure class the others cannot. Being able to name all five, and what only each one catches, is the whole competency.

git push
   │
   ├─ 2.4 pipeline scan gate      catches known-bad before publish
   ├─ 5.6 signature               catches tampering and untrusted builders
   ├─ 5.6 admission verification  catches images that bypassed the pipeline entirely
   ├─ 5.3 Pod Security Standards  catches dangerous runtime shape, regardless of provenance
   └─ 5.4 continuous rescanning   catches CVEs published after you shipped
   ▼
running pod

The chain, station by station

scan · sign · verify

Scanning belongs at two points, and the difference is the insight

In the pipeline (a Trivy task, --exit-code 1 --severity HIGH,CRITICAL) it is a gate: it stops a bad image before it exists in the registry. In the cluster (Trivy Operator, section 5.4) it is surveillance: it catches CVEs published after you shipped. Teams that only scan in CI are blind to every CVE younger than their last deploy. Say that sentence in an exam answer and mean it.

Practical gate design, because a gate that always fails gets disabled: scan for HIGH,CRITICAL, consider --ignore-unfixed (you cannot patch what has no fix), keep an auditable ignore file with expiry dates rather than a permanently loosened threshold, and fail the build rather than warning: a warning in CI is a warning nobody reads.

Signing

cosign signs an image digest with a key and pushes the signature to the registry alongside the image (classically as a sha256-<digest>.sig tag; newer versions use an OCI bundle format). The thing being signed is the digest, which is why tags are mutable but signed supply chains are not: verifying :latest means resolving it to a digest first, and whoever can move the tag cannot forge the signature.

Keyless mode is where the ecosystem is going: an OIDC identity (a CI workload's token) gets a short-lived certificate from Fulcio, the signature and certificate are recorded in Rekor, a public transparency log, and verification checks the identity and the log entry instead of a key you must store and rotate. Know those three nouns. Key-pair mode is what you can practise offline and what teaches the mechanics.

Verification at admission is what makes signatures matter

Without it, signing is decoration. A Kyverno image-verification policy holds the public key (or the expected keyless identity), intercepts pod creation, resolves each image, checks its signature, and rejects on failure. It can also mutate the image reference to its digest, which is the underrated half: after verification, the pod runs the exact bytes you verified, not whatever the tag points at later. Gatekeeper does this through external data providers; Kyverno's is built in and is the one to practise.

SLSA, since the word appears everywhere

Levels of provenance rigour, from "you have a build process" to hermetic, attested builds. At exam depth, keep the three documents straight: provenance is an attestation of how an artifact was built; SBOMs say what is inside; signatures say who vouches. Three documents, one trust story; an attestation is itself a signed statement, which is why cosign handles all of them.

Exercises

tick the dot when its check passes

From 2.4 you have build-and-scan and a real image in the registry. Rerun it (tkn pipeline start build-and-scan --last) and this time read the scan step's output in full: tkn pipelinerun logs --last -t scan.

verify: you can state what would have to appear in that output for the run to fail, and where the SBOM the task generates ends up (the workspace, plus a digest in the task's results; kubectl get taskrun <name> -o jsonpath='{.status.results}').

The pipeline pushed demo:v1 under its in-cluster name (kind-registry:5000); from the host the same store answers as localhost:5001, and the digest is identical from both sides because a digest names content, not a location:

cd $(mktemp -d) && cosign generate-key-pair   # passphrase: pick one, remember it
DIGEST=$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest)
cosign sign --key cosign.key --allow-http-registry -y localhost:5001/demo@$DIGEST
cosign verify --key cosign.pub --allow-http-registry localhost:5001/demo@$DIGEST

Then verify a tag instead of the digest and note cosign resolves it to the digest anyway; then try verifying an image you never signed (localhost:5001/validate:v1 exists if you ran make validate) and read the failure.

verify: "The signatures were verified against the specified public key". One success, one refusal, both understood.

Kyverno's current kind for this is ImageValidatingPolicy (kubectl explain imagevalidatingpolicies.spec; the API has moved before and the checking is the exercise). Three facts shape the setup, each bought with an hour of somebody's debugging:

  1. Kyverno fetches signatures itself, from inside the cluster, so the pod images and the policy globs must use the in-cluster registry name (kind-registry:5000, which the lab wires into CoreDNS and containerd), never localhost:5001, which inside the Kyverno pod is the Kyverno pod.
  2. The registry is plain http, and the knob for that is on the engine, not the policy: the lab installs Kyverno with features.registryClient.allowInsecure=true. Without it, every verification dies on server gave HTTP response to HTTPS client before any signature is read.
  3. cosign v3 pushes its new bundle format by default, and Kyverno's verifier currently reads the legacy .sig tag, so sign a second time in legacy form for the engine's benefit: cosign sign --key cosign.key --allow-http-registry -y --use-signing-config=false --new-bundle-format=false --tlog-upload=false localhost:5001/demo@$DIGEST. Version skew between signer and verifier is not a lab quirk; it is the current state of the ecosystem, and recognising "no signatures found" as a format problem is the transferable skill.

The policy: match pods, matchImageReferences glob kind-registry:5000/demo*, one attestor with cosign.key.data holding your cosign.pub (plus cosign.ctlog.insecureIgnoreTlog: true, since the legacy signature skipped the transparency log), and one validation expression from the Kyverno docs' IVP examples: images.containers.map(image, verifyImageSignatures(image, [attestors.keyed])).all(e, e > 0). Then:

kubectl -n team-b run signed --image=kind-registry:5000/demo:v1 --restart=Never -- sleep 60
docker tag busybox:1.37 localhost:5001/demo:unsigned && docker push localhost:5001/demo:unsigned
kubectl -n team-b run unsigned --image=kind-registry:5000/demo:unsigned --restart=Never -- sleep 60
verify: signed admits, unsigned is rejected with your policy's message. Same repo, same cluster, different provenance, different fate. If the signed pod is rejected too, kubectl -n kyverno logs deploy/kyverno-admission-controller | grep -i verif names the real reason (registry scheme, missing .sig, tlog), which is precisely the diagnosis ladder above. Clean up the policy so later sections' pods admit freely.

Five controls now exist between a git push and a running pod in this lab: pipeline scan gate (2.4), registry signature (here), admission verification (here), PSS (5.3), continuous rescanning (5.4). Write one line per control naming the failure class only it catches.

verify: against this: gate catches known-bad before publish; signature catches tampering and untrusted builders; admission catches bypassing the pipeline entirely; PSS catches dangerous runtime shape regardless of provenance; the operator catches CVEs discovered after ship. If your five lines cover those five distinct failure classes, this competency is done.

Self-check

answer before opening
Why scan in the pipeline and in the cluster?

The pipeline gate blocks known-bad before publication; it can say nothing about CVEs disclosed after the build. Continuous in-cluster scanning catches those, on images already running. Only-CI leaves you blind to every CVE younger than your last deploy; only-cluster lets bad images ship in the first place.

Why is signing by digest, not by tag, the point?

A digest names content; a tag is a mutable pointer. Signing the digest means the signature covers exactly those bytes, and anyone who moves the tag cannot forge it. Verification resolves the tag to a digest before checking, and a good admission policy also rewrites the pod's image to that digest so the runtime cannot drift from what you verified.

Signatures exist and someone still deployed an unsigned image. What was missing?

Admission verification. Signing without an enforcement point is decoration: nothing checks it at the door. Add an image-verification policy that matches the registry glob and denies on failure, and make sure it matches the reference form the pods actually use.

Kyverno reports "no signatures found" for an image you definitely signed. Three candidates?

Registry name/reachability (Kyverno resolves from inside the cluster, so localhost is wrong), scheme (plain http needs the engine's insecure flag), and signature format skew (a new-bundle signature where the verifier expects the legacy .sig tag). Also check the transparency-log expectation if you signed without uploading to Rekor.

Distinguish SBOM, provenance and signature in one sentence each.

SBOM: what is inside the artifact. Provenance: how and by whom it was built (a SLSA-style attestation). Signature: who vouches for these exact bytes. Three documents, one trust story; provenance and SBOMs are themselves usually signed attestations.

Docs to know your way around

study time, not exam time
  • docs.sigstore.dev: cosign sign/verify with keys; skim keyless (Fulcio, Rekor) so the words are familiar.
  • kyverno.io: image verification policies (current kind and fields).
  • slsa.dev: the levels table, five minutes.
  • Offline: cosign --help, trivy image --help, kubectl explain imagevalidatingpolicies.spec; the last one is the version arbiter when docs and cluster disagree.