5.6 Pipeline security: scan, sign, verify

Supply-chain security in one breath: scan what you build, sign what you ship, and refuse at admission anything unsigned or unscanned. The lab has all three stations, and the pipeline from 2.4 already contains the scan gate.

needsmake core cicd sec

Orientation

competency 5.5 · security scanning in deployment pipelines

Five controls sit between a git push and a running pod in this lab, and each catches a failure class the others cannot. Being able to name all five, and what only each one catches, is the whole competency.

git push
   │
   ├─ 2.4 pipeline scan gate      catches known-bad before publish
   ├─ 5.6 signature               catches tampering and untrusted builders
   ├─ 5.6 admission verification  catches images that bypassed the pipeline entirely
   ├─ 5.3 Pod Security Standards  catches dangerous runtime settings, regardless of provenance
   └─ 5.4 continuous rescanning   catches CVEs published after you shipped
   ▼
running pod

The chain, station by station

scan · sign · verify

Scanning belongs at two points, and the difference is the insight

In the pipeline (a Trivy task, --exit-code 1 --severity HIGH,CRITICAL) it is a gate: it stops a bad image before it exists in the registry. In the cluster (Trivy Operator, section 5.4) it is surveillance: it catches CVEs published after you shipped. Teams that only scan in CI are blind to every CVE younger than their last deploy. That distinction is a strong exam answer.

Gate design matters, because a gate that always fails gets disabled. Scan for HIGH,CRITICAL and consider --ignore-unfixed (you cannot patch what has no fix). Keep an auditable ignore file with expiry dates rather than a permanently loosened threshold. Fail the build rather than warning: a warning in CI is a warning nobody reads.

Signing

cosign signs an image digest with a key and pushes the signature to the registry alongside the image (classically as a sha256-<digest>.sig tag; newer versions use an OCI bundle format). The thing being signed is the digest, which is why tags are mutable but signed supply chains are not: verifying :latest means resolving it to a digest first, and whoever can move the tag cannot forge the signature.

Keyless mode is where the ecosystem is going: an OIDC identity (a CI workload's token) gets a short-lived certificate from Fulcio, and the signature and certificate are recorded in Rekor, a public transparency log. Verification checks the identity and the log entry instead of a key you must store and rotate. Know those three nouns. Key-pair mode is what you can practice offline and what teaches the mechanics.

Verification at admission is what makes signatures matter

Without it, signing is decoration. A Kyverno image-verification policy holds the public key (or the expected keyless identity), intercepts pod creation, resolves each image, checks its signature, and rejects on failure. It can also mutate the image reference to its digest: after verification, the pod runs the exact bytes you verified, not whatever the tag points at later. Gatekeeper does this through external data providers; Kyverno's is built in and is the one to practice.

SLSA, since the word appears everywhere

Levels of provenance rigor, from "you have a build process" to hermetic, attested builds. At exam depth, keep the three documents straight: provenance is an attestation of how an artifact was built; SBOMs say what is inside; signatures say who vouches. Three documents, one trust story; an attestation is itself a signed statement, which is why cosign handles all of them.

Stations, tools and the flags that gate

source, build, package, deploy, run; cosign and trivy flags

Station	Threat	Control	Tool words
Source	malicious or accidental commit, leaked credential	branch protection, signed commits, secret scanning, dependency review	gitleaks or trivy's secret scanner, Dependabot/Renovate, OpenSSF Scorecard
Build	compromised or unpinned dependency, tampered build	pinned base images by digest, hermetic and reproducible builds, provenance	SLSA levels, Tekton Chains, distroless or Chainguard bases
Package	known vulnerabilities shipped, unsigned artefact	scan gate, SBOM, signature, attestations in the registry	trivy, syft/grype, cosign sign/attest
Deploy	image that bypassed the pipeline, tag moved after verification	admission verification, digest pinning, registry allow-list	Kyverno ImageValidatingPolicy, Sigstore policy-controller, Gatekeeper + Ratify
Run	CVE published after ship, runtime compromise	continuous rescanning, runtime detection, PSS	Trivy Operator, Falco, Pod Security admission

cosign flags you will type

Key-pair. cosign generate-key-pair (or --kms awskms://..., gcpkms://, azurekms://, hashivault://, k8s://ns/secret to keep the key out of files); cosign sign --key cosign.key <image@sha256:...>; cosign verify --key cosign.pub <image>. Success prints Verified OK and the claims lines (The cosign claims were validated, The signatures were verified against the specified public key) followed by the signature payload JSON; a tag is resolved to a digest first and the payload's docker-manifest-digest is checked against it.
Keyless. cosign sign <image> with an OIDC login or a CI identity token (--identity-token, GitHub Actions' id-token: write permission) gets a short-lived certificate from Fulcio and logs to Rekor. Verification must pin the identity: cosign verify --certificate-identity <subject> --certificate-oidc-issuer <issuer URL> <image>, or the -regexp variants for a workflow path like https://github.com/org/repo/.github/workflows/build.yaml@refs/heads/main with issuer https://token.actions.githubusercontent.com. Without the identity flags the command refuses to run; that is the point.
Offline and private. --insecure-ignore-tlog skips the Rekor check (needed when you signed with --tlog-upload=false); --offline with a bundle; --certificate-chain or --ca-roots for a private Fulcio; --rekor-url for a private Rekor. -a key=value at sign time adds annotations checked with the same flag at verify time.
Attestations. cosign attest --predicate file --type slsaprovenance1|spdxjson|cyclonedx|vuln|custom --key ... <image>; cosign verify-attestation --type <type> --key ... <image>, optionally with --policy policy.cue or a Rego file to assert on predicate content (a maximum severity, a builder id). cosign tree <image> lists what is attached; cosign download attestation fetches it.
Storage formats. Signatures live as OCI artefacts next to the image: the legacy tag form sha256-<digest>.sig and .att, or the newer bundle stored via OCI 1.1 referrers (cosign v3 default). A verifier that only understands one form reports no signatures found for an image signed in the other. The lab's --new-bundle-format=false is that mismatch made visible.

trivy gate flags

--severity HIGH,CRITICAL filters; --exit-code 1 makes findings fail the step (default exit code is 0 even with findings); --ignore-unfixed is shorthand for --ignore-status affected,will_not_fix,fix_deferred,end_of_life; --scanners vuln,secret,misconfig,license picks scanners (secret scanning is on by default for images and slows large ones); --pkg-types os,library.
.trivyignore holds one ID per line with optional exp:2026-12-31 expiry; .trivyignore.yaml (passed with --ignorefile) separates vulnerabilities, misconfigurations, secrets and licenses with statements; --ignore-policy policy.rego filters with OPA (package trivy, rule ignore); --vex applies VEX statements. Expiry dates are what make an ignore auditable.
Outputs a pipeline stores: --format json|sarif|cyclonedx|spdx-json|template --output file; trivy image --format cyclonedx is the SBOM, trivy sbom rescans one later, trivy k8s scans a live cluster, trivy config and trivy fs cover manifests and repos.

Hygiene that admission cannot give you

Pin base images by digest (FROM alpine@sha256:...) and let a bot bump them; prefer distroless or Chainguard bases so the scanner has less to find and the attacker has no shell; multi-stage builds so build tools never ship; registry rules (immutable tags, a pull-through cache, an allow-list enforced by policy on image prefixes); dependency lockfiles committed and checked. OpenSSF Scorecard automates the repository side with named checks: Pinned-Dependencies, Token-Permissions, Branch-Protection, Signed-Releases, SBOM, Dangerous-Workflow, Vulnerabilities, Code-Review. When a task asks for "controls in the pipeline" beyond scanning, those names are the vocabulary.

Provenance: SLSA, Tekton Chains and the verification policies

levels, chains-config, IVP, policy-controller, Ratify

SLSA v1.0 Build track

Level	Requirement	Stops
Build L0	nothing	nothing
Build L1	provenance exists: the build platform generates a provenance attestation saying how the package was built (source, builder, parameters); consumers can read it	mistakes and documentation drift, not attackers
Build L2	hosted build platform generates and signs the provenance; the signing key is held by the platform, not the user	tampering after the build (forged or edited provenance)
Build L3	hardened build platform: builds isolated from each other and from user influence, signing material not accessible to the build steps, so the run itself cannot forge its provenance	tampering during the build

v1.0 has only the Build track; later versions add Source and build-environment tracks built the same way. Provenance is an in-toto Statement with predicateType: https://slsa.dev/provenance/v1 whose predicate has buildDefinition (buildType, externalParameters, resolvedDependencies) and runDetails (builder.id, metadata). Verification means checking the signature, the builder.id against the builders you trust, and the source in externalParameters against the repository you expect.

Tekton Chains

Chains watches TaskRuns and PipelineRuns, and on completion formats, signs and uploads provenance. Everything is in the chains-config ConfigMap in tekton-chains:

artifacts.taskrun.format / artifacts.pipelinerun.format: in-toto (alias slsa/v1, produces SLSA v0.2 provenance) or slsa/v2alpha3 / slsa/v2alpha4 (SLSA v1.0 provenance; v2alpha4 reads type-hinted results from StepActions). New setups want slsa/v2alpha3 or newer.
artifacts.*.storage: tekton (as annotations on the run), oci (as attestations next to the image), gcs, docdb, grafeas, archivista; comma-separated for several. artifacts.oci.storage: oci is what makes cosign verify-attestation on the image find anything.
artifacts.*.signer: x509 (a cosign.key in the signing-secrets Secret, or keyless via signers.x509.fulcio.enabled: true) or kms (signers.kms.kmsref); none stores unsigned provenance.
transparency.enabled: true uploads to Rekor (transparency.url); manual uploads only runs annotated chains.tekton.dev/transparency-upload: "true".
Type hinting is how Chains knows what was built: results named IMAGE_URL and IMAGE_DIGEST (or *_IMAGE_URL/*_IMAGE_DIGEST pairs, or an *ARTIFACT_OUTPUTS object result with uri and digest). A Task that pushes an image but emits no such result gets no OCI attestation, and the run is annotated chains.tekton.dev/signed: "true" anyway because the run payload itself was signed. That annotation is the first thing to read when "Chains did not sign my image"; the second is the controller log for a missing result or a registry auth error. artifacts.pipelinerun.enable-deep-inspection: "true" makes pipeline-level provenance include task-level results.

Verification at admission, three ways

Engine	Object	The fields
Kyverno	`ImageValidatingPolicy` (`policies.kyverno.io/v1`)	`matchImageReferences[].glob\|expression`; `attestors[].cosign.key.data` or `.keyless.identities[].subject/issuer` (plus `subjectRegExp`/`issuerRegExp`), `.ctlog` for Rekor settings, or `.certificate`; `attestations[]` as `referrer.type` or `intoto.type`; `validations[].expression` using `images.containers.map(image, verifyImageSignatures(image, [attestors.x])).all(e, e > 0)`, `verifyAttestationSignatures(image, attestations.sbom, [...])` and `extractPayload(image, attestations.sbom).bomFormat == 'CycloneDX'`; `mutateDigest: true` rewrites tags to digests, `verifyDigest` refuses tags, `required: true` fails images no rule covered; `credentials.secrets` because IVP does not read the pod's `imagePullSecrets`
Sigstore policy-controller	`ClusterImagePolicy` (`policy.sigstore.dev/v1beta1`)	`images[].glob`; `authorities[]` with `key.data`, `keyless.identities[].issuer/subject` (+ `url` for Fulcio), or `static.action: pass\|fail`; `attestations[]` with `predicateType` and a `policy` in cue or rego; `mode: enforce\|warn`. Only namespaces labeled `policy.sigstore.dev/include: "true"` are checked; every matching policy must pass (AND), any one authority inside a policy suffices (OR); it also resolves tags to digests
Gatekeeper	`Provider` (external data) + a ConstraintTemplate calling `external_data`	Ratify (or cosign-gatekeeper-provider) runs in-cluster, verifies signatures and attestations against its own `Verifier` and `Store` config, and returns a verdict per image; the Constraint denies on failure. Same effect, one more moving part

Digest pinning closes the last gap: verify the digest, then run the digest. Kyverno's mutateDigest, policy-controller's tag resolution and Argo CD's image updater writing digests into git all do it; a Deployment that still says :v1 after admission means whoever moves the tag decides what runs.

How this gets tested

"Sign this image and make the cluster refuse unsigned ones" is cosign plus one of the three objects above, with the registry name as the pods see it and the engine's insecure-registry setting if the lab registry is plain HTTP. "Generate provenance for this pipeline" is Chains configuration plus IMAGE_URL/IMAGE_DIGEST results in the Task. "Fail the build on CRITICAL-severity CVEs but not on unfixable ones" is --severity CRITICAL --exit-code 1 --ignore-unfixed with a dated ignore file for the exceptions. Each is graded on the artefact (a signature you can verify, an attestation you can download, a denied pod), so end by producing that artefact.

Exercises

tick the dot when its check passes

From 2.4 you have build-and-scan and a real image in the registry. Rerun it (tkn pipeline start build-and-scan --last) and this time read the scan step's output in full: tkn pipelinerun logs --last -t scan.

verify: you can state what would have to appear in that output for the run to fail, and where the SBOM the task generates ends up (the workspace, plus a digest in the task's results; kubectl get taskrun <name> -o jsonpath='{.status.results}').

The pipeline pushed demo:v1 under its in-cluster name (kind-registry:5000); from the host the same store answers as localhost:5001, and the digest is identical from both sides because a digest names content, not a location:

cd $(mktemp -d) && cosign generate-key-pair   # passphrase: pick one, remember it
DIGEST=$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest)
cosign sign --key cosign.key --allow-http-registry -y localhost:5001/demo@$DIGEST
cosign verify --key cosign.pub --allow-http-registry localhost:5001/demo@$DIGEST

outputcaptured 2026-08-26

$ cd $(mktemp -d) && cosign generate-key-pair   # passphrase: pick one, remember it
Private key written to cosign.key
Public key written to cosign.pub
$ DIGEST=$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest)
$ cosign sign --key cosign.key --allow-http-registry -y localhost:5001/demo@$DIGEST
Signing artifact...
Pushing signature to: localhost:5001/demo
$ cosign verify --key cosign.pub --allow-http-registry localhost:5001/demo@$DIGEST

Verification for localhost:5001/demo@sha256:33213719f2b030a8a1260d1c5358f55524f8f690a790e35b14b00c1f50b46ddf --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"localhost:5001/demo@sha256:33213719f2b030a8a1260d1c5358f55524f8f690a790e35b14b00c1f50b46ddf"},"image":{"docker-manifest-digest":"sha256:33213719f2b030a8a1260d1c5358f55524f8f690a790e35b14b00c1f50b46ddf"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}]

Then verify a tag instead of the digest and note cosign resolves it to the digest anyway; then try verifying an image you never signed (localhost:5001/validate:v1 exists if you ran make validate) and read the failure.

verify: "The signatures were verified against the specified public key". One success, one refusal, both understood.

Kyverno's current kind for this is ImageValidatingPolicy (kubectl explain imagevalidatingpolicies.spec; the API has moved before and the checking is the exercise). Three facts govern the setup:

Kyverno fetches signatures itself, from inside the cluster, so the pod images and the policy globs must use the in-cluster registry name (kind-registry:5000, which the lab wires into CoreDNS and containerd), never localhost:5001, which inside the Kyverno pod is the Kyverno pod.
The registry is plain http, and the knob for that is on the engine, not the policy: the lab installs Kyverno with features.registryClient.allowInsecure=true. Without it, every verification dies on server gave HTTP response to HTTPS client before any signature is read.
cosign v3 pushes its new bundle format by default, and Kyverno's verifier currently reads the legacy .sig tag, so sign a second time in legacy form for the engine's benefit: cosign sign --key cosign.key --allow-http-registry -y --use-signing-config=false --new-bundle-format=false --tlog-upload=false localhost:5001/demo@$DIGEST. Version skew between signer and verifier is not a lab quirk; it is the current state of the ecosystem, and recognizing "no signatures found" as a format problem is the transferable skill.

The policy: match pods, matchImageReferences glob kind-registry:5000/demo*, one attestor with cosign.key.data holding your cosign.pub (plus cosign.ctlog.insecureIgnoreTlog: true, since the legacy signature skipped the transparency log), and one validation expression from the Kyverno docs' IVP examples: images.containers.map(image, verifyImageSignatures(image, [attestors.keyed])).all(e, e > 0). Then:

kubectl -n team-b run signed --image=kind-registry:5000/demo:v1 --restart=Never -- sleep 60
docker tag busybox:1.37 localhost:5001/demo:unsigned && docker push localhost:5001/demo:unsigned
kubectl -n team-b run unsigned --image=kind-registry:5000/demo:unsigned --restart=Never -- sleep 60

outputcaptured 2026-08-26

$ kubectl -n team-b run signed --image=kind-registry:5000/demo:v1 --restart=Never -- sleep 60
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "signed" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "signed" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "signed" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "signed" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/signed created
$ docker tag busybox:1.37 localhost:5001/demo:unsigned && docker push localhost:5001/demo:unsigned
The push refers to repository [localhost:5001/demo]
436a1b1fd078: Layer already exists
unsigned: digest: sha256:7a3ebe5bfd1a4a19797d20b0c0bb39d44393e9a03fd852c0865b0f540d868df0 size: 610

 Info -> Not all multiplatform-content is present and only the available single-platform image was pushed
         sha256:9db7b59979c38555a39def84a31fb98b5296952f9e3afd4f6f11f05b07adfab0 -> sha256:7a3ebe5bfd1a4a19797d20b0c0bb39d44393e9a03fd852c0865b0f540d868df0
$ kubectl -n team-b run unsigned --image=kind-registry:5000/demo:unsigned --restart=Never -- sleep 60
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "unsigned" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "unsigned" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "unsigned" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "unsigned" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Error from server: admission webhook "ivpol.validate.kyverno.svc-fail" denied the request: Policy require-signed-demo failed: demo images must be signed with the platform key

verify: signed admits, unsigned is rejected with your policy's message. If the signed pod is rejected too, kubectl -n kyverno logs deploy/kyverno-admission-controller | grep -i verif names the real reason (registry scheme, missing .sig, tlog), which is precisely the diagnosis ladder above. Clean up the policy so later sections' pods admit freely.

Five controls now exist between a git push and a running pod in this lab: pipeline scan gate (2.4), registry signature (here), admission verification (here), PSS (5.3), continuous rescanning (5.4). Write one line per control naming the failure class only it catches.

verify: against this: gate catches known-bad before publish; signature catches tampering and untrusted builders; admission catches bypassing the pipeline entirely; PSS catches dangerous runtime settings regardless of provenance; the operator catches CVEs discovered after ship. If your five lines cover those five distinct failure classes, this competency is done.

Keyless verification without identity flags is not verification: it asks "was this signed by anyone", which is a question with a useless answer. cosign refuses to do it. The second and third commands are the pair worth knowing: a wrong identity fails with the identity that is actually on the certificate printed in the error, which is how you find the right flags for an image you did not build.

# no identity at all: cosign will not answer the question
cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 2>&1 | head -3
# an identity that does not match: the error names the one that is actually on the certificate
cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 \
  --certificate-identity-regexp='^https://github.com/sigstore/cosign/.*' \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com 2>&1 | head -4
# the identity the certificate really carries
cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 \
  --certificate-identity=keyless@projectsigstore.iam.gserviceaccount.com \
  --certificate-oidc-issuer=https://accounts.google.com 2>&1 | head -6

outputcaptured 2026-09-13

$ # no identity at all: cosign will not answer the question
$ cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 2>&1 | head -3
Error: --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode
error during command execution: --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode
$ # an identity that does not match: the error names the one that is actually on the certificate
$ cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 \
    --certificate-identity-regexp='^https://github.com/sigstore/cosign/.*' \
    --certificate-oidc-issuer=https://token.actions.githubusercontent.com 2>&1 | head -4
Error: no matching signatures: error verifying bundle: empty key
 none of the expected identities matched what was in the certificate, got subjects [keyless@projectsigstore.iam.gserviceaccount.com] with issuer https://accounts.google.com
error during command execution: no matching signatures: error verifying bundle: empty key
 none of the expected identities matched what was in the certificate, got subjects [keyless@projectsigstore.iam.gserviceaccount.com] with issuer https://accounts.google.com
$ # the identity the certificate really carries
$ cosign verify ghcr.io/sigstore/cosign/cosign:v2.4.1 \
    --certificate-identity=keyless@projectsigstore.iam.gserviceaccount.com \
    --certificate-oidc-issuer=https://accounts.google.com 2>&1 | head -6

Verification for ghcr.io/sigstore/cosign/cosign:v2.4.1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

verify: the first command refuses with --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode. The second fails with

none of the expected identities matched what was in the certificate, got subjects [keyless@projectsigstore.iam.gserviceaccount.com] with issuer https://accounts.google.com

, which hands you the answer. The third, using exactly that subject and issuer, prints Verification for ... and the list of checks performed. Note that this release is signed by a Google service account rather than a GitHub Actions workflow, which is why guessing the GitHub identity fails.

A signature says who built it. An attestation says what it contains, signed. Attach an SBOM as an attestation, look at the tree, and pull the predicate back out, because that round trip is the whole supply-chain story.

# generate-key-pair prompts for a passphrase; an empty COSIGN_PASSWORD is what makes it non-interactive
[ -f /tmp/cosign.key ] || COSIGN_PASSWORD='' cosign generate-key-pair --output-key-prefix /tmp/cosign
skopeo copy --dest-tls-verify=false docker://ghcr.io/stefanprodan/podinfo:6.7.1 docker://localhost:5001/demo:v1
trivy image --format cyclonedx --output /tmp/sbom.json localhost:5001/demo:v1
COSIGN_PASSWORD='' cosign attest --yes --predicate /tmp/sbom.json --type cyclonedx --key /tmp/cosign.key --allow-insecure-registry localhost:5001/demo:v1
cosign tree --allow-insecure-registry localhost:5001/demo:v1
cosign verify-attestation --type cyclonedx --key /tmp/cosign.pub --allow-insecure-registry localhost:5001/demo:v1 | jq -r .payload | base64 -d | jq '{predicateType, components: (.predicate.components | length)}'

outputcaptured 2026-09-13

$ # generate-key-pair prompts for a passphrase; an empty COSIGN_PASSWORD is what makes it non-interactive
$ [ -f /tmp/cosign.key ] || COSIGN_PASSWORD='' cosign generate-key-pair --output-key-prefix /tmp/cosign
Private key written to /tmp/cosign.key
Public key written to /tmp/cosign.pub
$ skopeo copy --dest-tls-verify=false docker://ghcr.io/stefanprodan/podinfo:6.7.1 docker://localhost:5001/demo:v1
Getting image source signatures
Copying blob sha256:7e517c53cd4ca936c3bc3d3dc66419278a680214f37444ad0ad1b8bd65968940
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170
Copying blob sha256:2bf70b2badb9fa00aa36d50d61c6b9cc094d7a18fb4a3c7a91ce950dd7202900
Copying blob sha256:35d14af0e611499048effc4d39a2341345f96dd9103c8c186b7cc239a687de5c
Copying blob sha256:31eaa2883f08a0ce854d061714a111f052444a3ba1b2323719cb12b872a0dbe3
Copying blob sha256:47f87f374862e0c7555781b116554936999ad2e8e4c6a4e7417688ffe57d8b88
Copying config sha256:c875de4397634b0dc138954cbe2c11b9d86d948b98b17bd9fc3227b3ad992eae
Writing manifest to image destination
$ trivy image --format cyclonedx --output /tmp/sbom.json localhost:5001/demo:v1
2026-09-13T00:11:59-04:00	INFO	"--format cyclonedx" disables security scanning. Specify "--scanners vuln" explicitly if you want to include vulnerabilities in the "cyclonedx" report.
2026-09-13T00:12:00-04:00	INFO	Detected OS	family="alpine" version="3.20.3"
2026-09-13T00:12:00-04:00	INFO	Number of language-specific files	num=2

📣 Notices:
  - Version 0.74.0 of Trivy is now available, current version is 0.73.0

To suppress version checks, run Trivy scans with the --skip-version-check flag
$ COSIGN_PASSWORD='' cosign attest --yes --predicate /tmp/sbom.json --type cyclonedx --key /tmp/cosign.key --allow-insecure-registry localhost:5001/demo:v1
WARNING: Image reference localhost:5001/demo:v1 uses a tag, not a digest, to identify the image to sign.
    This can lead you to sign a different image than the intended one. Please use a
    digest (example.com/ubuntu@sha256:abc123...) rather than tag
    (example.com/ubuntu:latest) for the input to cosign. The ability to refer to
    images by tag will be removed in a future release.

Using payload from: /tmp/sbom.json
Signing artifact...
$ cosign tree --allow-insecure-registry localhost:5001/demo:v1
📦 Supply Chain Security Related artifacts for an image: localhost:5001/demo:v1
└── 🔐 Signatures for an image tag: localhost:5001/demo:sha256-ed73e9871dfba28276b28daab3463cc2f9b3c15337dfe0445e8b0f79694e0cd6.sig
   └── 🍒 sha256:a217ae5b0b2d194cf70785da9ff77db49a8040f86cd50d518766deecce261c91
└── 🔗 https://cyclonedx.org/bom artifacts via OCI referrer: localhost:5001/demo@sha256:53833ef158bb16c0652edf40242d2f5a0da861f0b447271c5f55170da4a294da
   └── 🍒 sha256:6ff7639af536922fe0b7b8d5ee81bcd56fc432099b04a2bfb5e96fc8ef167558
└── 🔗 https://cyclonedx.org/bom artifacts via OCI referrer: localhost:5001/demo@sha256:c4ebc6c30e427e7656271ffaae5694e47c51831d2dfbcda1a306603877e4e620
   └── 🍒 sha256:9f5427192055dde8d943dfcf0e35900cebacef1b820258a2d06a17da84b0ceb4
$ cosign verify-attestation --type cyclonedx --key /tmp/cosign.pub --allow-insecure-registry localhost:5001/demo:v1 | jq -r .payload | base64 -d | jq '{predicateType, components: (.predicate.components | length)}'

Verification for localhost:5001/demo:v1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
{
  "predicateType": "https://cyclonedx.org/bom",
  "components": 114
}

verify: the tree shows an attestation alongside the image, and the verified payload's predicateType is the CycloneDX one with the component list intact.

A tag is a mutable pointer, so verifying a tag and then running it is a race. mutateDigest makes the policy rewrite the pod spec to the digest it verified, which closes the gap in the object itself.

[ -f /tmp/cosign.pub ] || COSIGN_PASSWORD='' cosign generate-key-pair --output-key-prefix /tmp/cosign
# Kyverno's verifier reads the legacy .sig tag, so the image needs a signature in that format
DIGEST=$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest)
COSIGN_PASSWORD='' cosign sign --key /tmp/cosign.key --allow-http-registry -y --use-signing-config=false --new-bundle-format=false --tlog-upload=false localhost:5001/demo@$DIGEST
kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: pin-digest }
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-and-pin
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [default]
      verifyImages:
        - imageReferences: ["kind-registry:5000/demo:v1"]
          mutateDigest: true
          verifyDigest: false
          required: false
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
$(sed 's/^/                      /' /tmp/cosign.pub)
                    rekor:
                      ignoreTlog: true
                    ctlog:
                      ignoreSCT: true
EOF
sleep 20
kubectl -n default run pinned --image=kind-registry:5000/demo:v1 --restart=Never
kubectl -n default get pod pinned -o jsonpath='{.spec.containers[0].image}{"\n"}'
kubectl -n default delete pod pinned --ignore-not-found
kubectl delete clusterpolicy pin-digest

outputcaptured 2026-09-13

$ [ -f /tmp/cosign.pub ] || COSIGN_PASSWORD='' cosign generate-key-pair --output-key-prefix /tmp/cosign
$ # Kyverno's verifier reads the legacy .sig tag, so the image needs a signature in that format
$ DIGEST=$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest)
$ COSIGN_PASSWORD='' cosign sign --key /tmp/cosign.key --allow-http-registry -y --use-signing-config=false --new-bundle-format=false --tlog-upload=false localhost:5001/demo@$DIGEST
Flag --new-bundle-format has been deprecated, this will be the only supported format in future versions
Flag --tlog-upload has been deprecated, prefer using a --signing-config file with no transparency log services
Signing artifact...
Pushing signature to: localhost:5001/demo
$ kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: pin-digest }
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-and-pin
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [default]
      verifyImages:
        - imageReferences: ["kind-registry:5000/demo:v1"]
          mutateDigest: true
          verifyDigest: false
          required: false
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
$(sed 's/^/                      /' /tmp/cosign.pub)
                    rekor:
                      ignoreTlog: true
                    ctlog:
                      ignoreSCT: true
EOF
Warning: kyverno.io/v1 ClusterPolicy is deprecated and will be removed in a future release; migrate to ValidatingPolicy, MutatingPolicy, GeneratingPolicy or ImageValidatingPolicy (policies.kyverno.io), see https://kyverno.io/docs/guides/migration-to-cel/
clusterpolicy.kyverno.io/pin-digest created
$ sleep 20
$ kubectl -n default run pinned --image=kind-registry:5000/demo:v1 --restart=Never
pod/pinned created
$ kubectl -n default get pod pinned -o jsonpath='{.spec.containers[0].image}{"\n"}'
kind-registry:5000/demo:v1@sha256:ed73e9871dfba28276b28daab3463cc2f9b3c15337dfe0445e8b0f79694e0cd6
$ kubectl -n default delete pod pinned --ignore-not-found
pod "pinned" deleted from default namespace
$ kubectl delete clusterpolicy pin-digest
Warning: kyverno.io/v1 ClusterPolicy is deprecated and will be removed in a future release; migrate to ValidatingPolicy, MutatingPolicy, GeneratingPolicy or ImageValidatingPolicy (policies.kyverno.io), see https://kyverno.io/docs/guides/migration-to-cel/
clusterpolicy.kyverno.io "pin-digest" deleted

verify: the running pod's image is ...@sha256:... even though you asked for a tag.

A scanner gate is a policy decision expressed as flags, and each flag changes who gets blocked. Run the same scan three ways and watch the verdict move without the image changing at all.

trivy image -q --exit-code 1 --severity CRITICAL localhost:5001/demo:v1 > /tmp/scan-strict.txt 2>&1; echo "strict exit: $?"
trivy image -q --exit-code 1 --severity CRITICAL --ignore-unfixed localhost:5001/demo:v1 > /tmp/scan-unfixed.txt 2>&1; echo "ignore-unfixed exit: $?"
# every CRITICAL id, not just the first: suppressing one of three moves no exit code
trivy image -q --severity CRITICAL --format json localhost:5001/demo:v1 | jq -r '[.Results[].Vulnerabilities // [] | .[].VulnerabilityID] | unique | .[]' > /tmp/cves.txt
cat /tmp/cves.txt
sed 's/$/ exp:2020-01-01/' /tmp/cves.txt > /tmp/.trivyignore
trivy image -q --exit-code 1 --severity CRITICAL --ignorefile /tmp/.trivyignore localhost:5001/demo:v1 > /tmp/scan-expired.txt 2>&1; echo "expired-ignore exit: $?"
sed 's/$/ exp:2099-01-01/' /tmp/cves.txt > /tmp/.trivyignore
trivy image -q --exit-code 1 --severity CRITICAL --ignorefile /tmp/.trivyignore localhost:5001/demo:v1 > /tmp/scan-live.txt 2>&1; echo "live-ignore exit: $?"
tail -5 /tmp/scan-live.txt

outputcaptured 2026-09-13

$ trivy image -q --exit-code 1 --severity CRITICAL localhost:5001/demo:v1 > /tmp/scan-strict.txt 2>&1; echo "strict exit: $?"
strict exit: 1
$ trivy image -q --exit-code 1 --severity CRITICAL --ignore-unfixed localhost:5001/demo:v1 > /tmp/scan-unfixed.txt 2>&1; echo "ignore-unfixed exit: $?"
ignore-unfixed exit: 1
$ # every CRITICAL id, not just the first: suppressing one of three moves no exit code
$ trivy image -q --severity CRITICAL --format json localhost:5001/demo:v1 | jq -r '[.Results[].Vulnerabilities // [] | .[].VulnerabilityID] | unique | .[]' > /tmp/cves.txt
$ cat /tmp/cves.txt
CVE-2025-68121
CVE-2026-31789
CVE-2026-33186
$ sed 's/$/ exp:2020-01-01/' /tmp/cves.txt > /tmp/.trivyignore
$ trivy image -q --exit-code 1 --severity CRITICAL --ignorefile /tmp/.trivyignore localhost:5001/demo:v1 > /tmp/scan-expired.txt 2>&1; echo "expired-ignore exit: $?"
expired-ignore exit: 1
$ sed 's/$/ exp:2099-01-01/' /tmp/cves.txt > /tmp/.trivyignore
$ trivy image -q --exit-code 1 --severity CRITICAL --ignorefile /tmp/.trivyignore localhost:5001/demo:v1 > /tmp/scan-live.txt 2>&1; echo "live-ignore exit: $?"
live-ignore exit: 0
$ tail -5 /tmp/scan-live.txt
└────────────────────────────────────────┴──────────┴─────────────────┴─────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

verify: four exit lines, the three CRITICAL ids the ignorefile is built from, and the last five lines of the final report. strict is 1. ignore-unfixed is still 1, because every CRITICAL here has a fixed version and nothing is dropped. The expired ignorefile is 1: an entry dated in the past no longer suppresses. The same file dated 2099 gives 0, and its report ends on the legend rather than a findings table. Redirecting each report to a file is deliberate: piping a scan into head or tail reports the exit code of the pager, not of trivy, and fifty lines of table hide the one number the gate is made of.

Self-check

answer before opening

Why scan in the pipeline and in the cluster?

The pipeline gate blocks known-bad before publication; it can say nothing about CVEs disclosed after the build. Continuous in-cluster scanning catches those, on images already running. Only-CI leaves you blind to every CVE younger than your last deploy; only-cluster lets bad images ship in the first place.

Why is signing by digest, not by tag, the point?

A digest names content; a tag is a mutable pointer. Signing the digest means the signature covers exactly those bytes, and anyone who moves the tag cannot forge it. Verification resolves the tag to a digest before checking, and a good admission policy also rewrites the pod's image to that digest so the runtime cannot drift from what you verified.

Signatures exist and someone still deployed an unsigned image. What was missing?

Admission verification. Signing without an enforcement point is decoration: nothing checks it at the door. Add an image-verification policy that matches the registry glob and denies on failure, and make sure it matches the reference form the pods actually use.

Kyverno reports "no signatures found" for an image you definitely signed. Three candidates?

Registry name/reachability (Kyverno resolves from inside the cluster, so localhost is wrong), scheme (plain http needs the engine's insecure flag), and signature format skew (a new-bundle signature where the verifier expects the legacy .sig tag). Also check the transparency-log expectation if you signed without uploading to Rekor.

Distinguish SBOM, provenance and signature in one sentence each.

SBOM: what is inside the artifact. Provenance: how and by whom it was built (a SLSA-style attestation). Signature: who vouches for these exact bytes. Three documents, one trust story; provenance and SBOMs are themselves usually signed attestations.

Keyless verification: which two flags are mandatory, and what does each pin?

--certificate-identity (or --certificate-identity-regexp) pins the SAN in the Fulcio certificate: the signer's email or, for CI, the workflow URL such as https://github.com/org/repo/.github/workflows/build.yaml@refs/heads/main. --certificate-oidc-issuer pins who authenticated that identity, for example https://token.actions.githubusercontent.com. Without both, any Fulcio-issued certificate would verify, so cosign refuses to run.

Tekton Chains signs the TaskRun (annotation chains.tekton.dev/signed: "true") but cosign verify-attestation on the image finds nothing. Two configuration causes?

The Task does not emit type-hinted results (IMAGE_URL and IMAGE_DIGEST, or *ARTIFACT_OUTPUTS), so Chains has no image to attach provenance to; or artifacts.taskrun.storage / artifacts.oci.storage does not include oci, so the attestation went to the tekton backend (run annotations) only. Check the results on the run and the chains-config ConfigMap, then the controller log.

What distinguishes SLSA Build L2 from L3, and which one does a signed provenance from a shared CI runner reach?

L2 requires a hosted platform to generate and sign the provenance with a key the user cannot access. L3 additionally requires the build to be isolated from other builds and from user-controlled steps so the run cannot influence its own provenance or reach the signing material. A shared runner where jobs can read each other's state or the signing credentials stops at L2; L3 needs ephemeral, isolated builders with the signer outside the job (Tekton Chains signing from its controller rather than inside the Task is the pattern).

A Kyverno ImageValidatingPolicy fails on a private-registry image with an authentication error, although the pod's imagePullSecrets work. Why?

IVP does not read the pod's imagePullSecrets and the global registry credential helper flag does not apply to it; registry access must be declared in the policy's own spec.credentials.secrets (Secrets in Kyverno's namespace) or credentials.providers for cloud ambient auth. Kyverno fetches signatures from inside its pod, so network, scheme (allowInsecureRegistry) and credentials are all evaluated there.

Docs to know your way around

study time, not exam time

docs.sigstore.dev: cosign sign/verify with keys; skim keyless (Fulcio, Rekor) so the words are familiar.
kyverno.io: image verification policies (current kind and fields).
slsa.dev: the levels table, five minutes.
Offline: cosign --help, trivy image --help, kubectl explain imagevalidatingpolicies.spec; the last one is the version arbiter when docs and cluster disagree.
docs.sigstore.dev/cosign/verifying/verify (the identity flags, --check-claims, private CA options) and /cosign/verifying/attestation (cue and rego policies on predicates).
tekton.dev/docs/chains/config (every chains-config key with its allowed values) and /docs/chains/slsa-provenance (type hints, deep inspection); slsa.dev/spec/v1.0/levels for the Build track.
kyverno.io/docs/policy-types/image-validating-policy (attestors, attestations, credentials, the CEL functions) and docs.sigstore.dev/policy-controller/overview (ClusterImagePolicy, namespace label, AND/OR semantics).