5.4 Audit trails, SBOMs, and compliance reports

Listed explicitly in the official PDF and the least-practised competency in the domain, because almost no home lab turns on API audit logging. This one does, at cluster build, precisely for that reason.

needsmake up cicdmake sec

Orientation

competency 5.3 · audit trails and policy compliance

Three artefacts, three questions. The audit log answers "who did what to what, and did it work". SBOMs answer "what is inside the thing we shipped". Compliance reports answer "which controls are we failing right now". Being able to produce each on demand is the competency.

API server audit logs

policy · levels · stages · the event schema

The pipeline: a policy file tells the API server what to record and at what level; a backend (here, a log file on the control-plane node; in production often a webhook to a SIEM) receives one JSON event per stage of each request.

Level	Records	Use for
None	nothing	noise: leases, events, health endpoints
Metadata	who, what, when, verb, response code	the sensible catch-all
Request	+ the request body	RBAC and policy changes
RequestResponse	+ the response body	the highest-value resources only; it is enormous

Stages: RequestReceived, ResponseStarted (long-running watches), ResponseComplete, Panic. Most analysis uses ResponseComplete. Policy rules match in order, first hit wins (the same mental model as Alertmanager routes), so the file reads as: drop the noise, elevate the sensitive, catch everything else at Metadata. Read kind/audit-policy.yaml in this repo and you will see exactly that shape.

Fields that answer real questions

Field	Answers
user.username, user.groups	who (a person, or `system:serviceaccount:ns:name`)
verb, objectRef.{resource,namespace,name}	did what, to what
responseStatus.code	did it work (403 = denied, 201 = created)
sourceIPs, userAgent	from where, with which client
annotations["authorization.k8s.io/decision"]	allow or forbid, plus the reason
annotations["pod-security.kubernetes.io/audit-violations"]	PSS violations from section 5.3, recorded quietly
stageTimestamp, auditID	when, and how to correlate the stages of one request

One operational scar the repo already documents

The apiserver is started with --audit-policy-file pointing into a mounted directory. If that file is missing, the API server does not start at all. An audit misconfiguration can therefore present as a completely dead cluster; file that away, because it is the least intuitive control-plane failure in this whole curriculum.

What audit is and is not

It records API requests. It does not record what happened inside a container, what a pod did on the network, or anything that bypassed the API server. Pair it with Falco-style runtime detection, network flow logs (Hubble), and image scanning for the other layers, and say so if a scenario asks for "a complete audit trail".

Supply-chain paper trail: Trivy Operator

compliance as queryable API objects

The operator rescans continuously and materialises results as CRDs, which is the platform move: compliance as queryable API objects rather than PDF attachments. That means kubectl, RBAC, GitOps and dashboards all work on your compliance data for free.

Kind	Holds
vulnerabilityreports	CVEs per workload image, with severity counts and fix versions
sbomreports	CycloneDX component inventory per image
configauditreports	misconfigurations in the workload spec itself
exposedsecretreports	credentials found baked into images
rbacassessmentreports	risky RBAC rules
clusterinfraassessmentreports	per-node control-plane and kubelet checks
clustercompliancereports	CIS, NSA and PSS rollups built from the above

make validate demands minimum counts of these, so a fresh cicd layer gives you a populated dataset to practise queries on.

SBOM vocabulary, because the words get tested

CycloneDX and SPDX are the two standard formats; Trivy emits both, and "which format" is a compatibility question, not a quality one.
An SBOM lists components and versions. It is not a vulnerability report; you join it against a CVE database, which is why an SBOM generated at build time stays useful after new CVEs are published.
The one-sentence purpose: when the next log4shell drops, you query your SBOMs for the package instead of rescanning the world.
Neighbours: provenance/attestation (how it was built; see section 5.6), signatures (who vouches for it). Three different documents, one trust story.

Exercises

tick the dot when its check passes

It lives on the control-plane node, reachable through docker:

docker exec cnpe-control-plane tail -1 /var/log/kubernetes/audit.log | jq .

Now answer three questions an auditor would ask, each with one jq line: who read any secret today (select(.objectRef.resource=="secrets" and .verb=="get"), print user and name); every delete that succeeded (select(.verb=="delete" and .responseStatus.code<300)); all denied requests (select(.annotations["authorization.k8s.io/decision"]=="forbid")). Then close the loop end to end: do something distinctive (kubectl -n team-a delete pod bare --ignore-not-found, or a kubectl auth can-i as dev-a from 5.1) and find your own action in the log within seconds.

verify: an audit trail you have personally queried for your own fingerprints is one you can be examined on.

With 5.3 done, grep the audit log for pod-security.kubernetes.io in annotations.

verify: the lazy pod's restricted violations were recorded by the namespace's audit: restricted label, timestamped and attributed. Three sections (5.2, 5.3, 5.4) just met in one log line, which is roughly how a real compliance program works.

Not "are there reports" but questions with answers:

kubectl get vulnerabilityreports -A -o json | jq -r '
  .items[] | [.metadata.namespace, .metadata.name,
  (.report.summary.criticalCount|tostring), (.report.summary.highCount|tostring)] | @tsv' | sort -t$'\t' -k3 -rn | head

Pick the top image and drill in: which CVE, which package, is there a fix version (.report.vulnerabilities[] | select(.severity=="CRITICAL")).

verify: a ranked worst-offenders list, and one actionable package bump identified. That triage, from fleet view to one action, is the whole vulnerability-management job in miniature.

Fleet-side: kubectl get sbomreports -A | head, then extract one and count its components (kubectl get sbomreport <name> -n <ns> -o jsonpath='{.report.components.components}' | jq length). Artifact-side, the pipeline angle from 2.4: trivy image --format cyclonedx --output sbom.json ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine and confirm both SBOMs speak the same CycloneDX.

verify: you can say what an SBOM is for in one sentence (when the next log4shell drops, you query your SBOMs for the package instead of rescanning the world).

kubectl get clustercompliancereports (CIS, NSA, PSS variants), then one in detail: kubectl get clustercompliancereport cis -o jsonpath='{.status.summary}', and find one failing control's ID and description in the full output.

verify: you can trace a failed control to the check behind it and say which earlier curriculum section would fix it. Most CIS findings in this lab trace back to sections 5.1–5.3, which is a satisfying way to discover the curriculum is a compliance program wearing a study plan's clothes.

Self-check

answer before opening

Which audit level would you set for Secrets, and which for Events? Why?

Secrets: Metadata at minimum, never RequestResponse, or you write every secret value into a log file. Events: None; they are high-volume and low-value, and dropping them keeps the log readable. First-match-wins ordering makes those two rules the top of the file.

Find every request a specific ServiceAccount was denied. Shape of the query?

Filter on .user.username == "system:serviceaccount:<ns>:<name>" and either .responseStatus.code == 403 or .annotations["authorization.k8s.io/decision"] == "forbid", then print verb and objectRef. That is the audit-log form of the RBAC debugging you did in 5.1.

An SBOM and a vulnerability report: what is the difference and why keep both?

The SBOM is an inventory of components; the vulnerability report is that inventory joined against a CVE database at a point in time. Keep both because the inventory stays true while the CVE list changes daily; that is exactly why continuous rescanning (the operator) matters more than a scan at build time alone.

Why is "compliance as CRDs" a platform-engineering idea rather than a Trivy feature?

Because it makes compliance data a first-class API object: queryable with kubectl, protected by RBAC, watchable by controllers, graphable in Grafana, and reviewable in git. The alternative, reports as files in a bucket, cannot be joined to anything the platform already does.

Your cluster's API server will not start after an audit change. First hypothesis?

The audit policy file is missing or unparseable at the path given by --audit-policy-file (or the volume mount is wrong). The API server refuses to start rather than run unaudited: a safety choice that presents as a dead control plane.

Docs to know your way around

study time, not exam time

kubernetes.io: Auditing (policy levels, stages, and the event schema).
aquasecurity.github.io/trivy-operator: the CRD reference pages, one per report kind.
cyclonedx.org / spdx.dev: enough to know which is which.
Offline: jq against the log file, and kubectl explain clustercompliancereport.status.