GitOps is one idea applied ruthlessly: git holds the desired state, a controller in the cluster pulls it and reconciles continuously, and nobody deploys by pushing manifests at the API server from a laptop or a CI job. Everything else in this domain (Argo CD, Flux, promotion, drift handling) is machinery for that one idea.
make coreOrientation
Domain 2 is the biggest slice of the exam and the most mechanical. The tasks are things like "make this change reach the cluster through git", "this app is OutOfSync, fix it", "pause reconciliation while I operate". All of them assume the model in this section is already automatic for you.
- Declarative: the system's desired state is expressed declaratively.
- Versioned and immutable: desired state is stored in a way that enforces immutability and versioning, and retains a complete version history.
- Pulled automatically: software agents pull the desired state declarations from the source.
- Continuously reconciled: agents continuously observe actual state and attempt to apply the desired state.
Graders like these words. Learn the four nouns (declarative, versioned/immutable, pulled, reconciled) and you can reconstruct the sentences.
The concepts questions hang off
Reconciliation is a loop, not an event
The controller compares live state with git on an interval and on webhooks. So a change made with kubectl edit does not fail; it gets reverted on the next loop if self-heal is on, or reported as drift if not. "Why did my manual fix disappear" is the symptom you will see most, and "OutOfSync" is how the controller reports it. The loop also means recovery is free: delete the whole namespace and the next reconcile rebuilds it. That is the disaster-recovery argument for GitOps.
┌──────────── git (desired) ────────────┐
│ demo-app/overlays/staging/… │
└────────────────┬──────────────────────┘
│ poll (~3 min) or webhook
▼
controller in the cluster ── compares ──▶ diff
│ │
automated? │ yes → apply selfHeal? │ live drift → revert
│ no → report OutOfSync prune? │ removed from git → delete live
▼
┌──────────── cluster (live) ───────────┐
└───────────────────────────────────────┘
Poke the loop. Commit a change, drift the cluster by hand, delete a manifest from git, then reconcile and see what each setting does:
Push vs pull
CI pushing manifests needs cluster credentials in the CI system, and only knows the state at deploy time. A pull-based agent keeps credentials in-cluster, never exposes them outward, and never stops comparing. That second property, continuous comparison, is why "GitOps" is not just "CI that runs kubectl".
CI builds and tests artifacts and, at most, opens a pull request that changes an image tag. CD is the in-cluster agent reconciling git. If a scenario describes a Jenkins job running kubectl apply against production, the expected critique is: credentials outside the cluster, no drift detection, no continuous reconciliation, and no single source of truth. Say all four.
Repo design
This lab seeds two Gitea repos that model the standard split: platform (cluster-level config: policies, tenants, the app-of-apps) and demo-app (one workload). The split matters because the two have different reviewers, different blast radius and different cadence.
| Choice | Common form | Why |
|---|---|---|
| App source vs config | separate repos | a config change should not rebuild an image, and a code merge should not deploy by accident |
| Environments | directories, not branches | long-lived env branches drift and merges fight; directories diff cleanly and promote by copy |
| Many apps | monorepo with per-app paths, or one repo per app plus a platform repo | monorepo eases atomic cross-app changes; per-app eases ownership and access control |
| Promotion | change flows staging → prod as a PR that bumps an image tag or a patch | the diff is the change record; approvals hang off it naturally |
| Rendered manifests | CI renders kustomize/Helm output to a branch the agent watches | reviewers see final YAML, not template intent; costs you a pipeline |
In this lab, environments are directories: demo-app/overlays/staging and overlays/prod inside the platform repo, and promotion is a change flowing from one directory to the next, usually as an image tag bump or a kustomize patch. Branches-per-environment exists in the wild and the received wisdom is to avoid it, for exactly the reason above.
Templating, both flavors
Kustomize layers patches over a base with no templating language: resources, patches (strategic-merge or JSON6902), images, namePrefix, labels (the v5 replacement for the deprecated commonLabels), configMapGenerator. What you see is YAML all the way down, and kustomize build shows you exactly what will be applied. Helm renders Go templates from values, giving conditionals and loops at the cost of "the manifest does not exist until something renders it". Argo CD and Flux both consume both, and both let you post-render one with the other.
Helm is also a release manager, not only a renderer, and a task can hand you that side of it: helm list -A (releases and revisions), helm upgrade --install --atomic --wait (roll back automatically if it does not become ready), helm history / helm rollback <rel> <rev>, helm get values --revision and helm diff if the plugin is there. Release state lives in Secrets of type helm.sh/release.v1 in the release namespace. That is how you discover what a previous operator did, and why a Flux HelmRelease can adopt or fight an existing release.
Never push an overlay you have not built locally. kustomize build <dir> (or kubectl kustomize) and helm template are the two commands that turn "I think this renders right" into "I know". They also work offline in the exam terminal.
Secrets
Plain Secrets cannot live in git; base64 is not encryption. The three sanctioned answers are Sealed Secrets (encrypt for git, only the in-cluster controller can decrypt), External Secrets Operator (git holds references, an external store holds truth) and SOPS (encrypted values in git, decrypted by the agent; Flux has native support). Section 5.1 does the mechanics. Here, know that "how do secrets work in GitOps" has a real answer, and "commit it base64-encoded" is not it.
Failure modes to recognize on sight
- Two controllers, one resource. Argo CD and Flux both managing the same manifest is an infinite reconcile war. In this lab they deliberately own different paths, and that separation is the lesson.
- Manual fix disappears. Self-heal did its job. The fix belongs in git; if you truly need a live change, suspend reconciliation first (Flux) or disable auto-sync (Argo), do the surgery, then fold it back.
- Prune surprises. Rename a resource in git without prune and the old one lives forever; with prune, deleting a file deletes production. Know which behavior you have before you push.
- Immutable fields. Service
clusterIP, selector labels, PVC shrink: apply fails with an API server rejection quoting the field. The GitOps answer is delete-and-recreate or a replace-style sync option, not editing live. - Stale but green. Suspended reconciliation, a lost webhook, a long interval. Everything reports healthy and nothing is current. Always check "when did this last sync" before believing a green dashboard.
Repos, promotion and rendered manifests
The OpenGitOps principles are versioned (v1.0.0); the glossary behind them defines desired state, state store, reconciliation and continuous, and "continuous" is defined there as "continues to happen" rather than instantaneous. The rest of this panel is the layout and promotion decisions that turn the principles into a repository.
Three layouts the Flux guide names, and when each wins
| Layout | Tree | Choose it when | Cost |
|---|---|---|---|
| Monorepo | apps/{base,staging,production} · infrastructure/{base,staging,production} · clusters/{staging,production} | one platform team, few tenants, atomic cross-cutting changes wanted | everyone can read production config; big PRs hide production changes |
| Repo per environment | same tree, but production in its own repository | production access must be narrower than staging access | promotion is a cross-repo PR; infrastructure changes need two commits |
| Repo per team | platform repo with teams/, infrastructure/, clusters/; each dev team owns an apps repo the platform registers as a source | tenants own their delivery; platform owns onboarding and cluster addons | the platform must reconcile tenant repos under a restricted identity (2.3 serviceAccountName, 2.2 AppProject) |
In all three, clusters/<name> is the entry point a controller is pointed at, and ordering is expressed there: infrastructure (CRDs, controllers, ingress) before apps, via Flux dependsOn or Argo CD waves in an app-of-apps.
Promotion patterns
- Tag bump by PR: CI (or Flux image automation) writes the new image tag into the staging overlay; a human or a bot copies the same change into production. The PR is the change record; approvals hang off it.
- Pin to a SHA or tag: production Applications and Kustomizations track an immutable ref; promotion is moving the pin. Nothing reaches production because a branch moved.
- Environment gates: Argo CD sync windows, Flux
suspend, or a Kargo-style promotion controller in front of the pins. The exam does not name Kargo; it does expect you to say that promotion is a git operation, not a click in a UI. - Anti-pattern: a branch per environment. Merges accumulate conflicts, cherry-picks lie, and the environments drift structurally.
Rendered manifests
Templating hides the YAML that will actually be applied. The rendered manifests pattern renders it in a pipeline and commits the output to a branch or directory the agent watches, so review sees final YAML and the diff between staging and production is a real diff. Argo CD ships this as the Source Hydrator (beta since 3.5): spec.sourceHydrator.drySource names the templated source, syncSource.targetBranch and path name where the hydrated output is pushed and synced from, and a push credential is a repo Secret labeled argocd.argoproj.io/secret-type: repository-write. Flux gets the same effect by having CI run flux push artifact and pointing an OCIRepository at the result.
OCI artifacts as a source
- Argo CD (3.1+):
repoURL: oci://registry/org/config,targetRevisiona tag or digest,path: .; Helm charts in OCI work the same way, withchartsemantics. - Flux:
flux push artifact oci://registry/org/config:tag --path ./deploy --source ... --revision ...produces an artifact with Flux media types; anOCIRepositorywithref.tag,ref.semverorref.digestconsumes it, optionally withverify.provider: cosignornotation. A Kustomization uses it exactly like a GitRepository. - Why: registries already hold the images, so config and image can share signing, retention and promotion (retag), and clusters without git access can still pull.
Kustomize vs Helm, decided
| Question | Kustomize | Helm |
|---|---|---|
| You own the manifests and want per-env deltas | bases and overlays, patches, images, replacements, components | possible via values, heavier |
| You consume third-party software | possible by vendoring or remote bases (Flux can forbid remote bases) | the chart is the product; pin version, override values |
| Conditionals, loops, generated names | none by design | Go templates |
| Review | plain YAML, kubectl kustomize shows the truth | helm template or the hydrator |
| Lifecycle | the GitOps agent is the lifecycle | release history, hooks, tests, rollback (Flux HelmRelease exposes them; Argo CD renders and applies, no Helm release object) |
Both engines let you post-render one with the other: Flux HelmRelease.spec.postRenderers[].kustomize, Argo CD spec.source.kustomize on a Helm chart via multiple sources or a plugin. Argo CD does not create Helm release Secrets; helm list shows nothing for an Argo-managed chart, which is a common surprise.
"Promote the staging image to production" means: find where production pins the image (overlay images:, HelmRelease values, Application targetRevision), change that in git, and verify by comparing rendered output (kubectl kustomize overlays/production) before the controller does. Changing the live Deployment scores zero even if it looks identical.
Secrets and engine choice
The three secret answers, field by field
| Question | Sealed Secrets | SOPS | External Secrets Operator |
|---|---|---|---|
| What is in git | a SealedSecret CR with ciphertext, encrypted to the controller's public key | the Secret manifest with data/stringData encrypted in place, plus a sops metadata block | an ExternalSecret naming keys in a store; no secret material |
| Who decrypts | the in-cluster controller only | the GitOps agent (Flux kustomize-controller natively via spec.decryption.provider: sops and a secretRef to the age or PGP key; Argo CD needs the KSOPS plugin) or a human with the key | the operator, reading the store with a SecretStore/ClusterSecretStore credential |
| Scope and rotation | --scope strict (name and namespace baked in, default), namespace-wide, cluster-wide; sealing keys renew every 30 days, old ones kept | .sops.yaml creation_rules with encrypted_regex: ^(data|stringData)$ so metadata stays readable; age recommended over PGP; KMS backends | refreshInterval (or refreshPolicy: CreatedOnce), templating, PushSecret to write back |
| Weak point | losing the controller's private key loses everything; cluster-bound | key distribution to every decrypting agent; secret rotation is a commit | the store is the truth, so the cluster depends on it; a store outage stops rotation but existing Secrets remain |
Argo CD vs Flux, when the task lets you choose
| Aspect | Argo CD 3.x | Flux 2.x |
|---|---|---|
| Unit | Application (one source or several) and ApplicationSet | GitRepository/OCIRepository + Kustomization / HelmRelease |
| UI and RBAC | built in: SSO, project roles, policy.csv | none; Kubernetes RBAC via impersonation (serviceAccountName) |
| Helm | renders the chart and applies plain manifests; no release history | real Helm releases with tests, remediation and rollback |
| Drift | selfHeal is opt-in; OutOfSync is reported even when manual | every interval re-applies; drift correction is implicit |
| Image write-back | none built in (Argo CD Image Updater is a separate project) | image-reflector and image-automation controllers |
| Fan-out | ApplicationSet generators | one Kustomization per cluster directory; Flux Operator ResourceSets in the wider ecosystem |
| Footprint | api-server, repo-server, application-controller, redis, dex, applicationset, notifications | four to six small controllers |
| Multi-tenant lockdown | AppProject source/destination allow-lists | --no-cross-namespace-refs, --no-remote-bases, --default-service-account |
Where a task does not name the engine, the grader reads the resulting cluster state and the git commit, and either engine produces both. Pick the one whose CLI you can drive without the docs.
Bootstrapping, both ways
- Flux:
flux bootstrap github --owner ORG --repository fleet --branch main --path clusters/prod --personal --token-auth(alsogitlab,gitea,bitbucket-server,git). It installs the controllers, commitsgotk-components.yamlandgotk-sync.yamlunder the path, and creates theflux-systemGitRepository plus Kustomization that from then on manages Flux itself.--components-extra image-reflector-controller,image-automation-controlleradds image automation;flux checkverifies;flux migratemoves objects to current API versions before an upgrade (2.9 removed thev1beta2image and notification APIs). - Argo CD: apply the install manifest, then create one Application pointing at the platform repo's
argocd/orclusters/directory that contains the Argo CD config and every other Application (app-of-apps). Argo CD then manages its ownargocd-cm,argocd-rbac-cmand Applications; the initial admin password inargocd-initial-admin-secretis meant to be rotated and deleted.
After bootstrapping both engines on one cluster, make the ownership split visible in the repo: Flux reconciles clusters/ and infrastructure/, Argo CD owns apps/, or the other way round, never both on the same directory. Argo CD's FailOnSharedResource=true and Flux's kustomize.toolkit.fluxcd.io/ssa: Ignore annotation are the escape hatches when the split is imperfect.
Exercises
Never push an overlay you haven't built locally:
kustomize build examples/demo-app/overlays/staging
kustomize build examples/demo-app/overlays/prod | grep -E 'replicas|name:'outputcaptured 2026-08-26
$ kustomize build examples/demo-app/overlays/staging
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/part-of: demo-app
name: staging-demo
namespace: team-a
spec:
ports:
- name: http
port: 80
targetPort: http
selector:
app: demo
app.kubernetes.io/part-of: demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: demo
app.kubernetes.io/part-of: demo-app
name: staging-demo
namespace: team-a
spec:
replicas: 1
selector:
matchLabels:
app: demo
app.kubernetes.io/part-of: demo-app
template:
metadata:
labels:
app: demo
app.kubernetes.io/part-of: demo-app
spec:
containers:
- image: ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine
name: web
ports:
- containerPort: 8080
name: http
readinessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 2
resources:
limits:
cpu: 200m
memory: 128Mi
requests:
cpu: 25m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- mountPath: /tmp
name: cache
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
topologySpreadConstraints:
- labelSelector:
matchLabels:
app: demo
app.kubernetes.io/part-of: demo-app
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
volumes:
- emptyDir: {}
name: cache
$ kustomize build examples/demo-app/overlays/prod | grep -E 'replicas|name:'
name: prod-demo
- name: http
name: prod-demo
replicas: 3
name: web
name: http
name: cache
name: cacheThe platform repo in Gitea carries a copy of examples/. Clone it, change the staging replica count, push:
source lab.env # for GITEA_PASS; anonymous clone works, push needs credentials
git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform && cd /tmp/platform
# edit demo-app/overlays/staging/kustomization.yaml: set replicas, or add a patch
git commit -am "staging: 3 replicas" && git pushoutputcaptured 2026-08-26
$ source lab.env # for GITEA_PASS; anonymous clone works, push needs credentials
$ git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform && cd /tmp/platform
Cloning into '/tmp/platform'...
# edit demo-app/overlays/staging/kustomization.yaml: set replicas, or add a patch
$ git commit -am "staging: 3 replicas" && git push
[main c7cb9a1] staging: 3 replicas
1 file changed, 1 insertion(+), 1 deletion(-)
To http://gitea.lab:3000/lab/platform.git
62d527f..c7cb9a1 main -> mainBoth controllers are already connected to Gitea:
kubectl -n argocd get secret gitea-repo -o jsonpath='{.data.url}' | base64 -d; echo
flux get sources gitoutputcaptured 2026-08-26
$ kubectl -n argocd get secret gitea-repo -o jsonpath='{.data.url}' | base64 -d; echo
http://gitea.lab:3000/lab/platform.git
$ flux get sources git
NAME REVISION SUSPENDED READY MESSAGE
platform main@sha1:c7cb9a1e False True stored artifact for revision 'main@sha1:c7cb9a1e' gitea.lab and the Flux GitRepository shows Ready True with a commit SHA. Now you know the truth both engines reconcile from, and it is the same repo you just pushed to.With nothing managing demo-app yet, apply it manually (kubectl create ns team-a if needed, then kubectl apply -k examples/demo-app/overlays/staging; the overlay pins its own namespace, so no -n flag). Scale it by hand. Nobody notices, because nothing is reconciling yet.
Git is the usual source, not the only one. Flux treats an OCI artifact as a first-class source, which is how teams ship a pinned, signed bundle instead of a moving branch. The lab registry is enough to do the whole loop.
flux push artifact oci://localhost:5001/platform-config:$(git rev-parse --short HEAD) \
--path ./examples/demo-app/base \
--source $(git remote get-url origin) \
--revision main@sha1:$(git rev-parse HEAD)
flux create source oci demo-oci \
--url oci://kind-registry:5000/platform-config \
--tag $(git rev-parse --short HEAD) \
--insecure
flux get sources oci demo-oci
kubectl -n flux-system get ocirepository demo-oci -o jsonpath='{.status.artifact.revision}{"\n"}'outputcaptured 2026-09-12
$ flux push artifact oci://localhost:5001/platform-config:$(git rev-parse --short HEAD) \
--path ./examples/demo-app/base \
--source $(git remote get-url origin) \
--revision main@sha1:$(git rev-parse HEAD)
► pushing artifact to localhost:5001/platform-config:13403b5
✔ artifact successfully pushed to localhost:5001/platform-config@sha256:9e73f795cf5ea00d6f8120f2a75ecc210d6aea9d70d9148d190428fea3eef73d
$ flux create source oci demo-oci \
--url oci://kind-registry:5000/platform-config \
--tag $(git rev-parse --short HEAD) \
--insecure
► applying OCIRepository
✔ OCIRepository created
◎ waiting for OCIRepository reconciliation
✔ OCIRepository reconciliation completed
✔ fetched revision: 13403b5@sha256:9e73f795cf5ea00d6f8120f2a75ecc210d6aea9d70d9148d190428fea3eef73d
$ flux get sources oci demo-oci
NAME REVISION SUSPENDED READY MESSAGE
demo-oci 13403b5@sha256:9e73f795 False True stored artifact for digest '13403b5@sha256:9e73f795'
$ kubectl -n flux-system get ocirepository demo-oci -o jsonpath='{.status.artifact.revision}{"\n"}'
13403b5@sha256:9e73f795cf5ea00d6f8120f2a75ecc210d6aea9d70d9148d190428fea3eef73dBefore you debug either engine, know what it has been told to look at. The two answers live in completely different places: a labeled Secret for Argo CD, a set of source objects for Flux.
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository -o name
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository -o jsonpath='{range .items[*]}{.data.url}{"\n"}{end}' | base64 -d
flux get sources all -Aoutputcaptured 2026-09-12
$ kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository -o name
secret/gitea-repo
$ kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository -o jsonpath='{range .items[*]}{.data.url}{"\n"}{end}' | base64 -d
http://gitea.lab:3000/lab/platform.git
$ flux get sources all -A
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system gitrepository/platform main@sha1:0cc7cba2 False True stored artifact for revision 'main@sha1:0cc7cba2'
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system helmrepository/podinfo sha256:e7dc68a4 False True stored artifact: revision 'sha256:e7dc68a4'
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system helmchart/flux-demo-podinfo 6.15.0 False True pulled 'podinfo' chart with version '6.15.0' Argo CD renders a chart and applies the manifests; there is no Helm release object afterwards. Flux's HelmRelease runs Helm properly and leaves one. The difference decides whether helm rollback is even available to you.
helm list -A
kubectl get secret -A -l owner=helm -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name | head -20
kubectl -n flux-demo get helmrelease 2>/dev/null || echo 'no HelmRelease yet: do the 2.3 HelmRelease exercise first'outputcaptured 2026-09-12
$ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
alloy monitoring 1 2026-09-12 15:21:20.891191379 -0400 EDT deployed alloy-1.12.1 v1.19.2
argo-rollouts argo-rollouts 1 2026-09-12 15:08:43.478084962 -0400 EDT deployed argo-rollouts-2.43.1 v1.10.0
argo-workflows argo 1 2026-09-12 15:09:11.745300271 -0400 EDT deployed argo-workflows-2.0.6 v4.1.3
argocd argocd 1 2026-09-12 15:07:43.560431837 -0400 EDT deployed argo-cd-10.9.0 v3.5.2
cilium kube-system 1 2026-09-12 15:04:37.709308421 -0400 EDT deployed cilium-1.20.1 1.20.1
cnpg cnpg-system 1 2026-09-12 15:13:32.372693432 -0400 EDT deployed cloudnative-pg-0.29.0 1.30.0
crossplane crossplane-system 1 2026-09-12 15:12:07.753874688 -0400 EDT deployed crossplane-2.4.0 2.4.0
external-secrets external-secrets 1 2026-09-12 15:25:24.406531885 -0400 EDT deployed external-secrets-2.10.0 v2.10.0
gatekeeper gatekeeper-system 1 2026-09-12 15:23:58.691236047 -0400 EDT deployed gatekeeper-3.23.1 v3.23.1
kro kro 1 2026-09-12 15:13:59.530435784 -0400 EDT deployed kro-0.4.1 0.4.1
kyverno kyverno 1 2026-09-12 15:23:18.115566636 -0400 EDT deployed kyverno-3.9.1 v1.19.1
loki monitoring 1 2026-09-12 15:20:00.047882452 -0400 EDT deployed loki-7.3.0 3.6.12
metrics-server kube-system 1 2026-09-12 15:06:18.107019833 -0400 EDT deployed metrics-server-3.14.0 0.9.0
opencost opencost 1 2026-09-12 15:22:04.687666397 -0400 EDT deployed opencost-2.5.31 1.121.2
opentelemetry-operator opentelemetry-operator-system 1 2026-09-12 15:19:29.193460188 -0400 EDT deployed opentelemetry-operator-0.122.0 0.158.0
podinfo flux-demo 1 2026-09-13 01:54:47.777432549 +0000 UTC deployed podinfo-6.15.0 6.15.0
prometheus monitoring 1 2026-09-12 15:14:56.63674145 -0400 EDT deployed kube-prometheus-stack-90.1.1 v0.93.1
sealed-secrets kube-system 1 2026-09-12 15:25:01.126919917 -0400 EDT deployed sealed-secrets-2.20.0 0.40.0
spire spire 2 2026-09-12 21:56:46.158610171 -0400 EDT deployed spire-0.30.2 1.15.3
spire-crds spire 2 2026-09-12 21:56:42.750335391 -0400 EDT deployed spire-crds-0.6.1 0.0.1
trivy-operator trivy-system 1 2026-09-12 15:11:44.973505519 -0400 EDT deployed trivy-operator-0.36.0 0.34.0
vpa vpa 1 2026-09-12 15:06:51.898902756 -0400 EDT deployed vpa-5.0.1 1.7.1
$ kubectl get secret -A -l owner=helm -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name | head -20
NS NAME
argo-rollouts sh.helm.release.v1.argo-rollouts.v1
argo sh.helm.release.v1.argo-workflows.v1
argocd sh.helm.release.v1.argocd.v1
cnpg-system sh.helm.release.v1.cnpg.v1
crossplane-system sh.helm.release.v1.crossplane.v1
external-secrets sh.helm.release.v1.external-secrets.v1
flux-demo sh.helm.release.v1.podinfo.v1
gatekeeper-system sh.helm.release.v1.gatekeeper.v1
kro sh.helm.release.v1.kro.v1
kube-system sh.helm.release.v1.cilium.v1
kube-system sh.helm.release.v1.metrics-server.v1
kube-system sh.helm.release.v1.sealed-secrets.v1
kyverno sh.helm.release.v1.kyverno.v1
monitoring sh.helm.release.v1.alloy.v1
monitoring sh.helm.release.v1.loki.v1
monitoring sh.helm.release.v1.prometheus.v1
opencost sh.helm.release.v1.opencost.v1
opentelemetry-operator-system sh.helm.release.v1.opentelemetry-operator.v1
spire sh.helm.release.v1.spire-crds.v1
$ kubectl -n flux-demo get helmrelease 2>/dev/null || echo 'no HelmRelease yet: do the 2.3 HelmRelease exercise first'
NAME AGE READY STATUS
podinfo 5m56s True Helm install succeeded for release flux-demo/podinfo.v1 with chart podinfo@6.15.0helm list while anything Flux installed through a HelmRelease is present. State what you would use instead of helm rollback on the Argo CD side.Argo CD has to recognize its own objects on a cluster full of other people's. Since 3.x it does that with an annotation rather than a label, which matters the moment you copy a manifest between applications and it starts fighting for ownership.
kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.application\.resourceTrackingMethod}' | grep . || echo 'unset, so the 3.x default is in force'
kubectl -n team-a get deploy staging-demo -o jsonpath='{.metadata.annotations}' | jq 'with_entries(select(.key | startswith("argocd")))'
kubectl -n team-a get deploy staging-demo -o jsonpath='{.metadata.labels}' | jqoutputcaptured 2026-09-13
$ kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.application\.resourceTrackingMethod}' | grep . || echo 'unset, so the 3.x default is in force'
unset, so the 3.x default is in force
$ kubectl -n team-a get deploy staging-demo -o jsonpath='{.metadata.annotations}' | jq 'with_entries(select(.key | startswith("argocd")))'
{
"argocd.argoproj.io/tracking-id": "demo-staging:apps/Deployment:team-a/staging-demo"
}
$ kubectl -n team-a get deploy staging-demo -o jsonpath='{.metadata.labels}' | jq
{
"app": "demo",
"app.kubernetes.io/part-of": "demo-app",
"cost-centre": "team-a"
}argocd-cm has no application.resourceTrackingMethod key at all, which is the 3.x default of annotation, and the annotation on the deployment names the application, the group/kind and the destination namespace in one string.Self-check
State the four OpenGitOps principles without looking.
Declarative; versioned and immutable; pulled automatically by agents; continuously reconciled. If you can only remember three, the one people drop is "versioned and immutable", which is also the one that carries the audit and rollback story.
Why are environment branches discouraged?
Long-lived branches accumulate divergent changes, so promotion becomes a merge with conflicts and cherry-picks rather than a reviewable diff, and the environments drift structurally rather than only in the values you meant to differ. Directories keep the difference explicit and diffable.
A colleague fixes production with kubectl edit during an incident. What happens next, and what should they have done?
Self-heal reverts it at the next reconcile (or it is reported as drift). The sanctioned move is to suspend/disable reconciliation for that app, apply the emergency change, then land the same change in git and resume, so the fix survives and the history records it.
What does prune actually change, and what is the failure mode in each setting?
Prune deletes live resources whose manifests disappeared from git. Off: renamed or removed resources linger forever as invisible cruft. On: deleting a file deletes the running thing, so a careless refactor is a production outage. Neither is safe by default; the safety comes from knowing which one you have.
Where do secrets live in a GitOps repo?
Encrypted (Sealed Secrets or SOPS) or by reference (External Secrets pointing at a real store). Never plain, never base64-only. The distinction to state: Sealed Secrets/SOPS keep ciphertext in git; ESO keeps only a pointer and leaves truth in the external store.
What is the rendered manifests pattern and how does Argo CD 3.5 implement it?
Render Helm or Kustomize in a pipeline and commit the resulting plain YAML to a branch or directory the agent syncs from, so reviewers diff final manifests. Argo CD's Source Hydrator (beta since 3.5) does it in the controller: sourceHydrator.drySource is the templated input, syncSource.targetBranch/path is where hydrated output is pushed and synced from, using a repository-write Secret.
Which secret approach keeps no secret material in git at all, and what is its dependency?
External Secrets Operator: git holds an ExternalSecret that names keys in a SecretStore or ClusterSecretStore; the operator fetches and writes the Kubernetes Secret on a refreshInterval. The dependency is the external store and its credential: if it is down, rotation stops, though already-created Secrets keep working. Sealed Secrets and SOPS keep ciphertext in git instead.
What does flux bootstrap actually create, and what is the name of the object that then manages Flux itself?
It installs the controllers, commits gotk-components.yaml and gotk-sync.yaml under --path in the repository, and creates a GitRepository and a Kustomization both named flux-system in the flux-system namespace. From then on that Kustomization reconciles the cluster directory, Flux included, so upgrades are a commit.
Why does helm list show nothing for a chart Argo CD deployed?
Argo CD renders the chart with helm template semantics and applies the manifests itself; it never writes Helm release Secrets. Flux's helm-controller, by contrast, performs real Helm install/upgrade actions and keeps release history, so helm list -n <storageNamespace> shows it. State that difference when comparing engines.
Docs to know your way around
- opengitops.dev: the four principles, in exactly the form graders like.
- kustomize.io: bases and overlays, patches, and the transformer list.
- argo-cd.readthedocs.io and fluxcd.io: each has a "core concepts" page worth ten minutes before the tool sections.
- Offline:
kubectl kustomize <dir>,helm template,git log --oneline: your desired state is a repo, so ordinary git tooling is half the diagnosis. - fluxcd.io/flux/guides/repository-structure: monorepo, repo per environment and repo per team with their trade-offs.
- argo-cd.readthedocs.io: Source Hydrator, OCI, Multiple Sources: the rendered manifests feature, oci:// sources and the $values pattern.
- fluxcd.io/flux/guides/mozilla-sops and github.com/bitnami-labs/sealed-secrets: decryption config for Flux; scopes and key renewal for Sealed Secrets.