Broken delivery is the most likely shape of a domain 2 exam task. Failures sort into four buckets, and the first diagnostic move is deciding which bucket you are in; everything here is reps for that decision.
make coreOrientation
Under time pressure, the expensive mistake is not a wrong fix; it is fixing in the wrong layer. Bucket first, then descend. Say the bucket out loud before you touch anything; it costs three seconds and it stops you editing live state when the problem is a commit.
argocd app get <app> # sync status + health + last sync time + conditions
flux get kustomizations -A # Ready / Suspended / revision
kubectl -n <ns> get pods # is anything even running
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -20Those four commands place you in one of the four buckets below almost every time.
The four buckets
1 · Git is wrong, the cluster is faithful
Synced + Degraded. Bad image tag, impossible resource request, missing ConfigMap key, a probe that can never pass. The controller did its job; fix the commit. Evidence: app health Degraded (or Progressing on its way there) while sync status is green, and pod events carry the real error: ImagePullBackOff, CreateContainerConfigError, CrashLoopBackOff.
2 · The cluster refuses what git says
Sync fails outright. An admission policy denies the manifest, the API version does not exist on this cluster, a field is immutable (Service clusterIP, label selectors, PVC shrink), the CRD is not installed yet. Evidence: the sync operation's error message, which quotes the API server's rejection verbatim. Immutable-field errors mean delete-and-recreate or Replace=true, and knowing that saves ten minutes.
3 · The controller lacks permission or access
Repo unreachable, credentials rotated, an Application targeting a namespace the controller's RBAC cannot touch, an ApplicationSet token missing a scope. Evidence: errors mention the controller's own identity, appear in controller logs (kubectl -n argocd logs deploy/argocd-repo-server, flux logs), and say nothing about your workload. No pod is ever involved; that absence is the tell.
4 · Nobody is wrong, the state is stale
Webhook lost, refresh interval not elapsed, reconciliation suspended and forgotten, a branch that moved while the app points at a tag. Evidence: everything green but old. flux get kustomizations shows a Suspended row; argocd app get shows a last-sync timestamp from an hour ago. Fix with argocd app get --refresh / flux reconcile … --with-source, and check for suspension first.
| Bucket | Sync | Health | Where the text comes from | Fix lives in |
|---|---|---|---|---|
| 1 git wrong | Synced | Degraded | pod events | a commit |
| 2 cluster refuses | Failed / OutOfSync | n/a | API server, quoted by the sync op | a commit + a sync option |
| 3 access | Unknown / error | n/a | controller logs | a Secret, RBAC, or a token |
| 4 stale | Synced | Healthy | timestamps, Suspended flag | a reconcile or a resume |
With self-heal on, drift self-corrects and the interesting question becomes "what keeps re-creating this thing I keep deleting". The answer is the controller, and kubectl get <res> -o jsonpath='{.metadata.ownerReferences}' or the app.kubernetes.io/instance label proves it. With self-heal off, argocd app diff is the tool that shows exactly what diverged.
The descent, one level at a time
Application / Kustomization status.conditions, last sync, revision
│
▼
rendered manifests argocd app manifests · kustomize build · flux build
│
▼
the apply sync operation message, API server rejection text
│
▼
controller objects Deployment → ReplicaSet → Pod ← quota and PSS errors live on the RS
│
▼
pod describe → events → logs → logs --previous → exec/debug
│
▼
prove recovery with the signal that showed the failure
Two habits that pay for themselves. First, compare rendered manifests with live objects rather than reading source YAML: argocd app manifests <app> shows exactly what would be applied, which catches overlay mistakes that source review misses. Second, check the middle layer: a Deployment that is created but produces no pods has its error on the ReplicaSet, and that indirection catches out almost everyone at least once.
Exercises
Do the diagnosis before the fix, and say the bucket out loud before touching anything.
Push newTag: 9.9.9-nope to the staging overlay in the platform repo. Watch demo-staging stay Synced while health goes Progressing (Degraded arrives only after the Deployment's ten-minute progress deadline; do not wait for it, the pod evidence is immediate). Diagnose down the stack: argocd app get demo-staging → kubectl -n team-a get pods → describe pod shows ImagePullBackOff. Fix in git only.
Add a patch to the staging overlay's kustomization setting spec.clusterIP: 10.96.99.99 on the demo Service (the Service manifest itself lives in base; overlays change it through patches, which is the kustomize idiom worth a rep on its own). Sync and read the error.
Deleting the repo secret outright would not do it here, and knowing why is the exercise's first half: the lab's repos are public, so anonymous cloning still works. Wrong credentials fail where absent ones would not, because Gitea rejects a bad password even on a public repo. So corrupt the secret:
kubectl -n argocd patch secret gitea-repo -p '{"stringData":{"password":"wrong"}}'
argocd app get demo-staging --refreshThe error mentions authentication, not manifests. Restore by re-running make gitops (idempotent) or patching the real token back from .gitea-token.
flux suspend kustomization demo-flux (from 2.3), push any change to demo-app/base in the platform repo, and observe that nothing happens and nothing errors. The absence of failure is the symptom.
flux get kustomizations shows Suspended, resume, and the change lands. Train yourself to check suspension first; it costs three seconds.FAULT=config make break corrupts something in team-a's delivery path under a 7-minute clock. Bucket it, fix it, then make break-answer to compare.
Self-check
"The app is Synced and Healthy but the feature we merged an hour ago is missing."
Bucket 4, stale. Check for suspension, then the last sync timestamp and the resolved revision against your commit SHA, then whether the app tracks the branch you pushed to. Refresh/reconcile before anything else.
"Sync fails: Service "demo" is invalid: spec.clusterIP: Invalid value: field is immutable."
Bucket 2: the cluster refused the manifest. The text came from the API server, relayed by the sync operation. Fix the manifest, or force replace semantics if the field genuinely must change (accepting the churn that implies).
"argocd-repo-server logs show authentication required; the workload is untouched and running fine."
Bucket 3: access. No pod of yours is implicated; the controller's identity is. Repo credentials, token scope, or a rotated secret. The workload keeps running because reconciliation is what broke, not the deployment.
"Deployment exists, no pods, no error on the Deployment."
Look one level down at the ReplicaSet: quota, PSS, or an admission webhook rejected the pod template. Same indirection as sections 1.4 and 5.3. Not strictly a delivery bug: the delivery worked, admission refused.
"I keep deleting this ConfigMap and it keeps coming back."
Something reconciles it: self-heal, a Flux Kustomization at interval, or an operator that owns it. Prove which with ownerReferences, the app.kubernetes.io/instance label, or flux trace. Then change the source of truth instead of the live object, or suspend first if you need a temporary window.
Docs to know your way around
- argo-cd.readthedocs.io: sync options (Replace, ServerSideApply), resource health checks, the troubleshooting section.
- fluxcd.io: the troubleshooting cheatsheet (flux logs, flux events, tracing a resource to its Kustomization).
- Offline:
argocd app manifests,argocd app diff,flux build kustomization <name> --path ./…, andkubectl get events -A --sort-by=.lastTimestamp.