2.2 Argo CD: applications, sync, drift

Argo CD is the tool most likely to be sitting in front of you on the exam, and the Application resource is its entire API. One spec: source, destination, syncPolicy. Everything in the UI is a view over that, and everything that goes wrong is visible in status.

needsmake core

Orientation

competency 2.1 · GitOps workflows

Two hours of Argo CD practice pays back more exam points than any other tool in this curriculum. Know the Application spec cold, know the two status axes, know the CLI, and know where the error text comes from when a sync fails.

The components

Component	Job	Errors it produces
repo-server	clones git, renders kustomize/Helm	auth failures, template/render errors, missing paths
application-controller	compares, syncs, reports health	API server rejections, prune/hook behavior, OutOfSync
api-server	UI, CLI, RBAC, SSO	login and permission errors
redis	cache of rendered manifests and live state	weird staleness after a redis restart

When a task says "the app will not sync", the first fork is: did rendering fail (repo-server) or did applying fail (controller)? The message tells you, and knowing which pod's logs to read halves the time.

The Application spec

source · destination · syncPolicy

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-staging
  namespace: argocd                      # apps live in the controller's namespace
spec:
  project: default                       # AppProject = the guardrails (see below)
  source:
    repoURL: http://gitea.lab:3000/lab/platform.git
    targetRevision: HEAD                 # branch, tag, or commit SHA
    path: demo-app/overlays/staging      # or chart: + helm: for a chart source
  destination:
    server: https://kubernetes.default.svc   # or name: in-cluster
    namespace: team-a
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff: { duration: 5s, factor: 2, maxDuration: 3m }

For targetRevision, HEAD or a branch means "whatever moves there"; a tag or SHA means immutable. Production apps pinned to a SHA and promoted by bumping it is a legitimate, auditable pattern.

The two status axes

Axis	Values	Answers
Sync	Synced · OutOfSync · Unknown	does live match git?
Health	Healthy · Progressing · Degraded · Missing · Suspended · Unknown	are the resources themselves okay?

Synced and Degraded is not a contradiction

Git said run a broken image, and the cluster faithfully runs it broken. Sync is green because the diff is empty; health is red because the Deployment cannot progress. Clicking sync again does nothing at all. The fix is a commit. Recognize this state on sight; section 2.6 opens with it.

Every combination means something different, and each has exactly one right first move:

Health is computed per resource kind by built-in checks (a Deployment is Healthy when its available replicas match, a Service with a LoadBalancer waits for an IP…). Custom kinds get custom Lua health checks, which is how Argo CD reports on CRDs like Rollouts. If a custom resource sits forever in Progressing, either it has no health check or its status conditions never settle.

syncPolicy, decided before you need it

Manual: reports OutOfSync and waits. Correct for production changes that need a human.
automated: syncs when git changes.
automated.selfHeal: true: also reverts live drift, continuously.
automated.prune: true: deletes live resources whose manifests vanished from git. The one that bites: without it, renaming a resource leaves the old one running forever; with it, removing a file deletes production.

Sync options worth memorizing

Option	What it fixes
CreateNamespace=true	destination namespace does not exist yet
Replace=true	immutable-field errors; does `kubectl replace` instead of apply
ServerSideApply=true	huge CRDs, field-manager conflicts, last-applied annotation limits
SkipDryRunOnMissingResource=true	CRs whose CRD arrives from elsewhere during the same sync
PrunePropagationPolicy=foreground	ordered deletion of owned resources
ApplyOutOfSyncOnly=true	huge apps where re-applying everything is slow

They can be set per-Application (spec.syncPolicy.syncOptions) or per-resource with the argocd.argoproj.io/sync-options annotation. Know both placements.

The rest of the sync options

Option	What it does	Placement
Prune=false · Delete=false	never prune this resource / keep it when the Application is deleted (PVCs, namespaces)	resource annotation, or app-wide
Prune=confirm · Delete=confirm	the sync stays in Syncing until a human confirms; confirmation is the `argocd.argoproj.io/deletion-approved` annotation on the Application (UI button, CLI `argocd app confirm-deletion`)	either
PruneLast=true	prune as a final implicit wave after everything else is healthy	either
RespectIgnoreDifferences=true	also honor `ignoreDifferences` when applying, so Argo stops overwriting the HPA's replica count	app
FailOnSharedResource=true	fail the sync if another Application already manages a resource	app
Validate=false	skip schema validation (needed with ServerSideApply partial manifests, RawExtension kinds)	either
Force=true	delete-and-recreate; only meaningful with `Replace=true` or client-side apply	resource annotation
ServerSideApply=true	`kubectl apply --server-side --force-conflicts`; also enables patching partial manifests and the automatic client-side-apply migration of managedFields	either; `ServerSideApply=false` opts one resource out

Replace=true takes precedence over ServerSideApply=true. Namespace metadata is the special case: syncPolicy.managedNamespaceMetadata sets labels and annotations on a namespace created by CreateNamespace=true; a Namespace manifest in the repo with the same name wins over it.

Ordering: waves and hooks

CRDs before CRs, migrations before rollout

A sync happens in phases, and within the Sync phase in waves. The annotation argocd.argoproj.io/sync-wave: "-1" runs earlier (lower first, default 0); Argo waits for each wave's resources to be healthy before starting the next.

PreSync  ──▶  Sync (wave -1 → 0 → 1 → …)  ──▶  PostSync  ──▶  SyncFail (only on failure)
   │                      │                       │
   │                      │                       └─ smoke-test Job
   │                      └─ CRDs, then CRs, then apps
   └─ db migration Job

Hooks are ordinary manifests annotated with argocd.argoproj.io/hook: PreSync|Sync|PostSync|SyncFail|PostDelete|Skip, plus hook-delete-policy (HookSucceeded, BeforeHookCreation, HookFailed) so old Jobs do not pile up. The stock examples: a database migration Job in PreSync, a smoke test in PostSync.

The "no matches for kind" fix has two forms

If you own both the CRD and the CR, put the CRD in wave -1. If the CRD arrives from somewhere you do not control, annotate the CR with SkipDryRunOnMissingResource=true so the pre-apply dry run stops failing.

Scale patterns and guardrails

AppProject · app-of-apps · ApplicationSet

AppProject

An AppProject is a policy boundary around a set of Applications: which sourceRepos they may deploy from, which destinations (cluster + namespace pairs) they may target, which cluster-scoped and namespaced resource kinds are allowed or denied, plus per-project RBAC roles and sync windows (time ranges when syncing is permitted or blocked). In a multi-tenant platform this is how you let teams own Applications without letting them deploy a ClusterRoleBinding into kube-system. If a task says "team X must only deploy to namespace Y from repo Z", the answer is an AppProject, not a Role.

app-of-apps vs ApplicationSet

app-of-apps: one Application whose manifests are other Applications. Simple, explicit, reviewable; you write each child by hand. Great for bootstrapping a cluster's platform layer in a defined order (with waves).
ApplicationSet: a template plus generators that stamp out Applications. Generators to recognize: list, clusters, git (directories or files in a repo), scmProvider (every repo in an org matching criteria), pullRequest (ephemeral preview environments), and matrix/merge to combine them. goTemplate: true switches the template engine to Go templating with sprig functions.

For the exam, be able to read a generator block and predict exactly which Applications will exist. That is the testable skill. In this lab, examples/argocd-appset.yaml generates one app per overlay directory found in git, and the Backstage golden path (section 3.6) generates apps from an SCM generator over the Gitea services org.

Where does the workload actually land?

An ApplicationSet template sets destination.namespace per generated app, but an explicit namespace: inside a manifest (or a kustomization) always wins. So the lab's overlays land in team-a/team-b regardless of what the template's destination says.

Other clusters

destination.server: https://kubernetes.default.svc (or name: in-cluster) is the local cluster. Any other cluster has to be registered first: argocd cluster add <context> creates a ServiceAccount there and stores its credentials in a Secret in argocd labeled argocd.argoproj.io/secret-type: cluster. That Secret is the thing an ApplicationSet's clusters generator iterates, which is why generator-driven fan-out and "deploy this to the second cluster" are the same mechanism. argocd cluster list tells you what is registered; the AppProject's destinations list decides who may target it.

Diffing quirks

ignoreDifferences (by group/kind/jsonPointers or a jq path) is how you stop a mutating webhook or an HPA-managed replica count from showing permanent drift. argocd.argoproj.io/compare-options: IgnoreExtraneous hides resources Argo did not create. Both exist because "OutOfSync forever on a field nobody edits" is a real and common state.

CLI, because the UI wastes exam time

argocd login <server> --username admin \
  --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d) --insecure
argocd app list
argocd app get demo-staging
argocd app diff demo-staging          # what would change
argocd app sync demo-staging          # --prune, --force, --resource
argocd app history demo-staging       # and: argocd app rollback demo-staging <id>
argocd app set demo-staging --sync-policy automated --self-heal

outputcaptured 2026-08-26

$ argocd login 172.18.0.9 --username admin \
  --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d) --insecure
WARNING: server is not configured with TLS. Proceed (y/n)? 'admin:login' logged in successfully
Context '172.18.0.9' updated
$ argocd app list
NAME                 CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH       SYNCPOLICY  CONDITIONS  REPO                                    PATH                       TARGET
argocd/demo-prod     https://kubernetes.default.svc  prod       default  Synced  Progressing  Auto-Prune  <none>      http://gitea.lab:3000/lab/platform.git  demo-app/overlays/prod     main
argocd/demo-staging  https://kubernetes.default.svc  staging    default  Synced  Progressing  Auto-Prune  <none>      http://gitea.lab:3000/lab/platform.git  demo-app/overlays/staging  main
$ argocd app get demo-staging
Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (c7cb9a1)
Health Status:      Progressing

GROUP  KIND        NAMESPACE  NAME          STATUS   HEALTH       HOOK  MESSAGE
       Namespace              staging       Running  Synced             namespace/staging created
       Service     team-a     staging-demo  Synced   Healthy            service/staging-demo created
apps   Deployment  team-a     staging-demo  Synced   Progressing        deployment.apps/staging-demo created
$ argocd app diff demo-staging          # what would change
$ argocd app sync demo-staging          # --prune, --force, --resource
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-08-26T22:17:01-04:00            Service      team-a          staging-demo    Synced  Healthy              
2026-08-26T22:17:01-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              
2026-08-26T22:17:02-04:00            Service      team-a          staging-demo    Synced  Healthy              service/staging-demo unchanged
2026-08-26T22:17:02-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              deployment.apps/staging-demo unchanged

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (c7cb9a1)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      c7cb9a1e37949219ff64e2c5dd17654109deea6d
Phase:              Succeeded
Start:              2026-08-26 22:17:01 -0400 EDT
Finished:           2026-08-26 22:17:02 -0400 EDT
Duration:           1s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo unchanged
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged
$ argocd app history demo-staging       # and: argocd app rollback demo-staging <id>
SOURCE  http://gitea.lab:3000/lab/platform.git
ID      DATE                           REVISION
0       2026-08-26 22:16:43 -0400 EDT  main (c7cb9a1)
1       2026-08-26 22:17:02 -0400 EDT  main (c7cb9a1)
$ argocd app set demo-staging --sync-policy automated --self-heal

argocd app get --refresh forces a re-comparison against git; --hard-refresh also drops the manifest cache, which is what you reach for when the repo changed but Argo insists it did not.

Argo CD 3.x defaults and the vocabulary of status

what changed in 3.0 to 3.5 · conditions · operation phases · deletion · refresh

Defaults that differ from what older tutorials assume

Annotation tracking is the default (application.resourceTrackingMethod: annotation): resources carry argocd.argoproj.io/tracking-id: <app>:<group>/<kind>:<ns>/<name> instead of the app.kubernetes.io/instance label. annotation+label keeps the label for other tools. A copied resource whose annotation names a different object is ignored, which is how HNC copies stop showing as drift.
Fine-grained RBAC: update and delete on applications no longer cover the Application's child resources; those need update/* and delete/* (or a specific update/apps/Deployment/ns/name). logs, get is a first-class permission; without it the logs tab is blank.
Default resource.exclusions drop high-churn kinds from the cache: Endpoints, EndpointSlice, Lease, the SubjectAccessReview family, CertificateSigningRequest, Kyverno reports, cert-manager CertificateRequest, Cilium identities and endpoints. If a task asks why Argo CD "does not see" a Lease, that is why.
Status ignored in diffs for all kinds (previously CRDs only), so a controller writing .status never causes OutOfSync.
3.1 added oci:// sources and CLI plugins; 3.2 loosened OCI layer rules and made UI deletion consistent; 3.3 requires server-side apply to install the ApplicationSet CRD (it exceeds the client-side annotation limit); 3.4 changed the cluster version format and Teams notifications; 3.5 (August 2026) made the Source Hydrator and impersonation beta, added internal mTLS, git commit signature verification, ApplicationSets in any namespace and Helm 4 support.

Status fields

Field	Values	Read it when
status.sync.status	Synced · OutOfSync · Unknown	Unknown means comparison itself failed: look at conditions
status.health.status	Healthy · Progressing · Degraded · Suspended · Missing · Unknown	aggregate of resource health; Missing means a desired resource does not exist live
status.operationState.phase	Running · Terminating · Succeeded · Failed · Error	Failed is an apply rejected by the API server; Error is Argo itself failing (render, network); Terminating follows `argocd app terminate-op`
status.operationState.syncResult.resources[].status	Synced · SyncFailed · Pruned · PruneSkipped	per-resource result with the API server's message verbatim
status.conditions[].type	ComparisonError · InvalidSpecError · SyncError · DeletionError · UnknownError · SharedResourceWarning · RepeatedResourceWarning · ExcludedResourceWarning · OrphanedResourceWarning	errors block, warnings do not; `SharedResourceWarning` is the two-apps-one-resource case, `InvalidSpecError` the AppProject refusal
status.sourceHydrator.currentOperation.phase	Hydrating · Hydrated · Failed	only with the hydrator

An InvalidSpecError with "application repo X is not permitted in project Y" or "destination namespace Z is not permitted" is the AppProject speaking; the fix is the project's sourceRepos or destinations, not the Application.

Deletion and refresh mechanics

Cascade deletion is the finalizer resources-finalizer.argocd.argoproj.io (foreground) or resources-finalizer.argocd.argoproj.io/background. argocd app delete X adds it; --cascade=false removes it and leaves the workload running (the "stop managing but keep" move). With kubectl, patch the finalizer in or out before deleting. PreDelete and PostDelete hooks run only on Application deletion and block it with a DeletionError condition if they fail.
Refresh without the CLI: annotate the Application argocd.argoproj.io/refresh: normal or hard; the controller removes the annotation when done. argocd.argoproj.io/skip-reconcile: "true" freezes an Application entirely (alpha; status stops updating).
Hook cleanup defaults to BeforeHookCreation when no hook-delete-policy is set; do not add ttlSecondsAfterFinished to hook Jobs, because Argo may then wait for a Job Kubernetes already deleted. Pruning runs waves in reverse; the delay between waves is 2 s (ARGOCD_SYNC_WAVE_DELAY).

Reflex

argocd app get X then kubectl -n argocd get app X -o jsonpath='{.status.conditions}'. The first tells you sync and health; the second names the error type, and the type tells you which object to edit: InvalidSpecError → AppProject or spec; ComparisonError → repo-server or credentials; SyncError → the API server's rejection.

ApplicationSet behavior, RBAC and notifications

generator specifics · policies · progressive syncs · policy.csv · subscriptions · sync windows

Generators: what each one yields

list: literal elements; the simplest fan-out and the one to reach for when a task gives you three clusters by name.
clusters: one set of parameters per cluster Secret (name, server, and the Secret's labels as metadata.labels.*); selector filters on those labels; the local cluster appears only if registered.
git directories: path: apps/* yields path.path, path.basename, path.basenameNormalized, path.segments; exclude: true entries remove directories; dot-directories are skipped. git files: every matching JSON or YAML file becomes one set of parameters with its keys.
matrix: cartesian product of exactly two generators; a child may use the other's parameters only if it comes second; combination generators nest one level only.
merge: base generator overridden by later generators on mergeKeys; same nesting rules.
scmProvider (repos in an org) and pullRequest (open PRs matching labels, branchMatch, titleMatch): Applications appear and disappear with the repo or PR, which is the preview-environment pattern. requeueAfterSeconds polls; webhooks make it instant.
plugin (RPC to your own service), clusterDecisionResource (a placement CR), oci (directories or files in an OCI artifact).

Templating: goTemplate: true with goTemplateOptions: ["missingkey=error"] so a typo fails loudly instead of rendering an empty string; normalize and slugify make names DNS-safe. Two generated Applications with the same name silently collapse into one.

Controlling what the controller may do

spec.syncPolicy.applicationsSync: sync (default: create, update, delete), create-only, create-update, create-delete. The controller flag --policy overrides unless policy override is enabled.
spec.syncPolicy.preserveResourcesOnDeletion: true keeps child resources when the ApplicationSet is deleted; to keep the Applications themselves you also need the finalizer on the ApplicationSet and background deletion.
spec.ignoreApplicationDifferences lets humans toggle auto-sync on a generated Application without the controller reverting it.
Progressive syncs (experimental, must be enabled): strategy.type: RollingSync with steps[].matchExpressions over Application labels and maxUpdate; each group must be Healthy before the next starts, and it forces generated Applications to auto-sync disabled.

RBAC in one table

Line in policy.csv	Meaning
p, role:team-a, applications, get, team-a/*, allow	role may read every Application in project team-a (object is `<project>/<app>`)
p, role:team-a, applications, sync, team-a/*, allow	may sync them; `update`, `delete`, `override`, `action/*` are separate verbs
p, role:team-a, logs, get, team-a/*, allow	needed since 3.0 to see pod logs in the UI
g, oidc-group-team-a, role:team-a	binds an SSO group (or user) to the role
policy.default: role:readonly	what an authenticated user gets with no other rule; empty means nothing

Resources you can name: applications, applicationsets, clusters, projects, repositories, accounts, certificates, gpgkeys, logs, exec, extensions. The same grammar works inside an AppProject's spec.roles[].policies, scoped to that project, and project roles can mint JWTs for CI. Project-scoped repositories and clusters (a project: field on the Secret) let a team register its own sources without seeing anyone else's.

Notifications and sync windows

Config lives in argocd-notifications-cm: service.slack, service.webhook.<name>, trigger.on-sync-failed, template.app-sync-failed; secrets in argocd-notifications-secret. Subscribing is an annotation on the Application or AppProject: notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my-channel. Installing the catalog gives you the standard triggers (on-deployed, on-health-degraded, on-sync-failed, on-sync-status-unknown).
Sync windows on the AppProject: kind: allow|deny, cron schedule, duration, and selectors by applications, namespaces, clusters; manualSync: true lets a human sync during a deny window; active deny beats active allow.

How this gets tested

"Team A may deploy only from repo R into namespace N and must be able to sync but not delete" is an AppProject with sourceRepos: [R], destinations: [{server, namespace: N}], and a project role whose policies grant applications, get and applications, sync but not delete, bound to the team's group. Verify by reading the Application's conditions after pointing it somewhere forbidden: InvalidSpecError is the proof the guardrail works.

Exercises

tick the dot when its check passes

Apply the example and predict its output before looking:

kubectl apply -f examples/argocd-appset.yaml
kubectl -n argocd get applications
kubectl -n team-a get deploy staging-demo && kubectl -n team-b get deploy prod-demo

outputcaptured 2026-08-26

$ kubectl apply -f examples/argocd-appset.yaml
applicationset.argoproj.io/demo-envs created
$ kubectl -n argocd get applications
NAME           SYNC STATUS   HEALTH STATUS
demo-prod      Synced        Progressing
demo-staging   Synced        Progressing
$ kubectl -n team-a get deploy staging-demo && kubectl -n team-b get deploy prod-demo
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
staging-demo   0/3     3            0           15s
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
prod-demo   0/3     3            0           16s

One nuance: the ApplicationSet template sets destination.namespace to the directory basename, but the overlays pin namespace: team-a and team-b in their kustomizations. An explicit namespace in a manifest always wins over the Application's destination default, so the workloads land in the tenant namespaces, prefixed staging- and prod-. If an app is stuck, read its conditions: kubectl -n argocd get app demo-staging -o jsonpath='{.status.conditions}' | jq.

verify: demo-staging and demo-prod Applications exist (one per overlay directory in the platform repo), each Synced/Healthy.

Your replica change from the fundamentals section is already in the platform repo, so demo-staging picks it up on the next poll (up to ~3 min) or immediately with argocd app sync demo-staging.

verify: kubectl -n team-a get deploy staging-demo -o jsonpath='{.spec.replicas}' matches your commit, and argocd app history demo-staging shows the revision.

The appset template sets automated + selfHeal + prune:

kubectl -n team-a scale deploy staging-demo --replicas=5
kubectl -n team-a get deploy staging-demo -w

outputcaptured 2026-08-26

$ kubectl -n team-a scale deploy staging-demo --replicas=5
deployment.apps/staging-demo scaled
$ kubectl -n team-a get deploy staging-demo -w
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
staging-demo   3/5     3            3           21s
staging-demo   3/5     3            3           21s
staging-demo   3/5     5            3           22s
staging-demo   3/3     5            3           22s
staging-demo   3/3     5            3           23s
staging-demo   3/3     5            3           23s
staging-demo   3/3     3            3           23s
staging-demo   3/3     3            3           26s
staging-demo   3/3     3            3           55s
^C

Then prove you understand prune the safe way: the service manifest lives in demo-app/base/, so drop service.yaml from the base kustomization's resources list, push, and watch the live Service disappear from both tenants (a base edit hits every overlay, so mind the blast radius). Revert the commit and watch them return.

verify: replicas snap back to the git value within seconds, argocd app get demo-staging never settles OutOfSync, and the Service round-trips with the commit. Git history is your undo.

Add to the staging overlay a manifest for a CR whose CRD does not exist (any made-up kind), push, and read the sync error. Then fix it properly for the case where you control both: put the CRD in the same overlay with argocd.argoproj.io/sync-wave: "-1" and the CR at wave 0. The other tool for this job is the argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true annotation, for when the CRD arrives from somewhere you don't control.

verify: sync succeeds and kubectl get <your-kind> returns the CR. Clean up the overlay afterwards; you'll reuse it.

Push an image tag that doesn't exist (newTag: does-not-exist in the staging kustomization), sync, and read the two statuses side by side: argocd app get demo-staging | head -20. One timing note: health shows Progressing for up to ten minutes (the Deployment's progressDeadlineSeconds) before flipping to Degraded. The pod's ImagePullBackOff is visible in events immediately, and Progressing-with-a-broken-pod already tells you everything.

verify: Sync status Synced, health on its way to Degraded, and you can articulate why clicking sync again is useless here. Revert the commit. This exact confusion is section 2.6's opening scenario.

Sync status and health are summaries. operationState is the record of the last attempt, with a result code per resource, and it is where a failed sync explains itself. Capture it clean, then break one resource and read it again.

# the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
argocd app wait demo-staging --operation --timeout 180 || true
argocd app sync demo-staging --timeout 120 || true
kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{range .status.operationState.syncResult.resources[*]}{.kind}/{.name} {.status} {.message}{"\n"}{end}'
kubectl -n team-a delete deploy staging-demo --cascade=orphan
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: { name: staging-demo, namespace: team-a }
spec:
  replicas: 1
  selector: { matchLabels: { app: something-else } }
  template:
    metadata: { labels: { app: something-else } }
    spec:
      containers: [{ name: web, image: nginx:1.27-alpine }]
EOF
argocd app wait demo-staging --operation --timeout 180 || true
argocd app sync demo-staging --timeout 120 || true
kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{range .status.operationState.syncResult.resources[*]}{.kind}/{.name} {.status} {.message}{"\n"}{end}'
kubectl -n team-a delete deploy staging-demo
argocd app wait demo-staging --operation --timeout 180 || true
argocd app sync demo-staging --timeout 120

outputcaptured 2026-09-13

$ # the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
$ ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
$ argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
'admin:login' logged in successfully
Context '172.18.0.4:32015' updated
$ argocd app wait demo-staging --operation --timeout 180 || true

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (83a41e2)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      83a41e21322e15eff2307bc1e9d5c89e33226d0d
Phase:              Succeeded
Start:              2026-09-13 00:53:59 -0400 EDT
Finished:           2026-09-13 00:54:00 -0400 EDT
Duration:           1s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo unchanged
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged
$ argocd app sync demo-staging --timeout 120 || true
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T00:54:56-04:00            Service      team-a          staging-demo    Synced  Healthy              
2026-09-13T00:54:56-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              
2026-09-13T00:54:57-04:00            Service      team-a          staging-demo    Synced  Healthy              service/staging-demo unchanged
2026-09-13T00:54:57-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              deployment.apps/staging-demo unchanged

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (83a41e2)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      83a41e21322e15eff2307bc1e9d5c89e33226d0d
Phase:              Succeeded
Start:              2026-09-13 00:54:56 -0400 EDT
Finished:           2026-09-13 00:54:57 -0400 EDT
Duration:           1s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo unchanged
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{range .status.operationState.syncResult.resources[*]}{.kind}/{.name} {.status} {.message}{"\n"}{end}'
Succeeded
Service/staging-demo Synced service/staging-demo unchanged
Deployment/staging-demo Synced deployment.apps/staging-demo unchanged
$ kubectl -n team-a delete deploy staging-demo --cascade=orphan
deployment.apps "staging-demo" deleted from team-a namespace
$ kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: { name: staging-demo, namespace: team-a }
spec:
  replicas: 1
  selector: { matchLabels: { app: something-else } }
  template:
    metadata: { labels: { app: something-else } }
    spec:
      containers: [{ name: web, image: nginx:1.27-alpine }]
EOF
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "web" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "web" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "web" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "web" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/staging-demo created
$ argocd app wait demo-staging --operation --timeout 180 || true
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME       STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T00:54:59-04:00            Service      team-a          staging-demo       Synced  Healthy              
2026-09-13T00:54:59-04:00   apps  Deployment      team-a          staging-demo       Synced  Missing              
2026-09-13T00:54:59-04:00   apps  ReplicaSet      team-a  staging-demo-66955f8974            Healthy              
2026-09-13T00:54:59-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Missing              
2026-09-13T00:55:00-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Missing              error when patching "/dev/shm/2994384382": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:55:10-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Healthy              error when patching "/dev/shm/2994384382": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:55:10-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Healthy              error when patching "/dev/shm/1714539329": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:55:30-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Healthy              error when patching "/dev/shm/128196255": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:56:10-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Healthy              error when patching "/dev/shm/1706085184": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:57:30-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Healthy              error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable

This is the state of the app after wait timed out:

The command timed out waiting for the conditions to be met.

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        OutOfSync from main (83a41e2)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      83a41e21322e15eff2307bc1e9d5c89e33226d0d
Phase:              Running
Start:              2026-09-13 00:54:59 -0400 EDT
Finished:           2026-09-13 00:57:30 -0400 EDT
Duration:           2m31s
Message:            one or more objects failed to apply, reason: error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable. Retrying attempt #5 at 4:58AM.

GROUP  KIND        NAMESPACE  NAME                     STATUS     HEALTH   HOOK  MESSAGE
apps   Deployment  team-a     staging-demo             OutOfSync  Healthy        error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
... 3 more lines
$ argocd app sync demo-staging --timeout 120 || true
{"level":"fatal","msg":"rpc error: code = FailedPrecondition desc = another operation is already in progress","time":"2026-09-13T00:57:59-04:00"}
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{range .status.operationState.syncResult.resources[*]}{.kind}/{.name} {.status} {.message}{"\n"}{end}'
Running
Deployment/staging-demo SyncFailed error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
$ kubectl -n team-a delete deploy staging-demo
deployment.apps "staging-demo" deleted from team-a namespace
$ argocd app wait demo-staging --operation --timeout 180 || true
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME       STATUS    HEALTH        HOOK  MESSAGE
2026-09-13T00:58:00-04:00   apps  Deployment      team-a          staging-demo     OutOfSync  Healthy              error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable
2026-09-13T00:58:00-04:00            Service      team-a          staging-demo       Synced   Healthy              
2026-09-13T00:58:00-04:00   apps  ReplicaSet      team-a  staging-demo-66955f8974             Healthy              
2026-09-13T01:00:10-04:00   apps  Deployment      team-a          staging-demo  OutOfSync  Missing              error when patching "/dev/shm/1824439474": Deployment.apps "staging-demo" is invalid: spec.selector: Invalid value: {"matchLabels":{"app":"demo","app.kubernetes.io/part-of":"demo-app"}}: field is immutable

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (83a41e2)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      83a41e21322e15eff2307bc1e9d5c89e33226d0d
Phase:              Succeeded
Start:              2026-09-13 00:54:59 -0400 EDT
Finished:           2026-09-13 01:00:10 -0400 EDT
Duration:           5m11s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo created
       Service     team-a     staging-demo  Synced  Healthy        
$ argocd app sync demo-staging --timeout 120
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T01:00:11-04:00            Service      team-a          staging-demo    Synced  Healthy              
2026-09-13T01:00:11-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              
2026-09-13T01:00:13-04:00            Service      team-a          staging-demo    Synced  Healthy              service/staging-demo unchanged
2026-09-13T01:00:13-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              deployment.apps/staging-demo unchanged

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (83a41e2)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      83a41e21322e15eff2307bc1e9d5c89e33226d0d
Phase:              Succeeded
Start:              2026-09-13 01:00:11 -0400 EDT
Finished:           2026-09-13 01:00:12 -0400 EDT
Duration:           1s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo unchanged
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged

verify: the clean run ends Succeeded with every resource Synced. After the selector is broken, operationState carries Deployment/staging-demo SyncFailed error when patching ...: spec.selector: Invalid value ...: field is immutable, which is the sentence you would paste into an incident channel. The manual sync in the middle may also come back another operation is already in progress: this app syncs automatically, so the controller was already doing the same work.

An AppProject is the boundary between what a tenant may ask for and what the platform allows. When it refuses, the Application does not fail to sync, it fails to be valid, and the difference tells you where to look.

# the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
kubectl -n argocd patch appproject default --type merge -p '{"spec":{"sourceRepos":["http://gitea.lab:3000/lab/platform.git"]}}'
# a throwaway Application: argocd app set tests repo connectivity before it writes, and the
# ApplicationSet that generated demo-staging puts the old repo back within seconds
kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: forbidden-repo, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination: { server: https://kubernetes.default.svc, namespace: default }
EOF
sleep 20
kubectl -n argocd get app forbidden-repo -o jsonpath='{.status.conditions}' | jq '.[] | {type, message}'
kubectl -n argocd delete app forbidden-repo
kubectl -n argocd patch appproject default --type merge -p '{"spec":{"sourceRepos":["*"]}}'
kubectl -n argocd get app demo-staging -o jsonpath='{.status.sync.status}{"\n"}'

outputcaptured 2026-09-13

$ # the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
$ ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
$ argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
'admin:login' logged in successfully
Context '172.18.0.4:32015' updated
$ kubectl -n argocd patch appproject default --type merge -p '{"spec":{"sourceRepos":["http://gitea.lab:3000/lab/platform.git"]}}'
appproject.argoproj.io/default patched
$ # a throwaway Application: argocd app set tests repo connectivity before it writes, and the
$ # ApplicationSet that generated demo-staging puts the old repo back within seconds
$ kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: forbidden-repo, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination: { server: https://kubernetes.default.svc, namespace: default }
EOF
application.argoproj.io/forbidden-repo created
$ sleep 20
$ kubectl -n argocd get app forbidden-repo -o jsonpath='{.status.conditions}' | jq '.[] | {type, message}'
{
  "type": "InvalidSpecError",
  "message": "application repo https://github.com/argoproj/argocd-example-apps.git is not permitted in project 'default'"
}
$ kubectl -n argocd delete app forbidden-repo
application.argoproj.io "forbidden-repo" deleted from argocd namespace
$ kubectl -n argocd patch appproject default --type merge -p '{"spec":{"sourceRepos":["*"]}}'
appproject.argoproj.io/default patched
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.sync.status}{"\n"}'
Synced

verify: the condition on forbidden-repo is InvalidSpecError and its message reads application repo https://github.com/argoproj/argocd-example-apps.git is not permitted in project 'default'. The API server accepted the object and the controller rejected it, which is why the fault is in status rather than in the apply. Do not try this by pointing demo-staging at another repo: argocd app set tests repository connectivity before it writes, and the ApplicationSet that generated demo-staging puts the old repo back within seconds.

Cascading deletion is a finalizer, not a law. Removing an Application without cascade leaves every object it managed running and unowned, which is exactly what you want during a migration and exactly what you must not do by accident.

# the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
# a standalone Application, not one the ApplicationSet generates: deleting a generated one
# only makes the generator recreate it, and deleting the generator takes the workload with it
kubectl -n argocd delete app orphan-demo --ignore-not-found
kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orphan-demo
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: default
  source:
    repoURL: http://gitea.lab:3000/lab/platform.git
    targetRevision: main
    path: demo-app/base
  destination: { server: https://kubernetes.default.svc, namespace: orphan }
  syncPolicy:
    automated: {}
    syncOptions: [CreateNamespace=true]
EOF
argocd app wait orphan-demo --health --timeout 180
kubectl -n argocd get app orphan-demo -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl -n orphan get deploy
argocd app delete orphan-demo --cascade=false -y
sleep 15
kubectl -n argocd get app orphan-demo || echo 'application gone'
kubectl -n orphan get deploy
kubectl delete ns orphan

outputcaptured 2026-09-12

$ # the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
$ ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
$ argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
'admin:login' logged in successfully
Context '172.18.0.4:32015' updated
$ # a standalone Application, not one the ApplicationSet generates: deleting a generated one
$ # only makes the generator recreate it, and deleting the generator takes the workload with it
$ kubectl -n argocd delete app orphan-demo --ignore-not-found
$ kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orphan-demo
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: default
  source:
    repoURL: http://gitea.lab:3000/lab/platform.git
    targetRevision: main
    path: demo-app/base
  destination: { server: https://kubernetes.default.svc, namespace: orphan }
  syncPolicy:
    automated: {}
    syncOptions: [CreateNamespace=true]
EOF
Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers
application.argoproj.io/orphan-demo created
$ argocd app wait orphan-demo --health --timeout 180
TIMESTAMP  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T11:58:57-04:00            Service      orphan                  demo  OutOfSync  Missing              
2026-09-13T11:58:57-04:00   apps  Deployment      orphan                  demo  OutOfSync  Missing              
2026-09-13T11:58:58-04:00          Namespace                            orphan   Running   Synced              namespace/orphan created

Name:               argocd/orphan-demo
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          orphan
URL:                https://argocd.example.com/applications/orphan-demo
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/base
SyncWindow:         Sync Allowed
Sync Policy:        Automated
Sync Status:        OutOfSync from main (b793ef2)
Health Status:      Healthy


GROUP  KIND        NAMESPACE  NAME    STATUS     HEALTH   HOOK  MESSAGE
       Namespace              orphan  Running    Synced         namespace/orphan created
       Service     orphan     demo    Synced     Healthy        service/demo created
apps   Deployment  orphan     demo    OutOfSync  Missing        deployment.apps/demo created
$ kubectl -n argocd get app orphan-demo -o jsonpath='{.metadata.finalizers}{"\n"}'
["resources-finalizer.argocd.argoproj.io"]
$ kubectl -n orphan get deploy
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
demo   0/2     2            0           1s
$ argocd app delete orphan-demo --cascade=false -y
application 'orphan-demo' deleted
$ sleep 15
$ kubectl -n argocd get app orphan-demo || echo 'application gone'
Error from server (NotFound): applications.argoproj.io "orphan-demo" not found
application gone
$ kubectl -n orphan get deploy
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
demo   2/2     2            2           17s
$ kubectl delete ns orphan
namespace "orphan" deleted

verify: the Application is gone and the Deployment it created is still running, unowned. Use a standalone Application for this: deleting one the ApplicationSet generated only makes the generator put it back within seconds, and deleting the generator instead takes the workload with it, because the generated Application still carries the cascade finalizer. The finalizer is the whole mechanism, and --cascade=false is what removes it without letting it run.

argocd app get --refresh is a convenience over an annotation. Knowing the annotation means you can force a refresh from any machine that can reach the API server, including one with no Argo CD CLI on it.

BEFORE=$(kubectl -n argocd get app demo-staging -o jsonpath='{.status.reconciledAt}'); echo "before: $BEFORE"
kubectl -n argocd annotate app demo-staging argocd.argoproj.io/refresh=hard --overwrite
sleep 10
kubectl -n argocd get app demo-staging -o jsonpath='{.metadata.annotations}' | jq
kubectl -n argocd get app demo-staging -o jsonpath='{.status.reconciledAt}{"\n"}' | sed 's/^/after:  /'

outputcaptured 2026-09-12

$ BEFORE=$(kubectl -n argocd get app demo-staging -o jsonpath='{.status.reconciledAt}'); echo "before: $BEFORE"
before: 2026-09-13T10:45:51Z
$ kubectl -n argocd annotate app demo-staging argocd.argoproj.io/refresh=hard --overwrite
application.argoproj.io/demo-staging annotated
$ sleep 10
$ kubectl -n argocd get app demo-staging -o jsonpath='{.metadata.annotations}' | jq
{
  "argocd.argoproj.io/hydrate": "normal"
}
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.reconciledAt}{"\n"}' | sed 's/^/after:  /'
after:  2026-09-13T10:49:16Z

verify: the annotation is gone by the time you read it back and reconciledAt has moved. The controller consumes the annotation as its signal, which is why you never see it in a healthy application.

Automatic pruning is the feature that deletes production when someone deletes a file. Prune=confirm turns the delete into a request that waits for a human, and the sync sits there saying so.

# the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
# git -C rather than cd: a cd here would follow you into every later command in the session
rm -rf /tmp/platform-prune && git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform-prune
# set it on the template: the ApplicationSet controller overwrites a spec edit on the generated Application
kubectl -n argocd patch applicationset demo-envs --type merge -p '{"spec":{"template":{"spec":{"syncPolicy":{"syncOptions":["CreateNamespace=true","Prune=confirm"]}}}}}'
sleep 15
git -C /tmp/platform-prune rm demo-app/base/service.yaml
sed -i '/service.yaml/d' /tmp/platform-prune/demo-app/base/kustomization.yaml
git -C /tmp/platform-prune commit -am 'drop the service' && git -C /tmp/platform-prune push
argocd app sync demo-staging --prune --timeout 60 || true
kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{.status.operationState.message}{"\n"}'
kubectl -n argocd annotate app demo-staging argocd.argoproj.io/deletion-approved="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite
sleep 20
kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}'
git -C /tmp/platform-prune revert --no-edit HEAD && git -C /tmp/platform-prune push
# unset removes the option; setting Prune=false would just record a different value
kubectl -n argocd patch applicationset demo-envs --type merge -p '{"spec":{"template":{"spec":{"syncPolicy":{"syncOptions":["CreateNamespace=true"]}}}}}'
argocd app sync demo-staging --timeout 120
kubectl -n argocd get app demo-staging -o jsonpath='{.spec.syncPolicy.syncOptions}{"\n"}'
rm -rf /tmp/platform-prune

outputcaptured 2026-09-13

$ # the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
$ ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
$ argocd login "$ARGO" --username admin --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" --plaintext --grpc-web
'admin:login' logged in successfully
Context '172.18.0.4:32015' updated
$ # git -C rather than cd: a cd here would follow you into every later command in the session
$ rm -rf /tmp/platform-prune && git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform-prune
Cloning into '/tmp/platform-prune'...
$ # set it on the template: the ApplicationSet controller overwrites a spec edit on the generated Application
$ kubectl -n argocd patch applicationset demo-envs --type merge -p '{"spec":{"template":{"spec":{"syncPolicy":{"syncOptions":["CreateNamespace=true","Prune=confirm"]}}}}}'
applicationset.argoproj.io/demo-envs patched
$ sleep 15
$ git -C /tmp/platform-prune rm demo-app/base/service.yaml
rm 'demo-app/base/service.yaml'
$ sed -i '/service.yaml/d' /tmp/platform-prune/demo-app/base/kustomization.yaml
$ git -C /tmp/platform-prune commit -am 'drop the service' && git -C /tmp/platform-prune push
[main b761904] drop the service
 2 files changed, 8 deletions(-)
 delete mode 100644 demo-app/base/service.yaml
To http://gitea.lab:3000/lab/platform.git
   894975d..b761904  main -> main
$ argocd app sync demo-staging --prune --timeout 60 || true
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T08:06:10-04:00            Service      team-a          staging-demo    Synced  Healthy              
2026-09-13T08:06:10-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              

This is the state of the app after wait timed out:

The command timed out waiting for the conditions to be met.

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (894975d)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      b761904983e66ec86d904ff126285f518ac6182b
Phase:              Running
Start:              2026-09-13 08:06:10 -0400 EDT
Finished:           <nil>
Duration:           1m0s
Message:            waiting for pruning confirmation of /Service/staging-demo

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        
apps   Deployment  team-a     staging-demo  Synced  Healthy        
{"level":"fatal","msg":"timed out (60s) waiting for app \"demo-staging\" match desired state","time":"2026-09-13T08:07:10-04:00"}
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}{.status.operationState.message}{"\n"}'
Running
waiting for pruning confirmation of /Service/staging-demo
$ kubectl -n argocd annotate app demo-staging argocd.argoproj.io/deletion-approved="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite
application.argoproj.io/demo-staging annotated
$ sleep 20
$ kubectl -n argocd get app demo-staging -o jsonpath='{.status.operationState.phase}{"\n"}'
Succeeded
$ git -C /tmp/platform-prune revert --no-edit HEAD && git -C /tmp/platform-prune push
[main 1d38064] Revert "drop the service"
 Date: Sun Sep 13 08:07:31 2026 -0400
 2 files changed, 8 insertions(+)
 create mode 100644 demo-app/base/service.yaml
To http://gitea.lab:3000/lab/platform.git
   b761904..1d38064  main -> main
$ # unset removes the option; setting Prune=false would just record a different value
$ kubectl -n argocd patch applicationset demo-envs --type merge -p '{"spec":{"template":{"spec":{"syncPolicy":{"syncOptions":["CreateNamespace=true"]}}}}}'
applicationset.argoproj.io/demo-envs patched
$ argocd app sync demo-staging --timeout 120
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2026-09-13T08:07:32-04:00   apps  Deployment      team-a          staging-demo    Synced  Healthy              
2026-09-13T08:07:32-04:00            Service      team-a          staging-demo  OutOfSync  Healthy              
2026-09-13T08:07:33-04:00            Service      team-a          staging-demo  OutOfSync  Healthy              service/staging-demo created
2026-09-13T08:07:33-04:00   apps  Deployment      team-a          staging-demo    Synced   Healthy              deployment.apps/staging-demo unchanged

Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          staging
URL:                https://argocd.example.com/applications/demo-staging
Source:
- Repo:             http://gitea.lab:3000/lab/platform.git
  Target:           main
  Path:             demo-app/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (1d38064)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      1d38064a22b6d6e6ee6d5acbc0044745d3960f3a
Phase:              Succeeded
Start:              2026-09-13 08:07:32 -0400 EDT
Finished:           2026-09-13 08:07:33 -0400 EDT
Duration:           1s
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo created
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged
$ kubectl -n argocd get app demo-staging -o jsonpath='{.spec.syncPolicy.syncOptions}{"\n"}'
["CreateNamespace=true"]
$ rm -rf /tmp/platform-prune

verify: the operation does not reach Succeeded, its message asks for the pruning to be confirmed, and the Service is still there until you apply the argocd.argoproj.io/deletion-approved annotation. Set the option on the ApplicationSet template, not on the generated Application: the ApplicationSet controller overwrites a spec edit on demo-staging within seconds and the prune then happens unattended, which is what makes this look like it worked when it did not.

A project role plus a token is how a tenant gets to sync their own application and nothing else. Hand the token to the role, then use it: a denial names the exact rule it wanted, which is the fastest way to see whether you granted what you meant to. Note that argocd account can-i does not answer for a project token, only for a logged-in account, so the token itself is the test.

ARGO_PW=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
# the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
argocd login "$ARGO" --username admin --password "$ARGO_PW" --plaintext --grpc-web
argocd proj role create default deployer
# add-policy prepends the project, so the object here is the bare app name: 'default/demo-staging' would become default/default/demo-staging and match nothing
argocd proj role add-policy default deployer --action get --permission allow --object 'demo-staging'
kubectl -n argocd get appproject default -o jsonpath='{.spec.roles[0].policies}{"\n"}'
TOKEN=$(argocd proj role create-token default deployer -t)
argocd app get demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web | head -3
argocd app sync demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web --timeout 30 2>&1 | tail -1
argocd proj role add-policy default deployer --action sync --permission allow --object 'demo-staging'
TOKEN=$(argocd proj role create-token default deployer -t)
argocd app sync demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web --timeout 60 2>&1 | tail -2
argocd proj role delete default deployer

outputcaptured 2026-09-13

$ ARGO_PW=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
$ # the LoadBalancer IP only exists when cloud-provider-kind runs; the NodePort is always there
$ ARGO=$(kubectl get node cnpe-control-plane -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'):$(kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
$ argocd login "$ARGO" --username admin --password "$ARGO_PW" --plaintext --grpc-web
'admin:login' logged in successfully
Context '172.18.0.4:32015' updated
$ argocd proj role create default deployer
Role 'deployer' created
$ # add-policy prepends the project, so the object here is the bare app name: 'default/demo-staging' would become default/default/demo-staging and match nothing
$ argocd proj role add-policy default deployer --action get --permission allow --object 'demo-staging'
$ kubectl -n argocd get appproject default -o jsonpath='{.spec.roles[0].policies}{"\n"}'
["p, proj:default:deployer, applications, get, default/demo-staging, allow"]
$ TOKEN=$(argocd proj role create-token default deployer -t)
$ argocd app get demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web | head -3
Name:               argocd/demo-staging
Project:            default
Server:             https://kubernetes.default.svc
$ argocd app sync demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web --timeout 30 2>&1 | tail -1
{"level":"fatal","msg":"rpc error: code = PermissionDenied desc = permission denied: applications, sync, default/demo-staging, sub: proj:default:deployer, iat: 2026-09-13T05:00:15Z","time":"2026-09-13T01:00:15-04:00"}
$ argocd proj role add-policy default deployer --action sync --permission allow --object 'demo-staging'
$ TOKEN=$(argocd proj role create-token default deployer -t)
$ argocd app sync demo-staging --auth-token "$TOKEN" --server "$ARGO" --plaintext --grpc-web --timeout 60 2>&1 | tail -2
       Service     team-a     staging-demo  Synced  Healthy        service/staging-demo unchanged
apps   Deployment  team-a     staging-demo  Synced  Healthy        deployment.apps/staging-demo unchanged
$ argocd proj role delete default deployer
Role 'deployer' deleted

verify: with only the get policy the token reads the app but the sync is refused with permission denied: applications, sync, default/demo-staging, sub: proj:default:deployer, and after the sync policy is added the same command syncs. The denial names the action, the object and the subject, which is three of the four things you need to fix it.

Self-check

answer before opening

An app is Synced/Degraded. What are you allowed to conclude, and what is the fix?

Live matches git, so Argo CD has done its job; the desired state itself is broken (bad image, impossible resources, missing config key). The fix is a commit. Re-syncing changes nothing because the diff is already empty.

A sync fails with "field is immutable". Two ways forward?

Set Replace=true (per app or per resource) so the sync uses kubectl replace instead of apply, or delete the resource and let Argo recreate it; Force=true combines the two. Choose deliberately: replace is not a free pass, since a Service manifest that omits clusterIP is still rejected as immutable, and a shrinking PVC is not going to be saved by either option.

What is an AppProject for, in one sentence, and name two things it constrains?

It is the guardrail around a set of Applications: permitted source repos and permitted destinations (cluster + namespace), plus allowed/denied resource kinds, per-project RBAC roles and sync windows. It is the answer to "let this team self-serve deployments without letting them deploy anything anywhere".

You are handed an ApplicationSet with a git directory generator over apps/*/overlays/*. How many Applications will exist?

One per matching directory in the repo at the current revision, so count the directories, then check the template for a name expression that could collide (two identical names silently overwrite). Predicting the exact set from the generator block is the testable skill; kubectl -n argocd get applications then confirms it.

An app shows OutOfSync forever on a field no human edits. What is happening and what fixes it?

Something in-cluster mutates the resource after apply: a mutating webhook, an HPA writing replicas, a defaulting controller. Fix with ignoreDifferences for that path (or stop managing the field). This is a configuration decision, not a bug to chase.

You changed the repo but Argo insists nothing changed. Sequence of moves?

argocd app get --refresh first (recompare), then --hard-refresh (drop the manifest cache), then check the app is on the branch you pushed to (targetRevision), then check repo-server logs for auth or render errors. Cheapest first, always.

A sync ends with operation phase Error rather than Failed. What is the difference and where do you look?

Failed means the apply reached the API server and was rejected (immutable field, admission denial); the message is in syncResult.resources[]. Error means Argo CD itself could not complete the operation: rendering failed, the repo was unreachable, a hook could not be created. For Error, read repo-server and application-controller logs; for Failed, read the resource message.

You must stop managing an Application but keep its workloads running. Exact move?

argocd app delete X --cascade=false, or with kubectl: patch metadata.finalizers to remove resources-finalizer.argocd.argoproj.io, then delete the Application. Deleting with the finalizer present cascades in the foreground and removes every managed resource. Delete=false as a resource annotation is the per-resource version of the same protection.

An ApplicationSet with a matrix of a git directories generator and a clusters generator: how many Applications, and what breaks if you nest another matrix inside?

directories × clusters, one Application per pair, named by the template (collisions silently merge). Matrix takes exactly two child generators and combination generators nest only one level deep, so a matrix inside a matrix inside a matrix is rejected at generation time, not by the API server's schema validation.

A user can see Applications and sync them but the Logs tab is empty. Which policy line is missing since 3.0?

p, role:<their-role>, logs, get, <project>/*, allow. From 3.0, logs RBAC is enforced by default and is no longer implied by applications, get. The same 3.0 change split update and delete on child resources into update/* and delete/*.

Docs to know your way around

study time, not exam time

argo-cd.readthedocs.io: Application spec reference, sync options, sync waves and hooks, AppProject, ApplicationSet generators.
Offline: argocd app --help covers most of what the docs would, and kubectl explain application.spec --recursive works because the Application is just a CRD.
argo-cd.readthedocs.io: Upgrading v2.14 to 3.0 (and 3.x to 3.y): annotation tracking, fine-grained RBAC, logs RBAC, default resource exclusions, ignored status; the per-minor pages list every behavioral change since.
argo-cd.readthedocs.io: RBAC Configuration and Projects: the policy.csv grammar, resource and action lists, project roles and JWT tokens, sync windows.
argo-cd.readthedocs.io: ApplicationSet Generators (per generator) and Controlling Resource Modification: parameters each generator emits, matrix/merge restrictions, applicationsSync policies, preserveResourcesOnDeletion.