Domain 3 hands you four ways to turn a request into resources: CRD+operator, Crossplane, workflows, and portal templates. This section adds the newest option, walks the lab's end-to-end golden path, and then forces the comparison, because "evaluate when to use operators, workflows, or pipelines" is a named outcome and a cheap scenario question.
make core apimake portalOrientation
Everything in domain 3 has been one engine at a time. This section is about the seam between them, which is where platform engineering actually happens: a Backstage form that commits to git, that an ApplicationSet notices, that deploys an XR, that a controller reconciles. Being able to narrate that chain is worth more than any single command in it.
kro in one sitting
kro (Kube Resource Orchestrator) occupies Crossplane's niche with far less apparatus: one ResourceGraphDefinition declares a schema and the resources it expands to; kro generates the CRD and runs the reconciliation. No providers, no functions, no packages. CEL expressions wire fields together, and references between resources (${ns.metadata.name}) build the dependency graph automatically; kro works out the ordering from the references you wrote.
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata: { name: tenantspace }
spec:
schema:
apiVersion: v1alpha1
kind: TenantSpace
spec:
team: string
cpu: string | default="1"
resources:
- id: ns
template:
apiVersion: v1
kind: Namespace
metadata:
name: ${schema.spec.team}
labels:
tenant: ${schema.spec.team}
- id: quota
template:
apiVersion: v1
kind: ResourceQuota
metadata:
name: tenant-quota
namespace: ${ns.metadata.name}
spec:
hard:
requests.cpu: ${schema.spec.cpu}Keep kro's ${...} substitutions in block style. Inside flow-style braces (labels: { tenant: ${schema.spec.team} }) the expression's own } closes the map early and the whole manifest fails to parse with a message that points nowhere useful.
kro is young and its API moves. If this manifest disagrees with your installed version, kubectl explain resourcegraphdefinition.spec is the arbiter, and practising that recovery is worth more than the manifest.
| Crossplane | kro | |
|---|---|---|
| Files to write | XRD + Composition (+ providers, functions) | one ResourceGraphDefinition |
| Wiring | patches / function pipeline | CEL references, ordering inferred |
| External clouds | rich provider ecosystem | whatever CRDs already exist in-cluster |
| Connection secrets | first-class | not really |
| Maturity | production, widely deployed | young, API in motion |
Both sentences are true: "same result, one file, no providers" and "no provider ecosystem, no connection secrets, alpha-grade stability". An exam answer may need either.
The golden path, hop by hop
Backstage is the portal half of the white paper's "web portals" capability: a software catalog (components, systems, APIs, owners, described by catalog-info.yaml in each repo), software templates (the scaffolder: parameters form + actions like fetch-skeleton, publish-to-git, register-in-catalog), TechDocs, and plugins that surface Kubernetes, CI and other systems in one place.
Backstage template (parameters form)
│ scaffolder actions: fetch skeleton → publish → register
▼
new repo in Gitea org services (contains k8s/ manifests + catalog-info.yaml)
│ noticed by the SCM-provider generator
▼
ApplicationSet golden-path ──generates──▶ Argo CD Application
│
▼
running workload ── and the component appears in the catalog with its owner
Each hop is a thing that can break and a thing you may be asked about: template action failures (credentials to the git host), the generator's filter (which repos match), the generated Application's destination, and the workload itself. Narrating this chain is the "golden path" competency; the commands below just prove each hop.
A common scenario trap: the portal is an interface over capabilities, not a capability itself. If the underlying provisioning is a ticket queue, adding Backstage produces a nicer ticket queue. Say that when a question describes a portal rollout with no automation behind it.
Choosing the engine
| Engine | Trigger | Lifetime | Best at |
|---|---|---|---|
| CRD + operator | an API object | forever | domain behaviour: failover, backup, upgrade |
| Crossplane | an API object | forever | composing many resources, incl. cloud APIs, behind one abstraction |
| kro | an API object | forever | the same, in-cluster only, with one file |
| Argo Workflows | submit, schedule, or event | run to completion | imperative sequences, batch, scheduled maintenance |
| Tekton | git event | run to completion | build, test, publish artifacts |
| Backstage template | a human filling a form | one-shot scaffolding | the front door: create repo + register + hand off to GitOps |
Three questions decide almost every case:
- Does the state need reconciling forever, or producing once? Forever → controller-shaped (operator, Crossplane, kro). Once → workflow or pipeline.
- What is the trigger? An API object, a schedule, a git event, or a human.
- Who is the audience? A developer writing YAML, a developer clicking a form, or another system.
And when two engines both work, say so and pick the thinnest one. That answer style scores, because it is the TVP instinct from section 3.1 applied to your own tooling.
Exercises
Apply the RGD above, then:
kubectl get rgd tenantspace -o jsonpath='{.status.state}{"\n"}' # wants: Active
kubectl apply -f - <<'EOF'
apiVersion: kro.run/v1alpha1
kind: TenantSpace
metadata: { name: team-h, namespace: default }
spec: { team: team-h, cpu: "2" }
EOF
kubectl get ns team-h && kubectl -n team-h get resourcequota tenant-quotaWith make portal done and Backstage running (cd portal && yarn start), create a component from the "Golden path service" template. Then trace every hop with commands, because each hop is a thing that can break: new repo in the Gitea services org (http://gitea.lab:3000/services), the SCM-generator ApplicationSet notices it (kubectl -n argocd get applicationset golden-path -o yaml, read the generator block), a generated Application appears (kubectl -n argocd get applications), workload deploys.
make validate flips the portal checks to green, including "Applications auto-generated from git". Being able to narrate the pipeline from template to running pod is a better exam asset than any single command in it.For each request, pick the engine and one sentence of why: (a) every team needs a Postgres with backups that self-heals for years; (b) on request, stamp a namespace with quotas and labels from three parameters; (c) nightly, rebuild and rescan all base images, notify on failure; (d) on every push, build, test, and deploy a service; (e) offer non-technical users a form that creates a new service from a skeleton.
Self-check
How does kro know to create the namespace before the quota?
From the reference: the quota template uses ${ns.metadata.name}, so kro builds a dependency graph from the CEL references and orders accordingly. You never write dependsOn, which is elegant, and also means a refactor that removes a reference can silently reorder your graph.
A team asks for "a portal so developers can self-serve". What do you ask before agreeing?
What happens after the form is submitted. If fulfillment is still manual, the portal is a prettier ticket queue; the automation (templates → git → GitOps → controllers) is the actual work, and the portal is the last 10%. That is the white paper's interface-versus-capability distinction.
Nightly image rebuild-and-rescan: which engine, and why not the others?
A CronWorkflow: scheduled, run-to-completion, DAG with retries and notifications. Not an operator (nothing to reconcile forever), not Tekton (no git event trigger, though it could run the pipeline), not Crossplane (no resource graph to maintain).
Give the three questions that decide the engine.
Does state need reconciling forever or producing once? What is the trigger: API object, schedule, git event, human? Who is the audience: YAML author, form filler, or another system? Answer those three and the table picks itself.
What does the SCM-provider generator actually watch, and how would you debug a repo that never appears?
It queries the git host's API for repos in an org matching filters (name pattern, topic, path existence), then templates one Application per match. Debug by reading the generator block (org, filters, credentials Secret), then checking the ApplicationSet's status/conditions and controller logs for API errors, usually a token scope or a filter that does not match.
Docs to know your way around
- kro.run: the ResourceGraphDefinition documentation and examples.
- backstage.io: Software Templates (actions and parameters) and the catalog model, plus the lab's own template at
backstage/template/template.yaml, which is small enough to read whole. - Offline:
kubectl explain resourcegraphdefinition.spec,kubectl -n argocd get applicationset <name> -o yaml.