Tracing answers the one question metrics and logs cannot: where, along a request's path through many services, did the time go or the failure start. The vocabulary is small and the exam expects it exact.
make up obsOrientation
Two things make tracing tasks solvable: knowing the exact OTLP address telemetry should go to, and knowing that a trace is only a trace if context propagated. Everything else is UI.
The OTel operator lives in opentelemetry-operator-system, the collector in tracing, Jaeger alongside it. So the in-cluster OTLP endpoints are otel-collector.tracing.svc:4317 (gRPC) and :4318 (HTTP). Memorise the shape <svc>.<ns>.svc:4317; every tracing task starts by knowing that address.
Vocabulary and plumbing
A trace is one request's journey. A span is one named, timed operation within it, carrying a trace ID, its own span ID, a parent span ID, a status, attributes and events. A trace is therefore a tree, and the root span is the entry point.
Context propagation is what links spans across service boundaries: the caller injects the trace context into headers, the callee extracts it and continues the trace instead of starting a new one. The standard header is W3C traceparent (plus tracestate, and baggage for user-defined key/values that travel with the request). Older stacks use B3 headers; a mixed fleet needs a propagator configured for both.
Broken propagation: every service reports spans, and every span is its own orphan trace. Symptom in Jaeger: many single-span traces, one per service, none joined. Causes: a proxy or gateway stripping unknown headers, an SDK without the right propagator, a manual HTTP client that never injects, or an async hop (queue, cron) where nobody thought to carry the context. When you see one-span traces from an instrumented system, name that cause immediately.
The OpenTelemetry pipeline
app (SDK or auto-instrumentation agent)
│ OTLP gRPC :4317 / HTTP :4318
▼
Collector receivers ──▶ processors ──▶ exporters
otlp, jaeger, batch, memory_limiter, otlp/jaeger,
prometheus, attributes, k8sattributes, logging,
filelog tail_sampling prometheus
▼
backend Jaeger (traces) · Prometheus (metrics) · Loki (logs)
The collector is the platform team's control point: sampling, batching, redaction and fan-out happen there, not in application code. That sentence is the platform-engineering answer to "how do we control telemetry cost": you change one deployment, not forty services.
| Sampling | Decides | Trade |
|---|---|---|
| Head-based | at the start of the trace, usually a fixed probability | cheap and simple; you may drop the one slow request you needed |
| Tail-based | after the trace completes, on its properties (errors, latency) | keeps the interesting traces; the collector must buffer whole traces, so it costs memory and needs all spans of a trace at one collector |
Kubernetes-side: two CRDs
OpenTelemetryCollector: the operator deploys and configures a collector (as a Deployment, DaemonSet, StatefulSet or Sidecar). Read its config to see the receiver/exporter chain.Instrumentation: the zero-code path. The lab ships one namedautoindefault; annotating a pod spec withinstrumentation.opentelemetry.io/inject-java: "default/auto"(or-python,-nodejs,-dotnet) makes a webhook inject the language agent as an init container at admission.
Annotate an existing Deployment and nothing happens to the running pods; they must restart to pick it up. And the annotation belongs on the pod template, not on the Deployment's own metadata. Those two mistakes account for most "the annotation does nothing" reports, and both are one-line fixes once you know to look.
Exercises
kubectl -n tracing get cm -o yaml | grep -A30 'receivers:' (or the OpenTelemetryCollector CR if present). Draw the chain on paper: which receivers listen, which exporter points at Jaeger, what processors sit between.
otel-collector.tracing.svc:4317 or the HTTP twin). Every tracing task starts by knowing that address.telemetrygen is the collector project's own traffic generator, and it makes tracing exercises deterministic:
kubectl run telemetrygen --restart=Never \
--image=ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
-- traces --otlp-endpoint otel-collector.tracing.svc:4317 --otlp-insecure \
--service curriculum-drill --traces 20 --child-spans 3make urls): service curriculum-drill appears in the dropdown, 20 traces, each of 4 spans (one root plus three children: four spans, two levels). Then verify the API way, because exams grade with curl: curl -s 'http://<jaeger>/api/traces?service=curriculum-drill&limit=1' | jq '.data[0].spans | length' returns 4.Open one trace, read the waterfall: parent/child structure, per-span duration, attributes on each span. Answer for that trace: which span is the critical path, and what would you look at next if the leaf span were slow (that span's service's logs, at that timestamp, and now you know why traces carry IDs you can grep logs for).
Run a small Java service (ghcr.io/open-telemetry/opentelemetry-java-examples images work, or any Spring Boot sample), annotate its pod template with instrumentation.opentelemetry.io/inject-java: "default/auto", restart, and hit its endpoint. If it does not appear, the diagnostic ladder is: annotation on the pod template (not the Deployment metadata), pod restarted since annotating, Instrumentation CR namespace/name correct in the annotation value, operator webhook alive.
kubectl describe pod shows the injected init container, and the service appears in Jaeger with HTTP spans you never wrote.No cluster needed: service A calls B through a proxy that strips unknown headers. Describe what Jaeger shows and which header must survive.
traceparent and predicts orphaned single-service traces, the concept is yours.Self-check
Jaeger shows hundreds of one-span traces. Diagnosis?
Context propagation is broken: each service starts a new trace instead of continuing the caller's. Look for a proxy stripping traceparent, a missing or mismatched propagator (W3C vs B3), a hand-rolled HTTP client that never injects, or an async boundary nobody instrumented.
Head-based versus tail-based sampling: when does the difference bite?
When the interesting traces are rare. Head sampling at 1% will usually miss the one 5-second request that mattered; tail sampling keeps traces that contain errors or exceed a latency threshold, at the cost of buffering complete traces in the collector (and needing all spans of a trace to reach the same collector instance).
Where should sampling and redaction be configured, and why there?
In the collector. It is the platform's control point: one config change applies to every service, whereas SDK-side changes need forty teams to redeploy. This is the same "interface owned by the platform" argument as section 3.1, applied to telemetry.
You annotated a Deployment for auto-instrumentation and nothing changed. Two most likely reasons?
The annotation is on the Deployment's own metadata rather than spec.template.metadata, or the pods have not restarted since (injection happens at admission, at pod creation). Third candidate: the annotation value's <namespace>/<name> does not match an existing Instrumentation CR.
How do traces, metrics and logs tie together in practice?
Exemplars link metric buckets to sample trace IDs; logs carry the trace ID as a field so you can jump from a slow span to its exact log lines; and the collector can emit all three from the same pipeline with the same resource attributes (namespace, pod, service). "Correlated by trace ID and resource attributes" is the sentence.
Docs to know your way around
- opentelemetry.io: the Concepts pages (signals, context propagation), the collector's receivers/processors/exporters reference, and the Kubernetes operator's Instrumentation injection docs.
- jaegertracing.io: mostly the UI is self-explanatory; know that the HTTP API exists for scripted verification.
- w3.org/TR/trace-context: one page, and it is the header everything agrees on.