Grafana is the pane of glass over both halves of this section: Prometheus metrics through dashboards, Loki logs through Explore. Credentials are genuinely admin/admin here; address from make urls.

needsmake up obs

Orientation

competency 4.1 · logging and actionable dashboards

The competency wording is "dashboards that provide actionable insight", which is a judgement claim, not a tooling claim. Two things make it real: dashboards shipped as code, and dashboards designed around a question someone actually asks during an incident.

Dashboards as code

the sidecar pattern

The exam-relevant fact about Grafana in a kube-prometheus-stack world: dashboards are provisioned, not clicked together. A sidecar container in the Grafana pod watches for ConfigMaps labelled grafana_dashboard: "1" and loads any dashboard JSON it finds inside. That is why the stack's forty-odd dashboards exist without anyone importing them, and it is how you ship a dashboard in a GitOps world: JSON in a ConfigMap in a git repo. Datasources provision the same way with grafana_datasource: "1".

git ──▶ ConfigMap (label grafana_dashboard=1) ──▶ sidecar writes /tmp/dashboards ──▶ Grafana loads it
                                                          │
   a dashboard clicked together in the UI lives only in Grafana's database
   and dies with the pod ────────────────────────────────┘

What makes a dashboard actionable rather than decorative

  • Lead with the user-facing symptom. RED for services (Rate, Errors, Duration); USE for resources (Utilization, Saturation, Errors); the four golden signals (latency, traffic, errors, saturation) if you prefer that vocabulary. Know all three acronyms and which applies to what; it is a cheap question to set.
  • Template over a variable ($namespace, $pod) instead of hardcoding, so one dashboard serves every tenant. Variables come from label_values() queries, which is how the stock dashboards populate their dropdowns.
  • Every panel answers a question someone would ask during an incident. Forty panels of everything is a wall, not a dashboard.
  • Annotations and links. Deployment markers turn "when did this start" into a glance, and a panel link to the matching logs query turns a metric spike into a root cause.

Grafana specifics worth naming once: datasources (Prometheus, Loki, Jaeger here), panels with unit and threshold settings, alerting (Grafana has its own alert engine, deliberately unused in this lab; Prometheus rules are the exam's model), and folders + permissions for multi-team setups.

Loki and LogQL

index labels, not content

Loki indexes labels, not content: log lines are stored against a small label set (namespace, pod, container, app) and content matching happens at query time by scanning the selected streams. That is why it is cheap to run, and why every query must start from a label selector before filtering. It is also why label cardinality discipline matters even more than in Prometheus: a label per request ID would create a stream per request.

Collection here is Alloy, a single-replica deployment in monitoring that tails every pod through the kubelet API and pushes to Loki; Loki itself stores them as a StatefulSet next door. Two hops, and make validate proves the pair works by demanding Loki actually returns streams, because a running Loki with no shipper looks healthy and holds nothing.

LogQL pieceExampleDoes
stream selector{namespace="team-a"}required first step; picks streams
line filter|= "error" · != "/healthz" · |~ "5[0-9]{2}"substring / regex match on the line
parser| json · | logfmt · | patternextract fields into labels for filtering
label filter| status >= 500filter on extracted fields
metric querysum by (pod) (rate({ns="x"} |= "error" [5m]))turns logs into a graphable series

Those five constructs cover exam depth. The one to internalise is the last: a metric query over logs is how you alert on something that only exists in text, and it is the bridge between this section and 4.2.

Terminal beats UI, often

stern <pattern> -n <ns> tails many pods at once with colour-coded names, and kubectl logs --previous is the only way to read a crashed container's last words. Loki is for "what happened an hour ago across the fleet"; stern and kubectl are for "what is happening right now". Knowing which question you are asking picks the tool.

The three signals, and when logs are the right one

Metrics answer "how much / how often" cheaply over long windows. Logs answer "what exactly happened to this one thing", at higher cost per query. Traces (4.4) answer "where in the request path did it go wrong". An exam scenario that says "find the error message" is a log question; "how many failed" is a metric question; "which service is slow" is a trace question. Choosing wrong costs minutes.

Exercises

tick the dot when its check passes

In Grafana: Connections → Data sources. If Loki is not there, add it the provisioned way rather than the UI way:

kubectl -n monitoring apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-datasource
  labels: { grafana_datasource: "1" }
data:
  loki.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki
        type: loki
        url: http://loki.monitoring.svc:3100
        access: proxy
EOF

The sidecar picks it up within a minute or so. If nothing arrives, bisect the two hops: shipper first (kubectl -n monitoring get deploy alloy, then its logs for push errors), Loki second.

verify: Explore → Loki → {namespace="monitoring"} returns log lines. Being able to say which hop is broken is the exercise inside the exercise.

Generate known lines, then find them:

kubectl -n team-a run chatty --image=busybox:1.37 --restart=Never -- \
  sh -c 'for i in $(seq 60); do echo "level=error msg=payment_failed attempt=$i"; sleep 1; done'

In Explore: {namespace="team-a"} |= "payment_failed", then the metric form sum(rate({namespace="team-a"} |= "payment_failed" [1m])).

verify: lines visible and the rate curve is about 1/s while the pod runs. Compare stern chatty -n team-a for the same lines live.

Build one panel in the UI first (New dashboard → add visualization → Prometheus → sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))), then export its JSON (share → Export → save JSON, set "export for sharing externally" off), and re-deliver it properly:

kubectl -n monitoring create configmap team-cpu-dashboard --from-file=dash.json=<your-file> \
  --dry-run=client -o yaml | kubectl label -f - --local --dry-run=client -o yaml grafana_dashboard=1 | kubectl apply -f -
verify: delete the hand-made dashboard in the UI, and the provisioned copy appears (default folder) and survives a Grafana pod delete, which the clicked one would not have. State the GitOps moral in one sentence.

Open "Kubernetes / Compute Resources / Namespace (Pods)" for team-a while the chatty pod runs. Answer: which panels are RED, which are USE, and which single panel would you keep if you could keep one during a pod-crash incident.

verify: there is no single right answer; having an answer with a reason is the skill the "actionable insights" wording points at.

Self-check

answer before opening
How does a dashboard get into Grafana in a GitOps platform, and why not click it?

JSON in a ConfigMap labelled grafana_dashboard: "1", delivered from git; the sidecar loads it. A clicked dashboard lives only in Grafana's database: unreviewable, unversioned, and gone with the pod unless persistence is configured.

Explain RED and USE, and which you would use for a queue worker.

RED = Rate, Errors, Duration, for request-driven services. USE = Utilization, Saturation, Errors, for resources. A queue worker is best served by both plus queue depth: USE for the worker pool, RED-ish for message processing, and saturation is really "is the backlog growing".

Loki returns nothing for {namespace="team-a"}. Two hops to check, in order.

The shipper first (Alloy running, not erroring on push, actually tailing that namespace), then Loki (ingester healthy, retention not eating the window, right tenant/org headers if multi-tenancy is on). A healthy Loki with no shipper looks perfectly fine and holds nothing, which is exactly why make validate demands streams, not readiness.

Why is a per-request-ID label a bad idea in Loki?

Labels define streams, and Loki's index is per stream. A unique label value per request creates a stream per request: index explosion, slow queries, high memory. Put high-cardinality identifiers in the log line and filter or parse them at query time; that is what Loki's design is optimised for.

Turn "alert me when payments start failing" into a LogQL-based rule, conceptually.

A metric query over logs: sum(rate({namespace="payments"} |= "payment_failed" [5m])) > 0.1, evaluated by Loki's ruler (or mirrored into Prometheus via a recording pipeline). Same alerting mechanics as 4.2; the only difference is where the series comes from.

Docs to know your way around

study time, not exam time
  • grafana.com/docs: provisioning (datasources and dashboards), dashboard variables, and the LogQL reference.
  • The Explore UI's query builder doubles as LogQL documentation under time pressure.
  • Offline: stern --help, kubectl logs --previous, and the Grafana panel inspector (it shows the exact query and the raw response).