SherlockLiu Logo SherlockLiu
Back to all posts
Engineering

Practical OpenTelemetry — Part 12: Adopting Observability

SL
Aug 23, 2026 11 min read
Practical OpenTelemetry — Part 12: Adopting Observability

Chapter 12 closes Practical OpenTelemetry the way it opened: not with an API, but with a question. Part 1 asked what observability actually means; this chapter asks how you know, concretely, whether you’ve achieved it — and what keeps a team from quietly regressing back into the “dashboards in one window, logs in another” workflow the whole book has been arguing against.

The Effectiveness Test

All the API precision from Parts 3 through 10 is wasted if engineers keep isolated workflows — a metrics dashboard in one tab, application logs in another, no habit of moving between them. The book’s fix isn’t a tooling mandate, it’s cultural: observability “champions” who challenge old workflows with evidence, not by decree.

The single most useful idea in the chapter is a test you can run on your own team, today, regardless of what tooling you have:

Debug a regression with zero prior knowledge of the system. If you reach the same root cause as the service owners, in a similar or shorter time, your observability is working. If you don’t, it isn’t — no matter how much telemetry you’re collecting.

This is worth sitting with, because it’s falsifiable in a way “we have good observability” isn’t. It doesn’t ask whether you have traces, metrics, and logs. It asks whether a stranger, armed with nothing but what those signals actually surface, can do the job a domain expert used to need years of tribal knowledge to do. Part 1’s Payment Service scenario — one engineer, no prior system knowledge, finding a non-direct dependency’s root cause — is this test, worked out as a story instead of a checklist.

Signal Strengths, in the Order the Book States Them

A quick hierarchy worth internalizing, because it corrects a common instinct to treat every signal as equally load-bearing:

  1. Metrics-driven alerts are the most reliable trigger — cheap, stable, always-on, exactly the strengths Part 3’s signal comparison table laid out.
  2. Semantic conventions are what make correlation possible at all — without them, Part 2’s Mars Climate Orbiter problem recurs at the scale of every dashboard in the org.
  3. Context links metrics → exemplars → traces → logs, in strictly increasing granularity — Part 7’s exemplars are the literal mechanism, one click from an aggregate to the individual transaction behind it.
  4. Context propagation is what connects services into one debuggable system — without Part 5’s propagators, every one of the first three points stops at a single service’s boundary.

Postmortems That Measure the Right Thing

The book’s postmortem guidance is specific enough to act on directly: focus on learning, not blame, and measure and document incident-lifecycle metrics — MTTD, MTTRes, and their sub-clocks from Part 1 — against real targets, not vibes. Three questions the book says every postmortem should answer:

  • Why weren’t engineers alerted sooner?
  • Which signals actually got used during the investigation?
  • Would a different signal, or a different correlation, have been faster?

That third question is really asking about one specific clock from Part 1’s family: MTTK, the time from “something’s wrong” to “I know why.” Most postmortem templates measure how fast the team recovered and stop there — this question is the one that actually improves the observability setup rather than just documenting what happened, because it’s the only one that interrogates the debugging process itself instead of its outcome. An observability guild — a standing forum sharing postmortem analyses, near-misses, and plain “dos and don’ts” across teams — is the book’s proposed mechanism for making those lessons compound instead of staying siloed inside whichever team happened to be paged.

Expanding Context Beyond One Service

No single service can achieve observability alone — the book insists on thinking in terms of the system as a whole, not the service boundary, which is really Part 1’s original argument restated as an operating principle. Auto-instrumentation is double-edged here: a faulty or simply missing instrumentation hop produces disjointed traces — “breaking the chain,” the same phrase Part 11 used for propagator migrations, now describing an instrumentation gap instead.

Practical rollout order: start from key transactions at the system’s actual entry point — browser XHR instrumentation generating CLIENT spans is the book’s example — and expand context inward toward the backend, rather than instrumenting services in whatever order teams happen to get to them.

Two concrete, checkable validation signals the book gives for “is context actually propagating”:

  • SERVER span names should be meaningful, not generic placeholders like HTTP GET.
  • A CLIENT span with no corresponding SERVER span downstream is a direct, unambiguous signal that context isn’t propagating across that specific hop — not a vague “something feels off.”

And a testing practice worth adopting wholesale: use in-memory exporters in unit tests to assert context propagation actually happens, so instrumentation correctness becomes part of the SDLC rather than something discovered for the first time during an incident. Pair that with the social side — multi-team sessions examining real traces together — and context stops being something one team owns and starts being something the whole organization can read.

Diagram of two validation signals for context propagation: a SERVER span with a meaningful name confirms instrumentation is working, while a CLIENT span with no matching downstream SERVER span signals a broken hop. Is Context Actually Propagating? Working CLIENT span net.peer.name set SERVER span /api/customer/{id} Broken hop CLIENT span net.peer.name set no SERVER span at all A meaningful SERVER name confirms propagation; a CLIENT span with no matching SERVER span names the exact broken hop. Make it part of the SDLC In-memory exporters in unit tests assert propagation before it ever reaches production

Figure: two cheap, concrete checks for context propagation — one confirms it, one names exactly where it broke.

Enablement Over Control

The book’s closing philosophy is also its most quotable line:

“Enablement over control.”

The named anti-pattern is centralization/control/limitation — approving every signal request individually, keeping default limits low. The book’s diagnosis is blunt: this reduces velocity, creates toil, and the restrictive conditions never get re-evaluated once they’re in place, so the friction only ever accumulates.

The alternative rests on three ideas, each of which shows up as a real mechanism earlier in the series, not just as a slogan here:

  • Principle of least effort — the enablement team’s golden path (Part 11) should be the objectively easiest path to take, so nobody has a reason to route around good practice.
  • Autonomy with accountability — distribute telemetry cost visibility via proxy metrics (ingest volume, span count, cardinality), attributed by owner. This is mechanical, not managerial, because Part 3’s service.name + service.namespace resource attributes already carry that ownership on every single piece of telemetry exported.
  • Visibility drives informed decisions — teams that can see their own telemetry cost make better tradeoffs than teams working under an opaque, centrally-imposed limit.

Attributing the Bill, Concretely

The book grounds “autonomy with accountability” in the same cost-attribution model most platform teams already use for compute, then carries it over to telemetry. Take a worker node with 2 CPU / 8 GB running two containers: one reserving 0.5 CPU / 2 GB, the other 1.5 CPU / 6 GB. Nobody argues about splitting that node’s bill 25%/75% — it’s just each container’s share of what it reserved.

Telemetry doesn’t have a clean “reserved capacity” number, so the book proposes a proxy metric per backend, matched to whatever actually limits that backend:

Backend Limiting factor Proxy metric to attribute cost by
Time-series database (Prometheus, etc.) Cardinality A service’s share of a metric’s cardinality ÷ total cardinality across all metrics
Logging backend Storage volume Bytes stored, by service.name / service.namespace
Tracing backend Storage volume Bytes stored, by service.name / service.namespace

The mechanism only works because service.name and service.namespace are on every signal by construction (Part 3) — cost attribution needs no separate tagging discipline layered on top. And the book is explicit that this isn’t about penalizing scale: the same metric that’s negligible at 100 replicas can be the thing that tips a TSDB over at 10,000. The point of attribution isn’t to shame large deployments, it’s to make the consequence of this specific attribute visible to the person who added it, at the moment they’re deciding whether to keep it.

The book’s closing number is worth carrying past this series: you can rarely cut 80% of a compute bill, but 80% telemetry cost savings is a realistic bar — its own example is moving from verbose access logs to sampled, correlated tracing, which cuts cost and increases debugging value at the same time, because the actual win was never “store less.” It was always “store the right transactions” — Part 1’s argument, all the way back at the start, now paying for itself in a line item.

What’s Not in This Book

Practical OpenTelemetry was written in 2023, and the specification hasn’t stood still since. Two things worth knowing before you go looking for them and don’t find them:

  • Profiling is now a fourth core signal. Continuous profiling (stack samples correlated to the same trace_id/span_id as everything else in this series) was accepted into the specification after this book’s publication, following the same design instinct as Part 8’s logs chapter: correlate an existing, well-understood practice rather than reinvent it. Treat it as this series’ Part 8 all over again — same “enrich, don’t replace” philosophy, applied to pprof instead of Log4j.
  • The Logs signal has kept maturing. Part 8 covered it as the newest, least stable signal — that was accurate at the book’s writing, and the chapter’s own advice (data model is stable, appender/MDC integration is production-ready, don’t wait for full API stability) is exactly why adopting it early was never actually risky. Check the current specification for where the API and SDK stability markers stand today rather than trusting this series’ 2023 snapshot.

Neither changes anything you’d do differently in the field — the mental model this series builds (resources, propagators, semantic conventions, the Collector as an integration point) is what makes each new signal absorbable rather than another framework to learn from scratch.

The Series in One Paragraph

Observability is a property, not a product (Part 1). OpenTelemetry makes it portable by separating instrumentation from backends (Part 2) through a specification with real stability guarantees (Part 3). Auto-instrumentation makes adoption nearly free (Part 4); context and baggage make telemetry correlate (Part 5); traces give per-transaction detail (Part 6); metrics give stable aggregates (Part 7); logs contextualize what you already have (Part 8); the Collector and OTLP make it all deployable at scale (Part 9); sampling and deployment topology make it affordable (Part 10); and enablement, migration shims, and a culture that measures MTTK instead of guessing at it make it durable (Part 11, this post).

That’s the book. The rest is practice — starting with the effectiveness test above, on whatever system you own.


Series hub: Practical OpenTelemetry


References

Comments