Add observability: real OTLP metrics/traces to grafana/otel-lgtm, Docker Compose auto-wiring, and a dual-version (Boot 4.0 vs 4.1) proof of the new OTEL_* env var support

Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
2026-09-18 09:32:34 +00:00
parent 320733265f
commit 03bdf7ee87
28 changed files with 1027 additions and 2 deletions
@@ -0,0 +1,89 @@
+# 1. Migrating to spring-boot-starter-opentelemetry
+
+[README](../README.md) | Next: [2. The Observation API and @Observed](02-observation-api.md)
+
+## What the old post 4700 used, and what replaces it
+
+The original version of this article's post built a Prometheus + Grafana stack by hand: add
+`micrometer-registry-prometheus`, expose `/actuator/prometheus`, run a Prometheus container with
+a hand-written `prometheus.yml` scrape config pointed at `host.docker.internal:8080`, run a
+Grafana container, wire up a data source, import a community dashboard by ID. That is still a
+completely valid way to run Prometheus and Grafana, and nothing about it stopped working on
+Spring Boot 4.1.
+
+What's new is a fourth option, built by the Spring team and shipped as its own starter:
+
+```xml
+<dependency>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-opentelemetry</artifactId>
+</dependency>
+```
+
+One dependency replaces `micrometer-registry-prometheus` (or any other vendor-specific registry)
+plus a tracing bridge. It pulls in `micrometer-registry-otlp` and
+`micrometer-tracing-bridge-otel`, and switches the whole export model from **pull** (something
+scrapes `/actuator/prometheus` on a timer) to **push** (the app itself POSTs metrics and traces,
+in one vendor-neutral wire format, OTLP, to wherever you point it). This repo's module,
+[`observability/`](../), uses this starter exclusively -- the deep, beginner-to-advanced
+treatment of Micrometer versus OpenTelemetry, the Observation API, cardinality, and context
+propagation already exists on this blog:
+[Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/).
+This module exists to verify one specific, narrower claim that guide states in passing but does
+not itself demonstrate end to end: that Boot 4.1 will auto-wire the OTLP export path against a
+real local collector with zero `management.otlp.*` properties, using nothing but Docker Compose.
+
+## Docker Compose does the wiring, verified
+
+[`compose.yaml`](../compose.yaml) names one image:
+
+```yaml
+services:
+  lgtm:
+    image: grafana/otel-lgtm:latest
+    ports:
+      - "3000:3000"   # Grafana UI
+      - "4317:4317"   # OTLP gRPC ingest
+      - "4318:4318"   # OTLP HTTP ingest
+      - "9090:9090"   # Prometheus-compatible query API
+      - "3200:3200"   # Tempo query API (traces)
+```
+
+[`grafana/otel-lgtm`](https://github.com/grafana/docker-otel-lgtm) is a single image bundling
+Loki, Grafana, Tempo and (Mimir-backed) Prometheus, plus an OTLP collector endpoint -- the exact
+image the Spring team's own OpenTelemetry starter documentation uses as its local-development
+example. [`application.yml`](../src/main/resources/application.yml) has no `management.otlp.*`
+properties in it at all. Running `./scripts/run.sh` (which is `mvn spring-boot:run`) produces:
+
+```console
+DockerComposeLifecycleManager : Using Docker Compose file .../compose.yaml
+DockerCli :  Container obs-module-lgtm-1 Starting
+DockerCli :  Container obs-module-lgtm-1 Started
+DockerCli :  Container obs-module-lgtm-1 Healthy
+PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service}
+```
+
+([00-docker-compose-auto-wiring.txt](output/00-docker-compose-auto-wiring.txt)) The important
+detail is `127.0.0.1`, not `localhost`. Micrometer's own `OtlpConfig` defaults already point at
+`localhost:4318` with nothing configured at all, so an app that happens to reuse the default OTLP
+port could look auto-wired when it is actually just coincidental. `127.0.0.1` is what Boot's
+Docker Compose service-connection support specifically resolves the container to; if you see
+`localhost` instead, the auto-wiring did not actually happen and you are looking at the bare
+Micrometer default.
+
+## Real traffic, real metrics, real backend
+
+8 real HTTP requests to a real running server, queried back out of the real Prometheus-compatible
+API bundled inside the container -- not the app's own `/actuator/prometheus`, and no scraping
+involved at all, since this is push, not pull:
+
+```console
+$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"orders_placed_total", ...},"value":[..., "8"]}]}}
+```
+
+([01-metrics-and-traces-in-lgtm.txt](output/01-metrics-and-traces-in-lgtm.txt), source:
+[`OrderController.java`](../src/main/java/com/ankurm/observability/OrderController.java))
+
+Next: [2. The Observation API and @Observed](02-observation-api.md) -- where the same request
+that produced this metric turns out, on its own, to produce only half the trace you'd expect.
@@ -0,0 +1,79 @@
+# 2. The Observation API and @Observed
+
+Previous: [1. Migrating to spring-boot-starter-opentelemetry](01-migrating-to-opentelemetry-starter.md) | [README](../README.md) | Next: [3. OTEL_* environment variables](03-otel-env-vars.md)
+
+## One recording, two signals -- in theory
+
+[`OrderController.placeOrder`](../src/main/java/com/ankurm/observability/OrderController.java)
+is annotated:
+
+```java
+@Observed(name = "place-order", contextualName = "order-controller#placeOrder")
+@PostMapping("/orders/{id}")
+Map<String, Object> placeOrder(@PathVariable String id) throws InterruptedException {
+```
+
+The idea, per Micrometer's own Observation API and repeated in the
+[deeper guide on this blog](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/#observation-api),
+is that one annotation produces two signals from one recording: a timer (exported as a metric)
+and a trace span, nested under whatever span is already active -- normally the incoming HTTP
+request's own server span.
+
+## What actually happened the first time this was tried
+
+`spring-boot-starter-opentelemetry` was on the classpath. `spring-boot-starter-aspectj` (the
+AspectJ weaver -- see the rename covered in
+[the resilience4j post's chapter 8](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/resilience/docs/08-starter-aop-renamed-to-starter-aspectj.md))
+was added specifically because `@Observed` is AOP-based and needs a proxy to intercept the
+method at all. Micrometer Tracing was active -- the HTTP server span for every request already
+showed up correctly in Tempo. And the `@Observed` child span still never appeared:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
+```
+
+One span. Not two. No error, no warning, no failed startup -- the method-level span is just
+silently absent.
+
+## The missing piece: ObservedAspect is not autoconfigured
+
+Spring Boot's autoconfiguration wires an `ObservationRegistry` bean for you the moment Micrometer
+Tracing is on the classpath. It does **not** also register a Micrometer `ObservedAspect` bean --
+AspectJ weaving being present is necessary but not sufficient, because without the aspect there
+is nothing for the weaver to apply `@Observed` through. The fix is one small
+`@Configuration` class:
+
+```java
+@Configuration(proxyBeanMethods = false)
+class ObservationConfig {
+
+    @Bean
+    ObservedAspect observedAspect(ObservationRegistry registry) {
+        return new ObservedAspect(registry);
+    }
+}
+```
+
+([`ObservationConfig.java`](../src/main/java/com/ankurm/observability/ObservationConfig.java))
+Same annotation, same request, new trace:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
+span: http post /orders/{id}      | kind: SPAN_KIND_SERVER   | parent: (root)
+```
+
+([03-observed-needs-explicit-bean.txt](output/03-observed-needs-explicit-bean.txt)) Two spans,
+correctly nested. This is worth stating plainly because the natural reading of "Boot wires this
+up for you when Micrometer Tracing is active" is that `@Observed` works out of the box -- it does
+not, and the failure mode (silently missing, not broken) is the kind that survives code review.
+
+- If your own `@Observed` spans aren't showing up, check for this bean before anything else --
+  AOP proxying issues (self-invocation, `final` methods/classes) are the usual second suspect,
+  covered generally in
+  [the caching module's self-invocation chapter](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/caching/docs/03-self-invocation-trap.md).
+
+Next: [3. OTEL_* environment variables](03-otel-env-vars.md) -- and a second thing this module's
+own traces caught, unrelated to `@Observed`: most of a short burst of local traffic doesn't
+produce a trace at all, by design.
@@ -0,0 +1,71 @@
+# 3. OTEL_* environment variables, and the sampling gotcha they surface
+
+Previous: [2. The Observation API and @Observed](02-observation-api.md) | [README](../README.md) | Next: [4. Production checklist](04-production-checklist.md)
+
+## What changed in 4.1, verified against two real Boot versions
+
+Spring Boot's own [4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
+state plainly: "Support has been added to read most of the OpenTelemetry environment variables."
+That is easy to either overstate (as if no `OTEL_*` variable worked before) or take on faith. The
+real, empirical answer, from [`env-var-proof/`](../env-var-proof) -- **identical** Java source
+compiled against `spring-boot-starter-parent` 4.0.8 and 4.1.1, run with only standard `OTEL_*`
+environment variables set (no `management.otlp.*` Spring properties anywhere) and pointed at a
+minimal stand-in OTLP receiver:
+
+| Variable | Boot 4.0.8 | Boot 4.1.1 |
+|---|---|---|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Honored | Honored |
+| `OTEL_METRIC_EXPORT_INTERVAL` | **Ignored** (stays at the 1-minute default) | Honored |
+
+```console
+--- Boot 4.1.1, both env vars set ---
+Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+... POST /v1/metrics  (x11, one every ~2s)
+
+--- Boot 4.0.8, same two env vars ---
+Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
+... POST /v1/metrics  (x1, at JVM shutdown only)
+```
+
+([04-otel-env-vars-4.0-vs-4.1.txt](output/04-otel-env-vars-4.0-vs-4.1.txt)) The endpoint variable
+already worked on 4.0 -- Micrometer's own `OtlpConfig` has long fallen back to
+`OTEL_EXPORTER_OTLP_ENDPOINT` independent of anything Spring-specific. What's actually new in 4.1
+is the rest of the standard variable surface -- export interval, protocol, per-signal overrides --
+which previously required Boot's own `management.otlp.*` properties (still fully supported; see
+the same transcript for `MANAGEMENT_OTLP_METRICS_EXPORT_STEP` working identically on 4.0). The
+practical win: the exact same environment variables already used to configure an OTel Collector,
+or a sidecar written in another language, now also configure this Spring Boot app, with nothing
+Spring-specific to learn.
+
+- Full variable-to-property mapping: [Spring Boot 4.1 Release Notes, OpenTelemetry section](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
+- Reproduce this yourself: [`scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh)
+
+## The sampling gotcha this surfaced along the way
+
+Building the env-var proof meant staring at trace counts for a while, which surfaced something
+worth a section of its own. `management.tracing.sampling.probability` defaults to **0.10** and
+is completely independent of metrics -- metrics are never sampled at all.
+
+```console
+15 requests sent to /orders/{101..115}, default sampling (0.10):
+  orders_placed_total increases by 15   -- every request counted
+  Tempo trace count increases by only 1  -- roughly 1 in 10 requests actually traced
+
+Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0):
+  10 requests sent to /orders/{401..410}
+  Tempo trace count increases by 10      -- all of them, every time
+```
+
+([02-low-sampling-demo.txt](output/02-low-sampling-demo.txt), profile:
+[`application.yml`](../src/main/resources/application.yml)) The default is the right choice for
+production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable
+starting point. It is the wrong choice for a five-minute local demo: send a handful of test
+requests at the default rate and the honest, common experience is "my trace isn't showing up,"
+which reads exactly like a broken pipeline rather than working-as-designed sampling. `fulltrace`
+exists in this repo for that reason -- flip it on for local poking, not for anything that sees
+real traffic.
+
+- If you enable `spring-boot-starter-opentelemetry` and metrics show up in your backend but
+  traces don't, check the sampling probability before you suspect the exporter.
+
+Next: [4. Production checklist](04-production-checklist.md)
@@ -0,0 +1,17 @@
+# 4. Production checklist
+
+Previous: [3. OTEL_* environment variables](03-otel-env-vars.md) | [README](../README.md)
+
+| Item | This module's finding |
+|---|---|
+| Which starter | `spring-boot-starter-opentelemetry` -- one dependency for OTLP metrics + traces, replacing a per-vendor registry jar plus a tracing bridge |
+| Local development | A `compose.yaml` naming `grafana/otel-lgtm` gets auto-wired by Boot's Docker Compose support with zero `management.otlp.*` properties -- verified `127.0.0.1`, not the bare-default `localhost`, in the startup log |
+| Any other environment | Set the export endpoint explicitly -- either `management.otlp.*` properties or, as of 4.1, standard `OTEL_EXPORTER_OTLP_*` variables. Auto-wiring is a dev-time convenience only |
+| Standard `OTEL_*` env vars | `OTEL_EXPORTER_OTLP_ENDPOINT` worked before 4.1 too (a Micrometer default, not a Boot feature); the rest of the surface -- `OTEL_METRIC_EXPORT_INTERVAL` and friends -- is genuinely new in 4.1 |
+| `@Observed` | Needs an explicit `ObservedAspect` @Bean. AspectJ weaving present + Micrometer Tracing active is *not* sufficient on its own -- the annotation is silently inert without it |
+| Trace sampling | Defaults to 0.10. Metrics are never sampled. A "why don't my traces show up" question at low local traffic is very often just this |
+| Cardinality, context propagation, the Observation API in depth | Already covered start to finish: [Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/) |
+
+## License
+
+MIT -- part of the [spring-boot-demo](../../) container repository; see [../../LICENSE](../../LICENSE).
@@ -0,0 +1,24 @@
+$ mvn spring-boot:run
+...
+DockerComposeLifecycleManager : Using Docker Compose file /tmp/obs-module/compose.yaml
+DockerCli :  Network obs-module_default Creating
+DockerCli :  Network obs-module_default Created
+DockerCli :  Container obs-module-lgtm-1 Creating
+DockerCli :  Container obs-module-lgtm-1 Created
+DockerCli :  Container obs-module-lgtm-1 Starting
+DockerCli :  Container obs-module-lgtm-1 Started
+DockerCli :  Container obs-module-lgtm-1 Waiting
+DockerCli :  Container obs-module-lgtm-1 Healthy
+...
+PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service}
+Started ObservabilityApplication in 34.426 seconds (process running for 34.652)
+
+$ docker ps
+CONTAINER ID   IMAGE                      COMMAND                  STATUS                    PORTS                                                                NAMES
+4f768a81a102   grafana/otel-lgtm:latest   "/otel-lgtm/run-all...."   Up 46 seconds (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:3200->3200/tcp, 0.0.0.0:4317-4318->4317-4318/tcp, 0.0.0.0:9090->9090/tcp   obs-module-lgtm-1
+
+No management.otlp.* property was set anywhere in application.yml or on the command line for
+this run. The endpoint (127.0.0.1:4318) came entirely from Boot's Docker Compose service
+connection detecting the grafana/otel-lgtm image named in compose.yaml. Note the literal
+"127.0.0.1" -- a plain default (no compose.yaml, no service connection) reports "localhost" for
+this same port instead; that difference is the tell for whether auto-wiring actually happened.
@@ -0,0 +1,29 @@
+8 real requests sent to the running app:
+  $ for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i"; done
+
+Queried straight from the real Prometheus-compatible API bundled inside the grafana/otel-lgtm
+container -- not the app's own /actuator endpoint, not a mock:
+
+  $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+  {
+    "status": "success",
+    "data": {
+      "resultType": "vector",
+      "result": [
+        {
+          "metric": {
+            "__name__": "orders_placed_total",
+            "application": "order-service",
+            "channel": "web",
+            "job": "order-service",
+            "service_name": "order-service"
+          },
+          "value": [1789722945.151, "8"]
+        }
+      ]
+    }
+  }
+
+Every one of the 8 POSTs landed as a real OTLP metric, pushed over the network, through the
+Docker Compose auto-wired endpoint, and queried back out of the real Prometheus-compatible
+backend Grafana LGTM bundles.
@@ -0,0 +1,23 @@
+Default management.tracing.sampling.probability (0.10, unset in application.yml):
+
+  $ for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+  # 15 requests sent
+
+  $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+  # metric value: 15   -- all 15 counted, metrics are never subject to trace sampling
+
+  $ curl -s "http://localhost:3200/api/search?limit=50"
+  # trace count increased by 1  -- only ~1 of the 15 requests was sampled into a trace
+
+Same 15 requests, same app, same collector. Every one produced a metric data point. Roughly
+1 in 10 produced a trace, because trace sampling and metric recording are governed by two
+completely different knobs, and only one of them defaults to "record everything."
+
+--- with the fulltrace profile (management.tracing.sampling.probability=1.0) ---
+
+  $ mvn -Dspring-boot.run.profiles=fulltrace spring-boot:run
+  $ for i in $(seq 401 410); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+  # 10 requests sent
+
+  $ curl -s "http://localhost:3200/api/search?limit=50"
+  # post /orders traces increased by 10 -- 10 of 10, all of them present
@@ -0,0 +1,32 @@
+OrderController#placeOrder is annotated @Observed(name = "place-order", contextualName =
+"order-controller#placeOrder"). Before ObservationConfig (an explicit ObservedAspect @Bean)
+existed in this repo, spring-boot-starter-opentelemetry, spring-boot-starter-aspectj (AspectJ
+weaver on the classpath) and micrometer-tracing were ALL already present and active -- and the
+annotation still did nothing. Real trace, fetched straight from Tempo's query API, before the fix:
+
+  $ curl -s "http://localhost:3200/api/traces/<trace-id>"
+  span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
+  # one span. @Observed's child span never appears.
+
+After adding:
+
+  @Configuration(proxyBeanMethods = false)
+  class ObservationConfig {
+      @Bean
+      ObservedAspect observedAspect(ObservationRegistry registry) {
+          return new ObservedAspect(registry);
+      }
+  }
+
+Same annotation, same request, new trace:
+
+  $ curl -s "http://localhost:3200/api/traces/<trace-id>"
+  span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
+  span: http post /orders/{id}      | kind: SPAN_KIND_SERVER   | parent: (root)
+  # two spans. The @Observed child span now nests correctly under the HTTP server span.
+
+Spring Boot's autoconfiguration wires an ObservationRegistry bean for you the moment Micrometer
+Tracing is on the classpath -- it does NOT also register an ObservedAspect. AspectJ weaving
+being present is necessary but not sufficient; without the aspect bean there is nothing for the
+weaver to apply. This is easy to miss because the app starts cleanly, the HTTP server span still
+shows up, and nothing logs a warning -- the method-level span is just silently absent.
@@ -0,0 +1,43 @@
+Identical Java source (env-var-proof/OtelEnvProofApplication.java), compiled twice against two
+different spring-boot-starter-parent versions -- 4.0.8 and 4.1.1 -- run with ONLY standard OTEL_*
+environment variables set (zero management.otlp.* Spring properties anywhere), against a
+minimal stand-in OTLP receiver that just logs every POST it gets:
+
+  OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:PORT
+  OTEL_METRIC_EXPORT_INTERVAL=2000
+
+--- Spring Boot 4.1.1 ---
+
+  Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8247   (x11, one every ~2s)
+
+  Both the endpoint AND the interval standard env vars were honored.
+
+--- Spring Boot 4.0.8, same two env vars ---
+
+  Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8247   (x1, at JVM shutdown only)
+
+  The endpoint env var was honored (Micrometer's own OtlpConfig has long fallen back to
+  OTEL_EXPORTER_OTLP_ENDPOINT independent of Spring Boot's own property binding). The interval
+  env var was NOT -- the exporter still logs "every 1m", the JVM's own default, and the only POST
+  the receiver saw was the one every PushMeterRegistry fires on shutdown regardless of interval.
+
+--- Spring Boot 4.0.8, old-style Spring property instead ---
+
+  MANAGEMENT_OTLP_METRICS_EXPORT_URL=http://localhost:PORT/v1/metrics
+  MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s
+
+  Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8435   (x11, one every ~2s)
+
+  Boot 4.0's own management.otlp.* properties always worked for this. What's new in 4.1 is that
+  the SAME standard OTEL_* variable now works too, without a Spring-specific property to learn --
+  useful the moment the same env vars are already used to configure something else in the same
+  deployment (the OTel Collector, another language's service, docker-compose.yml).