Add observability: real OTLP metrics/traces to grafana/otel-lgtm, Docker Compose auto-wiring, and a dual-version (Boot 4.0 vs 4.1) proof of the new OTEL_* env var support

Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
2026-09-18 09:32:34 +00:00
parent 320733265f
commit 03bdf7ee87
28 changed files with 1027 additions and 2 deletions
@@ -0,0 +1,79 @@
+# 2. The Observation API and @Observed
+
+Previous: [1. Migrating to spring-boot-starter-opentelemetry](01-migrating-to-opentelemetry-starter.md) | [README](../README.md) | Next: [3. OTEL_* environment variables](03-otel-env-vars.md)
+
+## One recording, two signals -- in theory
+
+[`OrderController.placeOrder`](../src/main/java/com/ankurm/observability/OrderController.java)
+is annotated:
+
+```java
+@Observed(name = "place-order", contextualName = "order-controller#placeOrder")
+@PostMapping("/orders/{id}")
+Map<String, Object> placeOrder(@PathVariable String id) throws InterruptedException {
+```
+
+The idea, per Micrometer's own Observation API and repeated in the
+[deeper guide on this blog](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/#observation-api),
+is that one annotation produces two signals from one recording: a timer (exported as a metric)
+and a trace span, nested under whatever span is already active -- normally the incoming HTTP
+request's own server span.
+
+## What actually happened the first time this was tried
+
+`spring-boot-starter-opentelemetry` was on the classpath. `spring-boot-starter-aspectj` (the
+AspectJ weaver -- see the rename covered in
+[the resilience4j post's chapter 8](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/resilience/docs/08-starter-aop-renamed-to-starter-aspectj.md))
+was added specifically because `@Observed` is AOP-based and needs a proxy to intercept the
+method at all. Micrometer Tracing was active -- the HTTP server span for every request already
+showed up correctly in Tempo. And the `@Observed` child span still never appeared:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
+```
+
+One span. Not two. No error, no warning, no failed startup -- the method-level span is just
+silently absent.
+
+## The missing piece: ObservedAspect is not autoconfigured
+
+Spring Boot's autoconfiguration wires an `ObservationRegistry` bean for you the moment Micrometer
+Tracing is on the classpath. It does **not** also register a Micrometer `ObservedAspect` bean --
+AspectJ weaving being present is necessary but not sufficient, because without the aspect there
+is nothing for the weaver to apply `@Observed` through. The fix is one small
+`@Configuration` class:
+
+```java
+@Configuration(proxyBeanMethods = false)
+class ObservationConfig {
+
+    @Bean
+    ObservedAspect observedAspect(ObservationRegistry registry) {
+        return new ObservedAspect(registry);
+    }
+}
+```
+
+([`ObservationConfig.java`](../src/main/java/com/ankurm/observability/ObservationConfig.java))
+Same annotation, same request, new trace:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
+span: http post /orders/{id}      | kind: SPAN_KIND_SERVER   | parent: (root)
+```
+
+([03-observed-needs-explicit-bean.txt](output/03-observed-needs-explicit-bean.txt)) Two spans,
+correctly nested. This is worth stating plainly because the natural reading of "Boot wires this
+up for you when Micrometer Tracing is active" is that `@Observed` works out of the box -- it does
+not, and the failure mode (silently missing, not broken) is the kind that survives code review.
+
+- If your own `@Observed` spans aren't showing up, check for this bean before anything else --
+  AOP proxying issues (self-invocation, `final` methods/classes) are the usual second suspect,
+  covered generally in
+  [the caching module's self-invocation chapter](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/caching/docs/03-self-invocation-trap.md).
+
+Next: [3. OTEL_* environment variables](03-otel-env-vars.md) -- and a second thing this module's
+own traces caught, unrelated to `@Observed`: most of a short burst of local traffic doesn't
+produce a trace at all, by design.