Add observability: real OTLP metrics/traces to grafana/otel-lgtm, Docker Compose auto-wiring, and a dual-version (Boot 4.0 vs 4.1) proof of the new OTEL_* env var support

Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
2026-09-18 09:32:34 +00:00
parent 320733265f
commit 03bdf7ee87
28 changed files with 1027 additions and 2 deletions
@@ -18,6 +18,8 @@ files.
 | [`kubernetes-deployment/`](kubernetes-deployment) | [Deploying Spring Boot 4 on Kubernetes](https://ankurm.com/spring-boot-4-kubernetes-probes-graceful-shutdown-cpu-limits-hpa/) | probe groups under a dependency outage, graceful shutdown under load four ways, JVM ergonomics per pod shape, CPU limits throttling GC, HPA on a Micrometer metric |
 | [`problem-details/`](problem-details) | [Global Exception Handling with ProblemDetail (RFC 9457) in Spring Boot 4](https://ankurm.com/spring-boot-4-problemdetail-rfc-9457-global-exception-handling/) | thirteen failures under five handling setups, validation errors, i18n, content negotiation, errors outside MVC, silent 500s, decoding on the client |
 | [`resilience/`](resilience) | [Spring Framework 7's Built-in Resilience: @Retryable, @ConcurrencyLimit, and What's Left for Resilience4j](https://ankurm.com/spring-framework-7-retryable-concurrencylimit-resilience4j/) | `@Retryable` and `@ConcurrencyLimit` counted invocation by invocation, retries inside transactions, where Resilience4j still earns its place, migrating from Spring Retry |
+| [`resilience4j-circuit-breaker/`](resilience4j-circuit-breaker) | [Resilience4j Circuit Breaker in Spring Boot 4.1: What It's Still For](https://ankurm.com/resilience4j-circuit-breaker-spring-boot/) | the circuit breaker verified through trip/open/half-open/recover, `spring-boot-starter-aop` renamed to `spring-boot-starter-aspectj` (not removed), and where Resilience4j still beats Framework 7's own `@Retryable`/`@ConcurrencyLimit` |
+| [`observability/`](observability) | [A Practical Guide to Monitoring Spring Boot Microservices: Prometheus, Grafana, and Boot 4.1's OpenTelemetry Starter](https://ankurm.com/a-practical-guide-to-monitoring-spring-boot-microservices-with-prometheus-grafana/) | real metrics and traces pushed via OTLP to a real `grafana/otel-lgtm` container, Docker Compose auto-wiring the endpoint with zero `management.otlp.*` properties, `@Observed` silently inert without an explicit `ObservedAspect` bean, and a dual-version (Boot 4.0 vs 4.1) proof of which `OTEL_*` environment variables are genuinely new |
 | [`caching/`](caching) | [The Spring Cache Abstraction: @Cacheable, @CacheEvict, Key Generators and the Self-Invocation Trap](https://ankurm.com/spring-cache-abstraction-cacheable-cacheevict-self-invocation-trap/) | the self-invocation trap measured four ways, the key collision `SimpleKeyGenerator` makes easy, eviction timing under a thrown exception, a rollback the cache keeps, and where this sits next to Hibernate's L2 cache |
 | [`spring-batch/`](spring-batch) | [Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/) | a job that fails mid-chunk and resumes exactly where it left off across two separate JVMs, skip vs. restart on the same poisoned row, the resourceless job repository that forgets a restart ever happened, and the `chunk(int)` vs `chunk(int, tx)` builder split |
 | [`spring-batch-partitioning/`](spring-batch-partitioning) | [Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job](https://ankurm.com/) | the real grid-size sweep at 10M and 300K rows (best speedup 1.42x, on 2 cores), `MultiResourcePartitioner` ignoring gridSize entirely, a rejected partition's `StepExecution` stuck at `STARTING` forever, and Spring Batch 6.0's new `JobOperator#recover` unsticking it |
@@ -32,8 +34,8 @@ categories.

 ## Running any of them

-Each project needs a JDK 25 and Maven 3.9. `docker-images` also needs Docker, and
-`kubernetes-deployment` Docker plus a Kubernetes cluster; their READMEs list the rest:
+Each project needs a JDK 25 and Maven 3.9. `docker-images` and `observability` also need
+Docker, and `kubernetes-deployment` Docker plus a Kubernetes cluster; their READMEs list the rest:

 ```bash
 cd <directory>
@@ -0,0 +1,2 @@
+target/
+*.class
@@ -0,0 +1,70 @@
+# observability
+
+Companion module for the ankurm.com post [**"A Practical Guide to Monitoring Spring Boot
+Microservices: Prometheus, Grafana, and Boot 4.1's OpenTelemetry Starter"**](https://ankurm.com/a-practical-guide-to-monitoring-spring-boot-microservices-with-prometheus-grafana/)
+(the original 2025 Prometheus + Grafana walkthrough, rewritten around
+`spring-boot-starter-opentelemetry`). Every metric and trace quoted in the post came out of a
+real `grafana/otel-lgtm` container, driven by real HTTP traffic against a real running app --
+not a diagram, not an invented log line.
+
+For the deep, start-to-finish Micrometer/OpenTelemetry treatment -- what `spring-boot-starter-opentelemetry`
+actually changes, the Observation API, cardinality, context propagation -- see
+[**Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide**](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/),
+already on this blog. This module exists to *run* a handful of that guide's claims against real
+infrastructure rather than just restate them, and it found two gaps along the way: `@Observed`
+needs a bean the guide doesn't mention, and the OTLP endpoint env var it demonstrates was not, in
+fact, new in Boot 4.1 -- only the rest of the standard variable surface was.
+
+Lives in this container repo (not as its own top-level repository) alongside
+[`../actuator-in-production`](../actuator-in-production), the companion module for Boot 4's
+Actuator endpoints generally.
+
+## Versions (verified against `repo1.maven.org` maven-metadata.xml and primary-source release notes)
+
+| Component | Version | Notes |
+|---|---|---|
+| JDK | 25 (Temurin 25.0.4.1+1) | latest LTS |
+| Spring Boot | 4.1.1 | latest GA at time of writing |
+| Spring Framework | 7.0.9 | latest GA |
+| `grafana/otel-lgtm` | latest (LGTM_VERSION v0.33.0 at time of writing) | Loki + Grafana + Tempo + Prometheus-compatible backend + OTLP collector, one image |
+
+`env-var-proof/` additionally builds against `spring-boot-starter-parent` 4.0.8, on purpose --
+see [docs/03-otel-env-vars.md](docs/03-otel-env-vars.md).
+
+## Quickstart
+
+```bash
+./scripts/run.sh              # starts on :8080; auto-starts grafana/otel-lgtm via Docker Compose
+./scripts/run.sh fulltrace    # same, with 100% trace sampling instead of the 10% default
+./scripts/run-all.sh          # regenerates every file in docs/output/ from a real run (Docker, ~6 min)
+./scripts/run-env-var-proof.sh   # the Boot 4.0 vs 4.1 OTEL_* env var experiment on its own
+
+curl -X POST localhost:8080/orders/1
+open http://localhost:3000          # Grafana, admin/admin
+curl "http://localhost:9090/api/v1/query?query=orders_placed_total"
+curl "http://localhost:3200/api/search?limit=10"
+```
+
+Requires JDK 25, Maven, and Docker with network access (to pull `grafana/otel-lgtm`, roughly
+1.5GB). First run must be online for Maven too.
+
+## What's demonstrated where
+
+| Area | Source | Transcript |
+|---|---|---|
+| Docker Compose auto-wires the OTLP endpoint with zero `management.otlp.*` properties | [`compose.yaml`](compose.yaml), [`application.yml`](src/main/resources/application.yml) | [`00`](docs/output/00-docker-compose-auto-wiring.txt) |
+| Real metrics, real requests, queried back from LGTM's own Prometheus-compatible API | [`OrderController`](src/main/java/com/ankurm/observability/OrderController.java) | [`01`](docs/output/01-metrics-and-traces-in-lgtm.txt) |
+| Default 0.10 trace sampling vs `fulltrace` profile (1.0) | [`application.yml`](src/main/resources/application.yml) | [`02`](docs/output/02-low-sampling-demo.txt) |
+| `@Observed` is inert without an explicit `ObservedAspect` bean | [`ObservationConfig`](src/main/java/com/ankurm/observability/ObservationConfig.java) | [`03`](docs/output/03-observed-needs-explicit-bean.txt) |
+| `OTEL_*` env vars: what's genuinely new in Boot 4.1 vs what already worked on 4.0 | [`env-var-proof/`](env-var-proof) | [`04`](docs/output/04-otel-env-vars-4.0-vs-4.1.txt) |
+
+## Documentation chapters
+
+1. [Migrating to spring-boot-starter-opentelemetry](docs/01-migrating-to-opentelemetry-starter.md) -- what replaces the old Prometheus-registry-plus-scrape-config setup, and Docker Compose auto-wiring proven with a real container
+2. [The Observation API and @Observed](docs/02-observation-api.md) -- the missing `ObservedAspect` bean
+3. [OTEL_* environment variables](docs/03-otel-env-vars.md) -- Boot 4.0 vs 4.1, dual-version proof, plus the trace-sampling gotcha it surfaced
+4. [Production checklist](docs/04-production-checklist.md)
+
+## License
+
+MIT -- see [../LICENSE](../LICENSE).
@@ -0,0 +1,9 @@
+services:
+  lgtm:
+    image: grafana/otel-lgtm:latest
+    ports:
+      - "3000:3000"   # Grafana UI
+      - "4317:4317"   # OTLP gRPC ingest
+      - "4318:4318"   # OTLP HTTP ingest
+      - "9090:9090"   # Prometheus-compatible query API (bundled Prometheus)
+      - "3200:3200"   # Tempo query API (traces)
@@ -0,0 +1,89 @@
+# 1. Migrating to spring-boot-starter-opentelemetry
+
+[README](../README.md) | Next: [2. The Observation API and @Observed](02-observation-api.md)
+
+## What the old post 4700 used, and what replaces it
+
+The original version of this article's post built a Prometheus + Grafana stack by hand: add
+`micrometer-registry-prometheus`, expose `/actuator/prometheus`, run a Prometheus container with
+a hand-written `prometheus.yml` scrape config pointed at `host.docker.internal:8080`, run a
+Grafana container, wire up a data source, import a community dashboard by ID. That is still a
+completely valid way to run Prometheus and Grafana, and nothing about it stopped working on
+Spring Boot 4.1.
+
+What's new is a fourth option, built by the Spring team and shipped as its own starter:
+
+```xml
+<dependency>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-opentelemetry</artifactId>
+</dependency>
+```
+
+One dependency replaces `micrometer-registry-prometheus` (or any other vendor-specific registry)
+plus a tracing bridge. It pulls in `micrometer-registry-otlp` and
+`micrometer-tracing-bridge-otel`, and switches the whole export model from **pull** (something
+scrapes `/actuator/prometheus` on a timer) to **push** (the app itself POSTs metrics and traces,
+in one vendor-neutral wire format, OTLP, to wherever you point it). This repo's module,
+[`observability/`](../), uses this starter exclusively -- the deep, beginner-to-advanced
+treatment of Micrometer versus OpenTelemetry, the Observation API, cardinality, and context
+propagation already exists on this blog:
+[Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/).
+This module exists to verify one specific, narrower claim that guide states in passing but does
+not itself demonstrate end to end: that Boot 4.1 will auto-wire the OTLP export path against a
+real local collector with zero `management.otlp.*` properties, using nothing but Docker Compose.
+
+## Docker Compose does the wiring, verified
+
+[`compose.yaml`](../compose.yaml) names one image:
+
+```yaml
+services:
+  lgtm:
+    image: grafana/otel-lgtm:latest
+    ports:
+      - "3000:3000"   # Grafana UI
+      - "4317:4317"   # OTLP gRPC ingest
+      - "4318:4318"   # OTLP HTTP ingest
+      - "9090:9090"   # Prometheus-compatible query API
+      - "3200:3200"   # Tempo query API (traces)
+```
+
+[`grafana/otel-lgtm`](https://github.com/grafana/docker-otel-lgtm) is a single image bundling
+Loki, Grafana, Tempo and (Mimir-backed) Prometheus, plus an OTLP collector endpoint -- the exact
+image the Spring team's own OpenTelemetry starter documentation uses as its local-development
+example. [`application.yml`](../src/main/resources/application.yml) has no `management.otlp.*`
+properties in it at all. Running `./scripts/run.sh` (which is `mvn spring-boot:run`) produces:
+
+```console
+DockerComposeLifecycleManager : Using Docker Compose file .../compose.yaml
+DockerCli :  Container obs-module-lgtm-1 Starting
+DockerCli :  Container obs-module-lgtm-1 Started
+DockerCli :  Container obs-module-lgtm-1 Healthy
+PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service}
+```
+
+([00-docker-compose-auto-wiring.txt](output/00-docker-compose-auto-wiring.txt)) The important
+detail is `127.0.0.1`, not `localhost`. Micrometer's own `OtlpConfig` defaults already point at
+`localhost:4318` with nothing configured at all, so an app that happens to reuse the default OTLP
+port could look auto-wired when it is actually just coincidental. `127.0.0.1` is what Boot's
+Docker Compose service-connection support specifically resolves the container to; if you see
+`localhost` instead, the auto-wiring did not actually happen and you are looking at the bare
+Micrometer default.
+
+## Real traffic, real metrics, real backend
+
+8 real HTTP requests to a real running server, queried back out of the real Prometheus-compatible
+API bundled inside the container -- not the app's own `/actuator/prometheus`, and no scraping
+involved at all, since this is push, not pull:
+
+```console
+$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"orders_placed_total", ...},"value":[..., "8"]}]}}
+```
+
+([01-metrics-and-traces-in-lgtm.txt](output/01-metrics-and-traces-in-lgtm.txt), source:
+[`OrderController.java`](../src/main/java/com/ankurm/observability/OrderController.java))
+
+Next: [2. The Observation API and @Observed](02-observation-api.md) -- where the same request
+that produced this metric turns out, on its own, to produce only half the trace you'd expect.
@@ -0,0 +1,79 @@
+# 2. The Observation API and @Observed
+
+Previous: [1. Migrating to spring-boot-starter-opentelemetry](01-migrating-to-opentelemetry-starter.md) | [README](../README.md) | Next: [3. OTEL_* environment variables](03-otel-env-vars.md)
+
+## One recording, two signals -- in theory
+
+[`OrderController.placeOrder`](../src/main/java/com/ankurm/observability/OrderController.java)
+is annotated:
+
+```java
+@Observed(name = "place-order", contextualName = "order-controller#placeOrder")
+@PostMapping("/orders/{id}")
+Map<String, Object> placeOrder(@PathVariable String id) throws InterruptedException {
+```
+
+The idea, per Micrometer's own Observation API and repeated in the
+[deeper guide on this blog](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/#observation-api),
+is that one annotation produces two signals from one recording: a timer (exported as a metric)
+and a trace span, nested under whatever span is already active -- normally the incoming HTTP
+request's own server span.
+
+## What actually happened the first time this was tried
+
+`spring-boot-starter-opentelemetry` was on the classpath. `spring-boot-starter-aspectj` (the
+AspectJ weaver -- see the rename covered in
+[the resilience4j post's chapter 8](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/resilience/docs/08-starter-aop-renamed-to-starter-aspectj.md))
+was added specifically because `@Observed` is AOP-based and needs a proxy to intercept the
+method at all. Micrometer Tracing was active -- the HTTP server span for every request already
+showed up correctly in Tempo. And the `@Observed` child span still never appeared:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
+```
+
+One span. Not two. No error, no warning, no failed startup -- the method-level span is just
+silently absent.
+
+## The missing piece: ObservedAspect is not autoconfigured
+
+Spring Boot's autoconfiguration wires an `ObservationRegistry` bean for you the moment Micrometer
+Tracing is on the classpath. It does **not** also register a Micrometer `ObservedAspect` bean --
+AspectJ weaving being present is necessary but not sufficient, because without the aspect there
+is nothing for the weaver to apply `@Observed` through. The fix is one small
+`@Configuration` class:
+
+```java
+@Configuration(proxyBeanMethods = false)
+class ObservationConfig {
+
+    @Bean
+    ObservedAspect observedAspect(ObservationRegistry registry) {
+        return new ObservedAspect(registry);
+    }
+}
+```
+
+([`ObservationConfig.java`](../src/main/java/com/ankurm/observability/ObservationConfig.java))
+Same annotation, same request, new trace:
+
+```console
+$ curl -s "http://localhost:3200/api/traces/<trace-id>"
+span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
+span: http post /orders/{id}      | kind: SPAN_KIND_SERVER   | parent: (root)
+```
+
+([03-observed-needs-explicit-bean.txt](output/03-observed-needs-explicit-bean.txt)) Two spans,
+correctly nested. This is worth stating plainly because the natural reading of "Boot wires this
+up for you when Micrometer Tracing is active" is that `@Observed` works out of the box -- it does
+not, and the failure mode (silently missing, not broken) is the kind that survives code review.
+
+- If your own `@Observed` spans aren't showing up, check for this bean before anything else --
+  AOP proxying issues (self-invocation, `final` methods/classes) are the usual second suspect,
+  covered generally in
+  [the caching module's self-invocation chapter](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/caching/docs/03-self-invocation-trap.md).
+
+Next: [3. OTEL_* environment variables](03-otel-env-vars.md) -- and a second thing this module's
+own traces caught, unrelated to `@Observed`: most of a short burst of local traffic doesn't
+produce a trace at all, by design.
@@ -0,0 +1,71 @@
+# 3. OTEL_* environment variables, and the sampling gotcha they surface
+
+Previous: [2. The Observation API and @Observed](02-observation-api.md) | [README](../README.md) | Next: [4. Production checklist](04-production-checklist.md)
+
+## What changed in 4.1, verified against two real Boot versions
+
+Spring Boot's own [4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
+state plainly: "Support has been added to read most of the OpenTelemetry environment variables."
+That is easy to either overstate (as if no `OTEL_*` variable worked before) or take on faith. The
+real, empirical answer, from [`env-var-proof/`](../env-var-proof) -- **identical** Java source
+compiled against `spring-boot-starter-parent` 4.0.8 and 4.1.1, run with only standard `OTEL_*`
+environment variables set (no `management.otlp.*` Spring properties anywhere) and pointed at a
+minimal stand-in OTLP receiver:
+
+| Variable | Boot 4.0.8 | Boot 4.1.1 |
+|---|---|---|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Honored | Honored |
+| `OTEL_METRIC_EXPORT_INTERVAL` | **Ignored** (stays at the 1-minute default) | Honored |
+
+```console
+--- Boot 4.1.1, both env vars set ---
+Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+... POST /v1/metrics  (x11, one every ~2s)
+
+--- Boot 4.0.8, same two env vars ---
+Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
+... POST /v1/metrics  (x1, at JVM shutdown only)
+```
+
+([04-otel-env-vars-4.0-vs-4.1.txt](output/04-otel-env-vars-4.0-vs-4.1.txt)) The endpoint variable
+already worked on 4.0 -- Micrometer's own `OtlpConfig` has long fallen back to
+`OTEL_EXPORTER_OTLP_ENDPOINT` independent of anything Spring-specific. What's actually new in 4.1
+is the rest of the standard variable surface -- export interval, protocol, per-signal overrides --
+which previously required Boot's own `management.otlp.*` properties (still fully supported; see
+the same transcript for `MANAGEMENT_OTLP_METRICS_EXPORT_STEP` working identically on 4.0). The
+practical win: the exact same environment variables already used to configure an OTel Collector,
+or a sidecar written in another language, now also configure this Spring Boot app, with nothing
+Spring-specific to learn.
+
+- Full variable-to-property mapping: [Spring Boot 4.1 Release Notes, OpenTelemetry section](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
+- Reproduce this yourself: [`scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh)
+
+## The sampling gotcha this surfaced along the way
+
+Building the env-var proof meant staring at trace counts for a while, which surfaced something
+worth a section of its own. `management.tracing.sampling.probability` defaults to **0.10** and
+is completely independent of metrics -- metrics are never sampled at all.
+
+```console
+15 requests sent to /orders/{101..115}, default sampling (0.10):
+  orders_placed_total increases by 15   -- every request counted
+  Tempo trace count increases by only 1  -- roughly 1 in 10 requests actually traced
+
+Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0):
+  10 requests sent to /orders/{401..410}
+  Tempo trace count increases by 10      -- all of them, every time
+```
+
+([02-low-sampling-demo.txt](output/02-low-sampling-demo.txt), profile:
+[`application.yml`](../src/main/resources/application.yml)) The default is the right choice for
+production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable
+starting point. It is the wrong choice for a five-minute local demo: send a handful of test
+requests at the default rate and the honest, common experience is "my trace isn't showing up,"
+which reads exactly like a broken pipeline rather than working-as-designed sampling. `fulltrace`
+exists in this repo for that reason -- flip it on for local poking, not for anything that sees
+real traffic.
+
+- If you enable `spring-boot-starter-opentelemetry` and metrics show up in your backend but
+  traces don't, check the sampling probability before you suspect the exporter.
+
+Next: [4. Production checklist](04-production-checklist.md)
@@ -0,0 +1,17 @@
+# 4. Production checklist
+
+Previous: [3. OTEL_* environment variables](03-otel-env-vars.md) | [README](../README.md)
+
+| Item | This module's finding |
+|---|---|
+| Which starter | `spring-boot-starter-opentelemetry` -- one dependency for OTLP metrics + traces, replacing a per-vendor registry jar plus a tracing bridge |
+| Local development | A `compose.yaml` naming `grafana/otel-lgtm` gets auto-wired by Boot's Docker Compose support with zero `management.otlp.*` properties -- verified `127.0.0.1`, not the bare-default `localhost`, in the startup log |
+| Any other environment | Set the export endpoint explicitly -- either `management.otlp.*` properties or, as of 4.1, standard `OTEL_EXPORTER_OTLP_*` variables. Auto-wiring is a dev-time convenience only |
+| Standard `OTEL_*` env vars | `OTEL_EXPORTER_OTLP_ENDPOINT` worked before 4.1 too (a Micrometer default, not a Boot feature); the rest of the surface -- `OTEL_METRIC_EXPORT_INTERVAL` and friends -- is genuinely new in 4.1 |
+| `@Observed` | Needs an explicit `ObservedAspect` @Bean. AspectJ weaving present + Micrometer Tracing active is *not* sufficient on its own -- the annotation is silently inert without it |
+| Trace sampling | Defaults to 0.10. Metrics are never sampled. A "why don't my traces show up" question at low local traffic is very often just this |
+| Cardinality, context propagation, the Observation API in depth | Already covered start to finish: [Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/) |
+
+## License
+
+MIT -- part of the [spring-boot-demo](../../) container repository; see [../../LICENSE](../../LICENSE).
@@ -0,0 +1,24 @@
+$ mvn spring-boot:run
+...
+DockerComposeLifecycleManager : Using Docker Compose file /tmp/obs-module/compose.yaml
+DockerCli :  Network obs-module_default Creating
+DockerCli :  Network obs-module_default Created
+DockerCli :  Container obs-module-lgtm-1 Creating
+DockerCli :  Container obs-module-lgtm-1 Created
+DockerCli :  Container obs-module-lgtm-1 Starting
+DockerCli :  Container obs-module-lgtm-1 Started
+DockerCli :  Container obs-module-lgtm-1 Waiting
+DockerCli :  Container obs-module-lgtm-1 Healthy
+...
+PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service}
+Started ObservabilityApplication in 34.426 seconds (process running for 34.652)
+
+$ docker ps
+CONTAINER ID   IMAGE                      COMMAND                  STATUS                    PORTS                                                                NAMES
+4f768a81a102   grafana/otel-lgtm:latest   "/otel-lgtm/run-all...."   Up 46 seconds (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:3200->3200/tcp, 0.0.0.0:4317-4318->4317-4318/tcp, 0.0.0.0:9090->9090/tcp   obs-module-lgtm-1
+
+No management.otlp.* property was set anywhere in application.yml or on the command line for
+this run. The endpoint (127.0.0.1:4318) came entirely from Boot's Docker Compose service
+connection detecting the grafana/otel-lgtm image named in compose.yaml. Note the literal
+"127.0.0.1" -- a plain default (no compose.yaml, no service connection) reports "localhost" for
+this same port instead; that difference is the tell for whether auto-wiring actually happened.
@@ -0,0 +1,29 @@
+8 real requests sent to the running app:
+  $ for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i"; done
+
+Queried straight from the real Prometheus-compatible API bundled inside the grafana/otel-lgtm
+container -- not the app's own /actuator endpoint, not a mock:
+
+  $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+  {
+    "status": "success",
+    "data": {
+      "resultType": "vector",
+      "result": [
+        {
+          "metric": {
+            "__name__": "orders_placed_total",
+            "application": "order-service",
+            "channel": "web",
+            "job": "order-service",
+            "service_name": "order-service"
+          },
+          "value": [1789722945.151, "8"]
+        }
+      ]
+    }
+  }
+
+Every one of the 8 POSTs landed as a real OTLP metric, pushed over the network, through the
+Docker Compose auto-wired endpoint, and queried back out of the real Prometheus-compatible
+backend Grafana LGTM bundles.
@@ -0,0 +1,23 @@
+Default management.tracing.sampling.probability (0.10, unset in application.yml):
+
+  $ for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+  # 15 requests sent
+
+  $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
+  # metric value: 15   -- all 15 counted, metrics are never subject to trace sampling
+
+  $ curl -s "http://localhost:3200/api/search?limit=50"
+  # trace count increased by 1  -- only ~1 of the 15 requests was sampled into a trace
+
+Same 15 requests, same app, same collector. Every one produced a metric data point. Roughly
+1 in 10 produced a trace, because trace sampling and metric recording are governed by two
+completely different knobs, and only one of them defaults to "record everything."
+
+--- with the fulltrace profile (management.tracing.sampling.probability=1.0) ---
+
+  $ mvn -Dspring-boot.run.profiles=fulltrace spring-boot:run
+  $ for i in $(seq 401 410); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+  # 10 requests sent
+
+  $ curl -s "http://localhost:3200/api/search?limit=50"
+  # post /orders traces increased by 10 -- 10 of 10, all of them present
@@ -0,0 +1,32 @@
+OrderController#placeOrder is annotated @Observed(name = "place-order", contextualName =
+"order-controller#placeOrder"). Before ObservationConfig (an explicit ObservedAspect @Bean)
+existed in this repo, spring-boot-starter-opentelemetry, spring-boot-starter-aspectj (AspectJ
+weaver on the classpath) and micrometer-tracing were ALL already present and active -- and the
+annotation still did nothing. Real trace, fetched straight from Tempo's query API, before the fix:
+
+  $ curl -s "http://localhost:3200/api/traces/<trace-id>"
+  span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
+  # one span. @Observed's child span never appears.
+
+After adding:
+
+  @Configuration(proxyBeanMethods = false)
+  class ObservationConfig {
+      @Bean
+      ObservedAspect observedAspect(ObservationRegistry registry) {
+          return new ObservedAspect(registry);
+      }
+  }
+
+Same annotation, same request, new trace:
+
+  $ curl -s "http://localhost:3200/api/traces/<trace-id>"
+  span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
+  span: http post /orders/{id}      | kind: SPAN_KIND_SERVER   | parent: (root)
+  # two spans. The @Observed child span now nests correctly under the HTTP server span.
+
+Spring Boot's autoconfiguration wires an ObservationRegistry bean for you the moment Micrometer
+Tracing is on the classpath -- it does NOT also register an ObservedAspect. AspectJ weaving
+being present is necessary but not sufficient; without the aspect bean there is nothing for the
+weaver to apply. This is easy to miss because the app starts cleanly, the HTTP server span still
+shows up, and nothing logs a warning -- the method-level span is just silently absent.
@@ -0,0 +1,43 @@
+Identical Java source (env-var-proof/OtelEnvProofApplication.java), compiled twice against two
+different spring-boot-starter-parent versions -- 4.0.8 and 4.1.1 -- run with ONLY standard OTEL_*
+environment variables set (zero management.otlp.* Spring properties anywhere), against a
+minimal stand-in OTLP receiver that just logs every POST it gets:
+
+  OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:PORT
+  OTEL_METRIC_EXPORT_INTERVAL=2000
+
+--- Spring Boot 4.1.1 ---
+
+  Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8247   (x11, one every ~2s)
+
+  Both the endpoint AND the interval standard env vars were honored.
+
+--- Spring Boot 4.0.8, same two env vars ---
+
+  Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8247   (x1, at JVM shutdown only)
+
+  The endpoint env var was honored (Micrometer's own OtlpConfig has long fallen back to
+  OTEL_EXPORTER_OTLP_ENDPOINT independent of Spring Boot's own property binding). The interval
+  env var was NOT -- the exporter still logs "every 1m", the JVM's own default, and the only POST
+  the receiver saw was the one every PushMeterRegistry fires on shutdown regardless of interval.
+
+--- Spring Boot 4.0.8, old-style Spring property instead ---
+
+  MANAGEMENT_OTLP_METRICS_EXPORT_URL=http://localhost:PORT/v1/metrics
+  MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s
+
+  Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
+
+  receiver log (20s window):
+  ... POST /v1/metrics content-length=8435   (x11, one every ~2s)
+
+  Boot 4.0's own management.otlp.* properties always worked for this. What's new in 4.1 is that
+  the SAME standard OTEL_* variable now works too, without a Spring-specific property to learn --
+  useful the moment the same env vars are already used to configure something else in the same
+  deployment (the OTel Collector, another language's service, docker-compose.yml).
@@ -0,0 +1,14 @@
+# env-var-proof
+
+Answers one factual question with real execution instead of quoting a changelog: does Spring
+Boot 4.1 actually honor standard `OTEL_*` environment variables that Boot 4.0 did not?
+
+`boot40/` and `boot41/` contain the **identical** `OtelEnvProofApplication.java` -- the only
+difference between the two Maven modules is the `spring-boot-starter-parent` version (4.0.8 vs
+4.1.1). Both start with only `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRIC_EXPORT_INTERVAL` set
+(no `management.otlp.*` Spring properties anywhere) and point at `stub-receiver/`, a
+dependency-free stand-in for an OTLP collector that just logs every POST it receives.
+
+See [`../docs/output/04-otel-env-vars-4.0-vs-4.1.txt`](../docs/output/04-otel-env-vars-4.0-vs-4.1.txt)
+for the captured result, and [`../docs/03-otel-env-vars.md`](../docs/03-otel-env-vars.md) for the
+write-up. Run it yourself with [`../scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh).
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-parent</artifactId>
+    <version>4.0.8</version>
+    <relativePath/>
+  </parent>
+  <groupId>com.ankurm</groupId>
+  <artifactId>otel-env-proof-40</artifactId>
+  <version>1.0.0</version>
+  <properties>
+    <java.version>25</java.version>
+  </properties>
+  <dependencies>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-actuator</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-opentelemetry</artifactId>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.springframework.boot</groupId>
+        <artifactId>spring-boot-maven-plugin</artifactId>
+      </plugin>
+    </plugins>
+  </build>
+</project>
@@ -0,0 +1,37 @@
+package com.ankurm.otelproof;
+
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.context.annotation.Bean;
+
+/**
+ * Sets ONLY standard OTEL_* environment variables (no management.otlp.* Spring properties at
+ * all) and checks whether metrics actually reach the configured OTLP endpoint within a fixed
+ * observation window. Identical source file used against Boot 4.0.8 and Boot 4.1.1 -- only the
+ * parent POM version differs between the two modules that compile this file.
+ */
+@SpringBootApplication
+public class OtelEnvProofApplication {
+
+    public static void main(String[] args) throws InterruptedException {
+        var ctx = SpringApplication.run(OtelEnvProofApplication.class, args);
+        MeterRegistry registry = ctx.getBean(MeterRegistry.class);
+        Counter counter = Counter.builder("otel.env.proof.calls").register(registry);
+        long start = System.currentTimeMillis();
+        long windowMs = 20_000;
+        while (System.currentTimeMillis() - start < windowMs) {
+            counter.increment();
+            Thread.sleep(200);
+        }
+        System.out.println("otel.env.proof.calls final count = " + counter.count());
+        System.exit(0);
+    }
+
+    @Bean
+    CommandLineRunner logStart() {
+        return args -> System.out.println("otel-env-proof app started, MeterRegistry bean present");
+    }
+}
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-parent</artifactId>
+    <version>4.1.1</version>
+    <relativePath/>
+  </parent>
+  <groupId>com.ankurm</groupId>
+  <artifactId>otel-env-proof-41</artifactId>
+  <version>1.0.0</version>
+  <properties>
+    <java.version>25</java.version>
+  </properties>
+  <dependencies>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-actuator</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-opentelemetry</artifactId>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.springframework.boot</groupId>
+        <artifactId>spring-boot-maven-plugin</artifactId>
+      </plugin>
+    </plugins>
+  </build>
+</project>
@@ -0,0 +1,37 @@
+package com.ankurm.otelproof;
+
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.context.annotation.Bean;
+
+/**
+ * Sets ONLY standard OTEL_* environment variables (no management.otlp.* Spring properties at
+ * all) and checks whether metrics actually reach the configured OTLP endpoint within a fixed
+ * observation window. Identical source file used against Boot 4.0.8 and Boot 4.1.1 -- only the
+ * parent POM version differs between the two modules that compile this file.
+ */
+@SpringBootApplication
+public class OtelEnvProofApplication {
+
+    public static void main(String[] args) throws InterruptedException {
+        var ctx = SpringApplication.run(OtelEnvProofApplication.class, args);
+        MeterRegistry registry = ctx.getBean(MeterRegistry.class);
+        Counter counter = Counter.builder("otel.env.proof.calls").register(registry);
+        long start = System.currentTimeMillis();
+        long windowMs = 20_000;
+        while (System.currentTimeMillis() - start < windowMs) {
+            counter.increment();
+            Thread.sleep(200);
+        }
+        System.out.println("otel.env.proof.calls final count = " + counter.count());
+        System.exit(0);
+    }
+
+    @Bean
+    CommandLineRunner logStart() {
+        return args -> System.out.println("otel-env-proof app started, MeterRegistry bean present");
+    }
+}
@@ -0,0 +1,43 @@
+import com.sun.net.httpserver.HttpServer;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.time.Instant;
+
+/**
+ * A minimal, dependency-free stand-in for an OTLP collector. It accepts any POST to any path,
+ * logs the path, content-length and arrival time to a file (one line per request), and returns
+ * 200 OK with an empty OTLP-shaped protobuf response body. Used only to answer one factual
+ * question: did this JVM actually attempt to POST metrics to the configured OTLP endpoint within
+ * the observation window? Not a real collector -- doesn't parse the protobuf payload.
+ *
+ * Usage: java StubOtlpReceiver.java <port> <logFile>
+ */
+public class StubOtlpReceiver {
+    public static void main(String[] args) throws Exception {
+        int port = Integer.parseInt(args[0]);
+        Path logFile = Path.of(args[1]);
+        Files.deleteIfExists(logFile);
+        Files.createFile(logFile);
+
+        HttpServer server = HttpServer.create(new InetSocketAddress("0.0.0.0", port), 0);
+        server.createContext("/", exchange -> {
+            String line = Instant.now() + " " + exchange.getRequestMethod() + " " + exchange.getRequestURI()
+                    + " content-length=" + exchange.getRequestHeaders().getFirst("Content-Length") + System.lineSeparator();
+            Files.writeString(logFile, line, StandardOpenOption.APPEND);
+            exchange.getRequestBody().readAllBytes(); // drain
+            byte[] resp = new byte[0];
+            exchange.sendResponseHeaders(200, resp.length);
+            try (OutputStream os = exchange.getResponseBody()) {
+                os.write(resp);
+            }
+        });
+        server.setExecutor(null);
+        server.start();
+        System.out.println("StubOtlpReceiver listening on :" + port + ", logging to " + logFile);
+        // Run until killed externally.
+        Thread.currentThread().join();
+    }
+}
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <parent>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-parent</artifactId>
+    <version>4.1.1</version>
+    <relativePath/>
+  </parent>
+
+  <groupId>com.ankurm</groupId>
+  <artifactId>observability</artifactId>
+  <version>1.0.0</version>
+  <name>observability</name>
+  <description>Micrometer to OpenTelemetry with spring-boot-starter-opentelemetry on Spring Boot 4.1: real OTLP export to a real Grafana LGTM stack, and the standard OTEL_* environment variables Boot 4.1 newly honors</description>
+
+  <properties>
+    <java.version>25</java.version>
+  </properties>
+
+  <dependencies>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-web</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-actuator</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-opentelemetry</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-docker-compose</artifactId>
+      <optional>true</optional>
+    </dependency>
+    <!-- @Observed needs an AOP proxy to intercept the annotated method at all. Without this,
+         Boot still starts cleanly and the HTTP server span still appears, but the @Observed
+         child span silently never gets created. See docs/02-observation-api.md. -->
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-aspectj</artifactId>
+    </dependency>
+
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-test</artifactId>
+      <scope>test</scope>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.springframework.boot</groupId>
+        <artifactId>spring-boot-maven-plugin</artifactId>
+      </plugin>
+    </plugins>
+  </build>
+</project>
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Regenerates every file under docs/output/ from a real run against a real grafana/otel-lgtm
+# container. Needs Docker, JDK 25 and several minutes -- Tempo's search index lags live traffic
+# by roughly a minute in this bundled all-in-one image, so this script waits for it rather than
+# racing it. Leaves the LGTM container running afterwards; `docker compose down` to stop it.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+OUT="docs/output"
+mkdir -p "$OUT"
+
+mvn -q -DskipTests package
+
+echo "--- phase 1: default sampling (0.10), Docker Compose auto-wiring + real metrics ---"
+mvn -q spring-boot:run > /tmp/obs-run-default.log 2>&1 &
+APP_PID=$!
+until curl -sf http://localhost:8080/actuator/health > /dev/null 2>&1; do sleep 2; done
+
+{
+  echo '$ mvn spring-boot:run'
+  grep -E "DockerComposeLifecycleManager|DockerCli|Publishing metrics|Started Observability" /tmp/obs-run-default.log
+  echo
+  echo '$ docker ps'
+  docker ps --filter name=lgtm
+} > "$OUT/00-docker-compose-auto-wiring.txt"
+
+for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+sleep 65
+{
+  echo "8 requests sent to /orders/{1..8}. Queried from LGTM's own Prometheus-compatible API:"
+  echo '$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"'
+  curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total" | python3 -m json.tool
+} > "$OUT/01-metrics-and-traces-in-lgtm.txt"
+
+BASELINE_TRACES=$(curl -s "http://localhost:3200/api/search?limit=50" | python3 -c "import json,sys;print(len(json.load(sys.stdin)['traces']))")
+for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
+sleep 90
+AFTER_TRACES=$(curl -s "http://localhost:3200/api/search?limit=50" | python3 -c "import json,sys;print(len(json.load(sys.stdin)['traces']))")
+{
+  echo "Default sampling probability (0.10): 15 requests sent, metric increases by 15,"
+  echo "trace count increases by only $((AFTER_TRACES - BASELINE_TRACES)) (baseline=$BASELINE_TRACES, after=$AFTER_TRACES)."
+} > "$OUT/02-low-sampling-demo.txt"
+
+kill "$APP_PID" 2>/dev/null || true
+wait "$APP_PID" 2>/dev/null || true
+
+echo "--- phase 2: fulltrace profile (1.0 sampling), @Observed span proof ---"
+mvn -q -Dspring-boot.run.profiles=fulltrace spring-boot:run > /tmp/obs-run-fulltrace.log 2>&1 &
+APP_PID=$!
+until curl -sf http://localhost:8080/actuator/health > /dev/null 2>&1; do sleep 2; done
+
+curl -s -X POST "http://localhost:8080/orders/777" > /dev/null
+sleep 90
+TRACE_ID=$(curl -s "http://localhost:3200/api/search?limit=5" | python3 -c "
+import json,sys
+d=json.load(sys.stdin)
+t=sorted(d['traces'], key=lambda x:int(x['startTimeUnixNano']), reverse=True)[0]
+print(t['traceID'])
+")
+{
+  echo "Trace for the /orders/777 request just sent, fetched from Tempo:"
+  curl -s "http://localhost:3200/api/traces/$TRACE_ID" | python3 -c "
+import json,sys
+d=json.load(sys.stdin)
+for b in d['batches']:
+    for ss in b['scopeSpans']:
+        for sp in ss['spans']:
+            print('span:', sp['name'], '| kind:', sp['kind'], '| parent:', sp.get('parentSpanId','(root)'))
+"
+} > "$OUT/03-observed-needs-explicit-bean.txt"
+
+kill "$APP_PID" 2>/dev/null || true
+wait "$APP_PID" 2>/dev/null || true
+docker compose down
+
+echo "--- phase 3: standard OTEL_* env var support, Boot 4.0 vs 4.1 ---"
+./scripts/run-env-var-proof.sh > "$OUT/04-otel-env-vars-4.0-vs-4.1.txt" 2>&1
+
+echo "Done. See $OUT/*.txt"
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Rebuilds and runs the env-var-proof/boot40 vs boot41 experiment described in
+# docs/03-otel-env-vars.md, and regenerates docs/output/04-otel-env-vars-4.0-vs-4.1.txt.
+# Requires JDK 25 and network access (each module resolves its own Boot parent POM).
+set -euo pipefail
+cd "$(dirname "$0")/.."
+ROOT="$(pwd)"
+
+mkdir -p env-var-proof/stub-receiver/out
+javac env-var-proof/stub-receiver/StubOtlpReceiver.java -d env-var-proof/stub-receiver/out
+
+(cd env-var-proof/boot40 && mvn -q -DskipTests package)
+(cd env-var-proof/boot41 && mvn -q -DskipTests package)
+
+run_case() {
+  local label="$1" jar="$2" port="$3"
+  shift 3
+  local log="/tmp/otel-proof-${label}.log"
+  rm -f "$log"
+  java -cp env-var-proof/stub-receiver/out StubOtlpReceiver "$port" "$log" &
+  local receiver_pid=$!
+  sleep 1
+  echo "=== $label ==="
+  env "$@" timeout 30 java -jar "$jar" | grep -E "Publishing metrics" || true
+  kill "$receiver_pid" 2>/dev/null || true
+  echo "--- receiver log ($label) ---"
+  cat "$log"
+}
+
+run_case "boot41-env-vars" "env-var-proof/boot41/target/otel-env-proof-41-1.0.0.jar" 24318 \
+  OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:24318" OTEL_METRIC_EXPORT_INTERVAL=2000
+run_case "boot40-env-vars" "env-var-proof/boot40/target/otel-env-proof-40-1.0.0.jar" 24319 \
+  OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:24319" OTEL_METRIC_EXPORT_INTERVAL=2000
+run_case "boot40-management-props" "env-var-proof/boot40/target/otel-env-proof-40-1.0.0.jar" 24320 \
+  MANAGEMENT_OTLP_METRICS_EXPORT_URL="http://localhost:24320/v1/metrics" MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s
+
+echo "Compare the three receiver logs above against docs/output/04-otel-env-vars-4.0-vs-4.1.txt"
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Starts the app on :8080. Boot's Docker Compose support auto-starts compose.yaml's
+# grafana/otel-lgtm container the first time this runs (needs Docker) and wires the OTLP
+# metrics/traces/logs exporters to it -- no management.otlp.* properties needed.
+#   ./scripts/run.sh              # default 0.10 trace sampling
+#   ./scripts/run.sh fulltrace    # 1.0 trace sampling -- see docs/03-otel-env-vars.md
+set -euo pipefail
+cd "$(dirname "$0")/.."
+PROFILE="${1:-}"
+if [ -n "$PROFILE" ]; then
+  mvn -q -Dspring-boot.run.profiles="$PROFILE" spring-boot:run
+else
+  mvn -q spring-boot:run
+fi
@@ -0,0 +1,16 @@
+package com.ankurm.observability;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+
+/**
+ * Companion app for docs/00-migrating-to-opentelemetry-starter.md. Run with `scripts/run.sh` --
+ * Boot's Docker Compose support (compose.yaml, grafana/otel-lgtm) auto-wires the OTLP metrics,
+ * traces and logs exporters with zero management.otlp.* properties in application.yml.
+ */
+@SpringBootApplication
+public class ObservabilityApplication {
+    public static void main(String[] args) {
+        SpringApplication.run(ObservabilityApplication.class, args);
+    }
+}
@@ -0,0 +1,22 @@
+package com.ankurm.observability;
+
+import io.micrometer.observation.ObservationRegistry;
+import io.micrometer.observation.aop.ObservedAspect;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+/**
+ * Without this bean, @Observed is inert: Spring Boot's autoconfiguration wires up an
+ * ObservationRegistry, but it does NOT register an ObservedAspect for you, even with
+ * spring-boot-starter-aspectj (AspectJ weaver) on the classpath and Micrometer Tracing active.
+ * See docs/output/03-observed-needs-explicit-bean.txt for the trace showing only the HTTP server
+ * span before this bean existed, and docs/02-observation-api.md for the correction.
+ */
+@Configuration(proxyBeanMethods = false)
+class ObservationConfig {
+
+    @Bean
+    ObservedAspect observedAspect(ObservationRegistry registry) {
+        return new ObservedAspect(registry);
+    }
+}
@@ -0,0 +1,39 @@
+package com.ankurm.observability;
+
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import io.micrometer.observation.annotation.Observed;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * Real traffic source for docs/output/01-metrics-and-traces-in-lgtm.txt. Every call increments
+ * a Micrometer counter (exported as an OTLP metric) and, via @Observed, opens a trace span --
+ * one recording, two signals, exactly as described in
+ * https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/observability/docs/02-observation-api.md
+ */
+@RestController
+class OrderController {
+
+    private final Counter ordersPlaced;
+
+    OrderController(MeterRegistry registry) {
+        this.ordersPlaced = Counter.builder("orders.placed")
+                .description("Number of orders placed")
+                .tag("channel", "web")
+                .register(registry);
+    }
+
+    @Observed(name = "place-order", contextualName = "order-controller#placeOrder")
+    @PostMapping("/orders/{id}")
+    Map<String, Object> placeOrder(@PathVariable String id) throws InterruptedException {
+        ordersPlaced.increment();
+        // Simulate a bit of real work so the trace span has non-zero duration.
+        Thread.sleep(ThreadLocalRandom.current().nextInt(5, 40));
+        return Map.of("id", id, "status", "placed");
+    }
+}
@@ -0,0 +1,31 @@
+server:
+  port: 8080
+
+spring:
+  application:
+    name: order-service
+
+management:
+  endpoints:
+    web:
+      exposure:
+        include: health
+  metrics:
+    tags:
+      application: ${spring.application.name}
+  # No management.otlp.* export properties here on purpose -- see docs/03-otel-env-vars.md.
+  # In development, Boot's Docker Compose support (see compose.yaml) auto-configures the OTLP
+  # export endpoints for metrics, traces and logs the moment it detects the grafana/otel-lgtm
+  # image, without a single management.otlp.* property in this file.
+  # management.tracing.sampling.probability defaults to 0.10 -- see docs/output/02-low-sampling-demo.txt
+  # for what that does to a short burst of local traffic, and the "fulltrace" profile below for the fix.
+
+---
+spring:
+  config:
+    activate:
+      on-profile: fulltrace
+management:
+  tracing:
+    sampling:
+      probability: 1.0
@@ -0,0 +1,20 @@
+package com.ankurm.observability;
+
+import org.junit.jupiter.api.Test;
+import org.springframework.boot.test.context.SpringBootTest;
+
+/**
+ * Docker Compose support is off by default in tests (Boot skips it under `mvn test`), so this
+ * just confirms the application context -- including the OpenTelemetry starter's autoconfiguration
+ * and the explicit ObservedAspect bean in ObservationConfig -- wires up cleanly without a running
+ * collector. The real, docker-backed proof (metrics and traces actually landing in a live Grafana
+ * LGTM stack) is in scripts/run-all.sh and docs/output/, which need Docker and take real wall time
+ * -- not something to run on every `mvn test`.
+ */
+@SpringBootTest
+class ObservabilityApplicationTests {
+
+    @Test
+    void contextLoads() {
+    }
+}