asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity

Add observability: real OTLP metrics/traces to grafana/otel-lgtm, Docker Compose auto-wiring, and a dual-version (Boot 4.0 vs 4.1) proof of the new OTEL_* env var support

Browse Source 
Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified
against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real
orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero
management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection
detection. Two real findings surfaced along the way and documented rather than smoothed over:
@Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving
alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT
already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1
is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against
identical source compiled on both Boot 4.0.8 and 4.1.1).

Also fixes the root README's module table, which was missing a row for
resilience4j-circuit-breaker (added in a previous commit but never indexed here).
This commit is contained in:
Claude
2026-09-18 09:32:34 +00:00
parent 320733265f
commit 03bdf7ee87
28 changed files with 1027 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
observability/docs/03-otel-env-vars.md
+71
View File
@@ -0,0 +1,71 @@
# 3. OTEL_* environment variables, and the sampling gotcha they surface
Previous: [2. The Observation API and @Observed](02-observation-api.md) | [README](../README.md) | Next: [4. Production checklist](04-production-checklist.md)
## What changed in 4.1, verified against two real Boot versions
Spring Boot's own [4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
state plainly: "Support has been added to read most of the OpenTelemetry environment variables."
That is easy to either overstate (as if no `OTEL_*` variable worked before) or take on faith. The
real, empirical answer, from [`env-var-proof/`](../env-var-proof) -- **identical** Java source
compiled against `spring-boot-starter-parent` 4.0.8 and 4.1.1, run with only standard `OTEL_*`
environment variables set (no `management.otlp.*` Spring properties anywhere) and pointed at a
minimal stand-in OTLP receiver:
| Variable | Boot 4.0.8 | Boot 4.1.1 |
|---|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Honored | Honored |
| `OTEL_METRIC_EXPORT_INTERVAL` | **Ignored** (stays at the 1-minute default) | Honored |
```console
--- Boot 4.1.1, both env vars set ---
Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
... POST /v1/metrics (x11, one every ~2s)
--- Boot 4.0.8, same two env vars ---
Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
... POST /v1/metrics (x1, at JVM shutdown only)
```
([04-otel-env-vars-4.0-vs-4.1.txt](output/04-otel-env-vars-4.0-vs-4.1.txt)) The endpoint variable
already worked on 4.0 -- Micrometer's own `OtlpConfig` has long fallen back to
`OTEL_EXPORTER_OTLP_ENDPOINT` independent of anything Spring-specific. What's actually new in 4.1
is the rest of the standard variable surface -- export interval, protocol, per-signal overrides --
which previously required Boot's own `management.otlp.*` properties (still fully supported; see
the same transcript for `MANAGEMENT_OTLP_METRICS_EXPORT_STEP` working identically on 4.0). The
practical win: the exact same environment variables already used to configure an OTel Collector,
or a sidecar written in another language, now also configure this Spring Boot app, with nothing
Spring-specific to learn.
- Full variable-to-property mapping: [Spring Boot 4.1 Release Notes, OpenTelemetry section](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
- Reproduce this yourself: [`scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh)
## The sampling gotcha this surfaced along the way
Building the env-var proof meant staring at trace counts for a while, which surfaced something
worth a section of its own. `management.tracing.sampling.probability` defaults to **0.10** and
is completely independent of metrics -- metrics are never sampled at all.
```console
15 requests sent to /orders/{101..115}, default sampling (0.10):
orders_placed_total increases by 15 -- every request counted
Tempo trace count increases by only 1 -- roughly 1 in 10 requests actually traced
Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0):
10 requests sent to /orders/{401..410}
Tempo trace count increases by 10 -- all of them, every time
```
([02-low-sampling-demo.txt](output/02-low-sampling-demo.txt), profile:
[`application.yml`](../src/main/resources/application.yml)) The default is the right choice for
production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable
starting point. It is the wrong choice for a five-minute local demo: send a handful of test
requests at the default rate and the honest, common experience is "my trace isn't showing up,"
which reads exactly like a broken pipeline rather than working-as-designed sampling. `fulltrace`
exists in this repo for that reason -- flip it on for local poking, not for anything that sees
real traffic.
- If you enable `spring-boot-starter-opentelemetry` and metrics show up in your backend but
traces don't, check the sampling probability before you suspect the exporter.
Next: [4. Production checklist](04-production-checklist.md)