Claude
03bdf7ee87
Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
4.1 KiB
3. OTEL_* environment variables, and the sampling gotcha they surface
Previous: 2. The Observation API and @Observed | README | Next: 4. Production checklist
What changed in 4.1, verified against two real Boot versions
Spring Boot's own 4.1 release notes
state plainly: "Support has been added to read most of the OpenTelemetry environment variables."
That is easy to either overstate (as if no OTEL_* variable worked before) or take on faith. The
real, empirical answer, from env-var-proof/ -- identical Java source
compiled against spring-boot-starter-parent 4.0.8 and 4.1.1, run with only standard OTEL_*
environment variables set (no management.otlp.* Spring properties anywhere) and pointed at a
minimal stand-in OTLP receiver:
| Variable | Boot 4.0.8 | Boot 4.1.1 |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
Honored | Honored |
OTEL_METRIC_EXPORT_INTERVAL |
Ignored (stays at the 1-minute default) | Honored |
--- Boot 4.1.1, both env vars set ---
Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
... POST /v1/metrics (x11, one every ~2s)
--- Boot 4.0.8, same two env vars ---
Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
... POST /v1/metrics (x1, at JVM shutdown only)
(04-otel-env-vars-4.0-vs-4.1.txt) The endpoint variable
already worked on 4.0 -- Micrometer's own OtlpConfig has long fallen back to
OTEL_EXPORTER_OTLP_ENDPOINT independent of anything Spring-specific. What's actually new in 4.1
is the rest of the standard variable surface -- export interval, protocol, per-signal overrides --
which previously required Boot's own management.otlp.* properties (still fully supported; see
the same transcript for MANAGEMENT_OTLP_METRICS_EXPORT_STEP working identically on 4.0). The
practical win: the exact same environment variables already used to configure an OTel Collector,
or a sidecar written in another language, now also configure this Spring Boot app, with nothing
Spring-specific to learn.
- Full variable-to-property mapping: Spring Boot 4.1 Release Notes, OpenTelemetry section
- Reproduce this yourself:
scripts/run-env-var-proof.sh
The sampling gotcha this surfaced along the way
Building the env-var proof meant staring at trace counts for a while, which surfaced something
worth a section of its own. management.tracing.sampling.probability defaults to 0.10 and
is completely independent of metrics -- metrics are never sampled at all.
15 requests sent to /orders/{101..115}, default sampling (0.10):
orders_placed_total increases by 15 -- every request counted
Tempo trace count increases by only 1 -- roughly 1 in 10 requests actually traced
Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0):
10 requests sent to /orders/{401..410}
Tempo trace count increases by 10 -- all of them, every time
(02-low-sampling-demo.txt, profile:
application.yml) The default is the right choice for
production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable
starting point. It is the wrong choice for a five-minute local demo: send a handful of test
requests at the default rate and the honest, common experience is "my trace isn't showing up,"
which reads exactly like a broken pipeline rather than working-as-designed sampling. fulltrace
exists in this repo for that reason -- flip it on for local poking, not for anything that sees
real traffic.
- If you enable
spring-boot-starter-opentelemetryand metrics show up in your backend but traces don't, check the sampling probability before you suspect the exporter.
Next: 4. Production checklist