Claude
03bdf7ee87
Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
72 lines
4.1 KiB
Markdown
72 lines
4.1 KiB
Markdown
# 3. OTEL_* environment variables, and the sampling gotcha they surface
|
|
|
|
Previous: [2. The Observation API and @Observed](02-observation-api.md) | [README](../README.md) | Next: [4. Production checklist](04-production-checklist.md)
|
|
|
|
## What changed in 4.1, verified against two real Boot versions
|
|
|
|
Spring Boot's own [4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
|
|
state plainly: "Support has been added to read most of the OpenTelemetry environment variables."
|
|
That is easy to either overstate (as if no `OTEL_*` variable worked before) or take on faith. The
|
|
real, empirical answer, from [`env-var-proof/`](../env-var-proof) -- **identical** Java source
|
|
compiled against `spring-boot-starter-parent` 4.0.8 and 4.1.1, run with only standard `OTEL_*`
|
|
environment variables set (no `management.otlp.*` Spring properties anywhere) and pointed at a
|
|
minimal stand-in OTLP receiver:
|
|
|
|
| Variable | Boot 4.0.8 | Boot 4.1.1 |
|
|
|---|---|---|
|
|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Honored | Honored |
|
|
| `OTEL_METRIC_EXPORT_INTERVAL` | **Ignored** (stays at the 1-minute default) | Honored |
|
|
|
|
```console
|
|
--- Boot 4.1.1, both env vars set ---
|
|
Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
|
|
... POST /v1/metrics (x11, one every ~2s)
|
|
|
|
--- Boot 4.0.8, same two env vars ---
|
|
Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
|
|
... POST /v1/metrics (x1, at JVM shutdown only)
|
|
```
|
|
|
|
([04-otel-env-vars-4.0-vs-4.1.txt](output/04-otel-env-vars-4.0-vs-4.1.txt)) The endpoint variable
|
|
already worked on 4.0 -- Micrometer's own `OtlpConfig` has long fallen back to
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT` independent of anything Spring-specific. What's actually new in 4.1
|
|
is the rest of the standard variable surface -- export interval, protocol, per-signal overrides --
|
|
which previously required Boot's own `management.otlp.*` properties (still fully supported; see
|
|
the same transcript for `MANAGEMENT_OTLP_METRICS_EXPORT_STEP` working identically on 4.0). The
|
|
practical win: the exact same environment variables already used to configure an OTel Collector,
|
|
or a sidecar written in another language, now also configure this Spring Boot app, with nothing
|
|
Spring-specific to learn.
|
|
|
|
- Full variable-to-property mapping: [Spring Boot 4.1 Release Notes, OpenTelemetry section](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes)
|
|
- Reproduce this yourself: [`scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh)
|
|
|
|
## The sampling gotcha this surfaced along the way
|
|
|
|
Building the env-var proof meant staring at trace counts for a while, which surfaced something
|
|
worth a section of its own. `management.tracing.sampling.probability` defaults to **0.10** and
|
|
is completely independent of metrics -- metrics are never sampled at all.
|
|
|
|
```console
|
|
15 requests sent to /orders/{101..115}, default sampling (0.10):
|
|
orders_placed_total increases by 15 -- every request counted
|
|
Tempo trace count increases by only 1 -- roughly 1 in 10 requests actually traced
|
|
|
|
Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0):
|
|
10 requests sent to /orders/{401..410}
|
|
Tempo trace count increases by 10 -- all of them, every time
|
|
```
|
|
|
|
([02-low-sampling-demo.txt](output/02-low-sampling-demo.txt), profile:
|
|
[`application.yml`](../src/main/resources/application.yml)) The default is the right choice for
|
|
production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable
|
|
starting point. It is the wrong choice for a five-minute local demo: send a handful of test
|
|
requests at the default rate and the honest, common experience is "my trace isn't showing up,"
|
|
which reads exactly like a broken pipeline rather than working-as-designed sampling. `fulltrace`
|
|
exists in this repo for that reason -- flip it on for local poking, not for anything that sees
|
|
real traffic.
|
|
|
|
- If you enable `spring-boot-starter-opentelemetry` and metrics show up in your backend but
|
|
traces don't, check the sampling probability before you suspect the exporter.
|
|
|
|
Next: [4. Production checklist](04-production-checklist.md)
|