Add observability: real OTLP metrics/traces to grafana/otel-lgtm, Docker Compose auto-wiring, and a dual-version (Boot 4.0 vs 4.1) proof of the new OTEL_* env var support
Companion module for the rewritten ankurm.com Prometheus/Grafana monitoring post. Verified against a real running grafana/otel-lgtm container (not mocked): 8 real requests produce a real orders_placed_total metric queried back from the bundled Prometheus-compatible API with zero management.otlp.* properties, auto-wired entirely by Boot's Docker Compose service-connection detection. Two real findings surfaced along the way and documented rather than smoothed over: @Observed silently produces no span without an explicit ObservedAspect bean (AspectJ weaving alone is not sufficient, despite Micrometer Tracing being active), and OTEL_EXPORTER_OTLP_ENDPOINT already worked on Boot 4.0 via Micrometer's own OtlpConfig fallback -- what's actually new in 4.1 is the rest of the standard OTEL_* surface (verified with OTEL_METRIC_EXPORT_INTERVAL against identical source compiled on both Boot 4.0.8 and 4.1.1). Also fixes the root README's module table, which was missing a row for resilience4j-circuit-breaker (added in a previous commit but never indexed here).
This commit is contained in:
Claude
@@ -0,0 +1,24 @@
|
||||
$ mvn spring-boot:run
|
||||
...
|
||||
DockerComposeLifecycleManager : Using Docker Compose file /tmp/obs-module/compose.yaml
|
||||
DockerCli : Network obs-module_default Creating
|
||||
DockerCli : Network obs-module_default Created
|
||||
DockerCli : Container obs-module-lgtm-1 Creating
|
||||
DockerCli : Container obs-module-lgtm-1 Created
|
||||
DockerCli : Container obs-module-lgtm-1 Starting
|
||||
DockerCli : Container obs-module-lgtm-1 Started
|
||||
DockerCli : Container obs-module-lgtm-1 Waiting
|
||||
DockerCli : Container obs-module-lgtm-1 Healthy
|
||||
...
|
||||
PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service}
|
||||
Started ObservabilityApplication in 34.426 seconds (process running for 34.652)
|
||||
|
||||
$ docker ps
|
||||
CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES
|
||||
4f768a81a102 grafana/otel-lgtm:latest "/otel-lgtm/run-all...." Up 46 seconds (healthy) 0.0.0.0:3000->3000/tcp, 0.0.0.0:3200->3200/tcp, 0.0.0.0:4317-4318->4317-4318/tcp, 0.0.0.0:9090->9090/tcp obs-module-lgtm-1
|
||||
|
||||
No management.otlp.* property was set anywhere in application.yml or on the command line for
|
||||
this run. The endpoint (127.0.0.1:4318) came entirely from Boot's Docker Compose service
|
||||
connection detecting the grafana/otel-lgtm image named in compose.yaml. Note the literal
|
||||
"127.0.0.1" -- a plain default (no compose.yaml, no service connection) reports "localhost" for
|
||||
this same port instead; that difference is the tell for whether auto-wiring actually happened.
|
||||
@@ -0,0 +1,29 @@
|
||||
8 real requests sent to the running app:
|
||||
$ for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i"; done
|
||||
|
||||
Queried straight from the real Prometheus-compatible API bundled inside the grafana/otel-lgtm
|
||||
container -- not the app's own /actuator endpoint, not a mock:
|
||||
|
||||
$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "vector",
|
||||
"result": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "orders_placed_total",
|
||||
"application": "order-service",
|
||||
"channel": "web",
|
||||
"job": "order-service",
|
||||
"service_name": "order-service"
|
||||
},
|
||||
"value": [1789722945.151, "8"]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
Every one of the 8 POSTs landed as a real OTLP metric, pushed over the network, through the
|
||||
Docker Compose auto-wired endpoint, and queried back out of the real Prometheus-compatible
|
||||
backend Grafana LGTM bundles.
|
||||
@@ -0,0 +1,23 @@
|
||||
Default management.tracing.sampling.probability (0.10, unset in application.yml):
|
||||
|
||||
$ for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
|
||||
# 15 requests sent
|
||||
|
||||
$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"
|
||||
# metric value: 15 -- all 15 counted, metrics are never subject to trace sampling
|
||||
|
||||
$ curl -s "http://localhost:3200/api/search?limit=50"
|
||||
# trace count increased by 1 -- only ~1 of the 15 requests was sampled into a trace
|
||||
|
||||
Same 15 requests, same app, same collector. Every one produced a metric data point. Roughly
|
||||
1 in 10 produced a trace, because trace sampling and metric recording are governed by two
|
||||
completely different knobs, and only one of them defaults to "record everything."
|
||||
|
||||
--- with the fulltrace profile (management.tracing.sampling.probability=1.0) ---
|
||||
|
||||
$ mvn -Dspring-boot.run.profiles=fulltrace spring-boot:run
|
||||
$ for i in $(seq 401 410); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done
|
||||
# 10 requests sent
|
||||
|
||||
$ curl -s "http://localhost:3200/api/search?limit=50"
|
||||
# post /orders traces increased by 10 -- 10 of 10, all of them present
|
||||
@@ -0,0 +1,32 @@
|
||||
OrderController#placeOrder is annotated @Observed(name = "place-order", contextualName =
|
||||
"order-controller#placeOrder"). Before ObservationConfig (an explicit ObservedAspect @Bean)
|
||||
existed in this repo, spring-boot-starter-opentelemetry, spring-boot-starter-aspectj (AspectJ
|
||||
weaver on the classpath) and micrometer-tracing were ALL already present and active -- and the
|
||||
annotation still did nothing. Real trace, fetched straight from Tempo's query API, before the fix:
|
||||
|
||||
$ curl -s "http://localhost:3200/api/traces/<trace-id>"
|
||||
span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
|
||||
# one span. @Observed's child span never appears.
|
||||
|
||||
After adding:
|
||||
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
class ObservationConfig {
|
||||
@Bean
|
||||
ObservedAspect observedAspect(ObservationRegistry registry) {
|
||||
return new ObservedAspect(registry);
|
||||
}
|
||||
}
|
||||
|
||||
Same annotation, same request, new trace:
|
||||
|
||||
$ curl -s "http://localhost:3200/api/traces/<trace-id>"
|
||||
span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: <server-span-id>
|
||||
span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root)
|
||||
# two spans. The @Observed child span now nests correctly under the HTTP server span.
|
||||
|
||||
Spring Boot's autoconfiguration wires an ObservationRegistry bean for you the moment Micrometer
|
||||
Tracing is on the classpath -- it does NOT also register an ObservedAspect. AspectJ weaving
|
||||
being present is necessary but not sufficient; without the aspect bean there is nothing for the
|
||||
weaver to apply. This is easy to miss because the app starts cleanly, the HTTP server span still
|
||||
shows up, and nothing logs a warning -- the method-level span is just silently absent.
|
||||
@@ -0,0 +1,43 @@
|
||||
Identical Java source (env-var-proof/OtelEnvProofApplication.java), compiled twice against two
|
||||
different spring-boot-starter-parent versions -- 4.0.8 and 4.1.1 -- run with ONLY standard OTEL_*
|
||||
environment variables set (zero management.otlp.* Spring properties anywhere), against a
|
||||
minimal stand-in OTLP receiver that just logs every POST it gets:
|
||||
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:PORT
|
||||
OTEL_METRIC_EXPORT_INTERVAL=2000
|
||||
|
||||
--- Spring Boot 4.1.1 ---
|
||||
|
||||
Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
|
||||
|
||||
receiver log (20s window):
|
||||
... POST /v1/metrics content-length=8247 (x11, one every ~2s)
|
||||
|
||||
Both the endpoint AND the interval standard env vars were honored.
|
||||
|
||||
--- Spring Boot 4.0.8, same two env vars ---
|
||||
|
||||
Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ...
|
||||
|
||||
receiver log (20s window):
|
||||
... POST /v1/metrics content-length=8247 (x1, at JVM shutdown only)
|
||||
|
||||
The endpoint env var was honored (Micrometer's own OtlpConfig has long fallen back to
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT independent of Spring Boot's own property binding). The interval
|
||||
env var was NOT -- the exporter still logs "every 1m", the JVM's own default, and the only POST
|
||||
the receiver saw was the one every PushMeterRegistry fires on shutdown regardless of interval.
|
||||
|
||||
--- Spring Boot 4.0.8, old-style Spring property instead ---
|
||||
|
||||
MANAGEMENT_OTLP_METRICS_EXPORT_URL=http://localhost:PORT/v1/metrics
|
||||
MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s
|
||||
|
||||
Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ...
|
||||
|
||||
receiver log (20s window):
|
||||
... POST /v1/metrics content-length=8435 (x11, one every ~2s)
|
||||
|
||||
Boot 4.0's own management.otlp.* properties always worked for this. What's new in 4.1 is that
|
||||
the SAME standard OTEL_* variable now works too, without a Spring-specific property to learn --
|
||||
useful the moment the same env vars are already used to configure something else in the same
|
||||
deployment (the OTel Collector, another language's service, docker-compose.yml).
|
||||
Reference in New Issue
Block a user