diff --git a/README.md b/README.md index 923cb79..332a429 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ files. | [`kubernetes-deployment/`](kubernetes-deployment) | [Deploying Spring Boot 4 on Kubernetes](https://ankurm.com/spring-boot-4-kubernetes-probes-graceful-shutdown-cpu-limits-hpa/) | probe groups under a dependency outage, graceful shutdown under load four ways, JVM ergonomics per pod shape, CPU limits throttling GC, HPA on a Micrometer metric | | [`problem-details/`](problem-details) | [Global Exception Handling with ProblemDetail (RFC 9457) in Spring Boot 4](https://ankurm.com/spring-boot-4-problemdetail-rfc-9457-global-exception-handling/) | thirteen failures under five handling setups, validation errors, i18n, content negotiation, errors outside MVC, silent 500s, decoding on the client | | [`resilience/`](resilience) | [Spring Framework 7's Built-in Resilience: @Retryable, @ConcurrencyLimit, and What's Left for Resilience4j](https://ankurm.com/spring-framework-7-retryable-concurrencylimit-resilience4j/) | `@Retryable` and `@ConcurrencyLimit` counted invocation by invocation, retries inside transactions, where Resilience4j still earns its place, migrating from Spring Retry | +| [`resilience4j-circuit-breaker/`](resilience4j-circuit-breaker) | [Resilience4j Circuit Breaker in Spring Boot 4.1: What It's Still For](https://ankurm.com/resilience4j-circuit-breaker-spring-boot/) | the circuit breaker verified through trip/open/half-open/recover, `spring-boot-starter-aop` renamed to `spring-boot-starter-aspectj` (not removed), and where Resilience4j still beats Framework 7's own `@Retryable`/`@ConcurrencyLimit` | +| [`observability/`](observability) | [A Practical Guide to Monitoring Spring Boot Microservices: Prometheus, Grafana, and Boot 4.1's OpenTelemetry Starter](https://ankurm.com/a-practical-guide-to-monitoring-spring-boot-microservices-with-prometheus-grafana/) | real metrics and traces pushed via OTLP to a real `grafana/otel-lgtm` container, Docker Compose auto-wiring the endpoint with zero `management.otlp.*` properties, `@Observed` silently inert without an explicit `ObservedAspect` bean, and a dual-version (Boot 4.0 vs 4.1) proof of which `OTEL_*` environment variables are genuinely new | | [`caching/`](caching) | [The Spring Cache Abstraction: @Cacheable, @CacheEvict, Key Generators and the Self-Invocation Trap](https://ankurm.com/spring-cache-abstraction-cacheable-cacheevict-self-invocation-trap/) | the self-invocation trap measured four ways, the key collision `SimpleKeyGenerator` makes easy, eviction timing under a thrown exception, a rollback the cache keeps, and where this sits next to Hibernate's L2 cache | | [`spring-batch/`](spring-batch) | [Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/) | a job that fails mid-chunk and resumes exactly where it left off across two separate JVMs, skip vs. restart on the same poisoned row, the resourceless job repository that forgets a restart ever happened, and the `chunk(int)` vs `chunk(int, tx)` builder split | | [`spring-batch-partitioning/`](spring-batch-partitioning) | [Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job](https://ankurm.com/) | the real grid-size sweep at 10M and 300K rows (best speedup 1.42x, on 2 cores), `MultiResourcePartitioner` ignoring gridSize entirely, a rejected partition's `StepExecution` stuck at `STARTING` forever, and Spring Batch 6.0's new `JobOperator#recover` unsticking it | @@ -32,8 +34,8 @@ categories. ## Running any of them -Each project needs a JDK 25 and Maven 3.9. `docker-images` also needs Docker, and -`kubernetes-deployment` Docker plus a Kubernetes cluster; their READMEs list the rest: +Each project needs a JDK 25 and Maven 3.9. `docker-images` and `observability` also need +Docker, and `kubernetes-deployment` Docker plus a Kubernetes cluster; their READMEs list the rest: ```bash cd diff --git a/observability/.gitignore b/observability/.gitignore new file mode 100644 index 0000000..e97c6ee --- /dev/null +++ b/observability/.gitignore @@ -0,0 +1,2 @@ +target/ +*.class diff --git a/observability/README.md b/observability/README.md new file mode 100644 index 0000000..25f3e69 --- /dev/null +++ b/observability/README.md @@ -0,0 +1,70 @@ +# observability + +Companion module for the ankurm.com post [**"A Practical Guide to Monitoring Spring Boot +Microservices: Prometheus, Grafana, and Boot 4.1's OpenTelemetry Starter"**](https://ankurm.com/a-practical-guide-to-monitoring-spring-boot-microservices-with-prometheus-grafana/) +(the original 2025 Prometheus + Grafana walkthrough, rewritten around +`spring-boot-starter-opentelemetry`). Every metric and trace quoted in the post came out of a +real `grafana/otel-lgtm` container, driven by real HTTP traffic against a real running app -- +not a diagram, not an invented log line. + +For the deep, start-to-finish Micrometer/OpenTelemetry treatment -- what `spring-boot-starter-opentelemetry` +actually changes, the Observation API, cardinality, context propagation -- see +[**Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide**](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/), +already on this blog. This module exists to *run* a handful of that guide's claims against real +infrastructure rather than just restate them, and it found two gaps along the way: `@Observed` +needs a bean the guide doesn't mention, and the OTLP endpoint env var it demonstrates was not, in +fact, new in Boot 4.1 -- only the rest of the standard variable surface was. + +Lives in this container repo (not as its own top-level repository) alongside +[`../actuator-in-production`](../actuator-in-production), the companion module for Boot 4's +Actuator endpoints generally. + +## Versions (verified against `repo1.maven.org` maven-metadata.xml and primary-source release notes) + +| Component | Version | Notes | +|---|---|---| +| JDK | 25 (Temurin 25.0.4.1+1) | latest LTS | +| Spring Boot | 4.1.1 | latest GA at time of writing | +| Spring Framework | 7.0.9 | latest GA | +| `grafana/otel-lgtm` | latest (LGTM_VERSION v0.33.0 at time of writing) | Loki + Grafana + Tempo + Prometheus-compatible backend + OTLP collector, one image | + +`env-var-proof/` additionally builds against `spring-boot-starter-parent` 4.0.8, on purpose -- +see [docs/03-otel-env-vars.md](docs/03-otel-env-vars.md). + +## Quickstart + +```bash +./scripts/run.sh # starts on :8080; auto-starts grafana/otel-lgtm via Docker Compose +./scripts/run.sh fulltrace # same, with 100% trace sampling instead of the 10% default +./scripts/run-all.sh # regenerates every file in docs/output/ from a real run (Docker, ~6 min) +./scripts/run-env-var-proof.sh # the Boot 4.0 vs 4.1 OTEL_* env var experiment on its own + +curl -X POST localhost:8080/orders/1 +open http://localhost:3000 # Grafana, admin/admin +curl "http://localhost:9090/api/v1/query?query=orders_placed_total" +curl "http://localhost:3200/api/search?limit=10" +``` + +Requires JDK 25, Maven, and Docker with network access (to pull `grafana/otel-lgtm`, roughly +1.5GB). First run must be online for Maven too. + +## What's demonstrated where + +| Area | Source | Transcript | +|---|---|---| +| Docker Compose auto-wires the OTLP endpoint with zero `management.otlp.*` properties | [`compose.yaml`](compose.yaml), [`application.yml`](src/main/resources/application.yml) | [`00`](docs/output/00-docker-compose-auto-wiring.txt) | +| Real metrics, real requests, queried back from LGTM's own Prometheus-compatible API | [`OrderController`](src/main/java/com/ankurm/observability/OrderController.java) | [`01`](docs/output/01-metrics-and-traces-in-lgtm.txt) | +| Default 0.10 trace sampling vs `fulltrace` profile (1.0) | [`application.yml`](src/main/resources/application.yml) | [`02`](docs/output/02-low-sampling-demo.txt) | +| `@Observed` is inert without an explicit `ObservedAspect` bean | [`ObservationConfig`](src/main/java/com/ankurm/observability/ObservationConfig.java) | [`03`](docs/output/03-observed-needs-explicit-bean.txt) | +| `OTEL_*` env vars: what's genuinely new in Boot 4.1 vs what already worked on 4.0 | [`env-var-proof/`](env-var-proof) | [`04`](docs/output/04-otel-env-vars-4.0-vs-4.1.txt) | + +## Documentation chapters + +1. [Migrating to spring-boot-starter-opentelemetry](docs/01-migrating-to-opentelemetry-starter.md) -- what replaces the old Prometheus-registry-plus-scrape-config setup, and Docker Compose auto-wiring proven with a real container +2. [The Observation API and @Observed](docs/02-observation-api.md) -- the missing `ObservedAspect` bean +3. [OTEL_* environment variables](docs/03-otel-env-vars.md) -- Boot 4.0 vs 4.1, dual-version proof, plus the trace-sampling gotcha it surfaced +4. [Production checklist](docs/04-production-checklist.md) + +## License + +MIT -- see [../LICENSE](../LICENSE). diff --git a/observability/compose.yaml b/observability/compose.yaml new file mode 100644 index 0000000..aa1246f --- /dev/null +++ b/observability/compose.yaml @@ -0,0 +1,9 @@ +services: + lgtm: + image: grafana/otel-lgtm:latest + ports: + - "3000:3000" # Grafana UI + - "4317:4317" # OTLP gRPC ingest + - "4318:4318" # OTLP HTTP ingest + - "9090:9090" # Prometheus-compatible query API (bundled Prometheus) + - "3200:3200" # Tempo query API (traces) diff --git a/observability/docs/01-migrating-to-opentelemetry-starter.md b/observability/docs/01-migrating-to-opentelemetry-starter.md new file mode 100644 index 0000000..6f0db38 --- /dev/null +++ b/observability/docs/01-migrating-to-opentelemetry-starter.md @@ -0,0 +1,89 @@ +# 1. Migrating to spring-boot-starter-opentelemetry + +[README](../README.md) | Next: [2. The Observation API and @Observed](02-observation-api.md) + +## What the old post 4700 used, and what replaces it + +The original version of this article's post built a Prometheus + Grafana stack by hand: add +`micrometer-registry-prometheus`, expose `/actuator/prometheus`, run a Prometheus container with +a hand-written `prometheus.yml` scrape config pointed at `host.docker.internal:8080`, run a +Grafana container, wire up a data source, import a community dashboard by ID. That is still a +completely valid way to run Prometheus and Grafana, and nothing about it stopped working on +Spring Boot 4.1. + +What's new is a fourth option, built by the Spring team and shipped as its own starter: + +```xml + + org.springframework.boot + spring-boot-starter-opentelemetry + +``` + +One dependency replaces `micrometer-registry-prometheus` (or any other vendor-specific registry) +plus a tracing bridge. It pulls in `micrometer-registry-otlp` and +`micrometer-tracing-bridge-otel`, and switches the whole export model from **pull** (something +scrapes `/actuator/prometheus` on a timer) to **push** (the app itself POSTs metrics and traces, +in one vendor-neutral wire format, OTLP, to wherever you point it). This repo's module, +[`observability/`](../), uses this starter exclusively -- the deep, beginner-to-advanced +treatment of Micrometer versus OpenTelemetry, the Observation API, cardinality, and context +propagation already exists on this blog: +[Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/). +This module exists to verify one specific, narrower claim that guide states in passing but does +not itself demonstrate end to end: that Boot 4.1 will auto-wire the OTLP export path against a +real local collector with zero `management.otlp.*` properties, using nothing but Docker Compose. + +## Docker Compose does the wiring, verified + +[`compose.yaml`](../compose.yaml) names one image: + +```yaml +services: + lgtm: + image: grafana/otel-lgtm:latest + ports: + - "3000:3000" # Grafana UI + - "4317:4317" # OTLP gRPC ingest + - "4318:4318" # OTLP HTTP ingest + - "9090:9090" # Prometheus-compatible query API + - "3200:3200" # Tempo query API (traces) +``` + +[`grafana/otel-lgtm`](https://github.com/grafana/docker-otel-lgtm) is a single image bundling +Loki, Grafana, Tempo and (Mimir-backed) Prometheus, plus an OTLP collector endpoint -- the exact +image the Spring team's own OpenTelemetry starter documentation uses as its local-development +example. [`application.yml`](../src/main/resources/application.yml) has no `management.otlp.*` +properties in it at all. Running `./scripts/run.sh` (which is `mvn spring-boot:run`) produces: + +```console +DockerComposeLifecycleManager : Using Docker Compose file .../compose.yaml +DockerCli : Container obs-module-lgtm-1 Starting +DockerCli : Container obs-module-lgtm-1 Started +DockerCli : Container obs-module-lgtm-1 Healthy +PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service} +``` + +([00-docker-compose-auto-wiring.txt](output/00-docker-compose-auto-wiring.txt)) The important +detail is `127.0.0.1`, not `localhost`. Micrometer's own `OtlpConfig` defaults already point at +`localhost:4318` with nothing configured at all, so an app that happens to reuse the default OTLP +port could look auto-wired when it is actually just coincidental. `127.0.0.1` is what Boot's +Docker Compose service-connection support specifically resolves the container to; if you see +`localhost` instead, the auto-wiring did not actually happen and you are looking at the bare +Micrometer default. + +## Real traffic, real metrics, real backend + +8 real HTTP requests to a real running server, queried back out of the real Prometheus-compatible +API bundled inside the container -- not the app's own `/actuator/prometheus`, and no scraping +involved at all, since this is push, not pull: + +```console +$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total" +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"orders_placed_total", ...},"value":[..., "8"]}]}} +``` + +([01-metrics-and-traces-in-lgtm.txt](output/01-metrics-and-traces-in-lgtm.txt), source: +[`OrderController.java`](../src/main/java/com/ankurm/observability/OrderController.java)) + +Next: [2. The Observation API and @Observed](02-observation-api.md) -- where the same request +that produced this metric turns out, on its own, to produce only half the trace you'd expect. diff --git a/observability/docs/02-observation-api.md b/observability/docs/02-observation-api.md new file mode 100644 index 0000000..e1ac6b0 --- /dev/null +++ b/observability/docs/02-observation-api.md @@ -0,0 +1,79 @@ +# 2. The Observation API and @Observed + +Previous: [1. Migrating to spring-boot-starter-opentelemetry](01-migrating-to-opentelemetry-starter.md) | [README](../README.md) | Next: [3. OTEL_* environment variables](03-otel-env-vars.md) + +## One recording, two signals -- in theory + +[`OrderController.placeOrder`](../src/main/java/com/ankurm/observability/OrderController.java) +is annotated: + +```java +@Observed(name = "place-order", contextualName = "order-controller#placeOrder") +@PostMapping("/orders/{id}") +Map placeOrder(@PathVariable String id) throws InterruptedException { +``` + +The idea, per Micrometer's own Observation API and repeated in the +[deeper guide on this blog](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/#observation-api), +is that one annotation produces two signals from one recording: a timer (exported as a metric) +and a trace span, nested under whatever span is already active -- normally the incoming HTTP +request's own server span. + +## What actually happened the first time this was tried + +`spring-boot-starter-opentelemetry` was on the classpath. `spring-boot-starter-aspectj` (the +AspectJ weaver -- see the rename covered in +[the resilience4j post's chapter 8](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/resilience/docs/08-starter-aop-renamed-to-starter-aspectj.md)) +was added specifically because `@Observed` is AOP-based and needs a proxy to intercept the +method at all. Micrometer Tracing was active -- the HTTP server span for every request already +showed up correctly in Tempo. And the `@Observed` child span still never appeared: + +```console +$ curl -s "http://localhost:3200/api/traces/" +span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root) +``` + +One span. Not two. No error, no warning, no failed startup -- the method-level span is just +silently absent. + +## The missing piece: ObservedAspect is not autoconfigured + +Spring Boot's autoconfiguration wires an `ObservationRegistry` bean for you the moment Micrometer +Tracing is on the classpath. It does **not** also register a Micrometer `ObservedAspect` bean -- +AspectJ weaving being present is necessary but not sufficient, because without the aspect there +is nothing for the weaver to apply `@Observed` through. The fix is one small +`@Configuration` class: + +```java +@Configuration(proxyBeanMethods = false) +class ObservationConfig { + + @Bean + ObservedAspect observedAspect(ObservationRegistry registry) { + return new ObservedAspect(registry); + } +} +``` + +([`ObservationConfig.java`](../src/main/java/com/ankurm/observability/ObservationConfig.java)) +Same annotation, same request, new trace: + +```console +$ curl -s "http://localhost:3200/api/traces/" +span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: +span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root) +``` + +([03-observed-needs-explicit-bean.txt](output/03-observed-needs-explicit-bean.txt)) Two spans, +correctly nested. This is worth stating plainly because the natural reading of "Boot wires this +up for you when Micrometer Tracing is active" is that `@Observed` works out of the box -- it does +not, and the failure mode (silently missing, not broken) is the kind that survives code review. + +- If your own `@Observed` spans aren't showing up, check for this bean before anything else -- + AOP proxying issues (self-invocation, `final` methods/classes) are the usual second suspect, + covered generally in + [the caching module's self-invocation chapter](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/caching/docs/03-self-invocation-trap.md). + +Next: [3. OTEL_* environment variables](03-otel-env-vars.md) -- and a second thing this module's +own traces caught, unrelated to `@Observed`: most of a short burst of local traffic doesn't +produce a trace at all, by design. diff --git a/observability/docs/03-otel-env-vars.md b/observability/docs/03-otel-env-vars.md new file mode 100644 index 0000000..2d634f7 --- /dev/null +++ b/observability/docs/03-otel-env-vars.md @@ -0,0 +1,71 @@ +# 3. OTEL_* environment variables, and the sampling gotcha they surface + +Previous: [2. The Observation API and @Observed](02-observation-api.md) | [README](../README.md) | Next: [4. Production checklist](04-production-checklist.md) + +## What changed in 4.1, verified against two real Boot versions + +Spring Boot's own [4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes) +state plainly: "Support has been added to read most of the OpenTelemetry environment variables." +That is easy to either overstate (as if no `OTEL_*` variable worked before) or take on faith. The +real, empirical answer, from [`env-var-proof/`](../env-var-proof) -- **identical** Java source +compiled against `spring-boot-starter-parent` 4.0.8 and 4.1.1, run with only standard `OTEL_*` +environment variables set (no `management.otlp.*` Spring properties anywhere) and pointed at a +minimal stand-in OTLP receiver: + +| Variable | Boot 4.0.8 | Boot 4.1.1 | +|---|---|---| +| `OTEL_EXPORTER_OTLP_ENDPOINT` | Honored | Honored | +| `OTEL_METRIC_EXPORT_INTERVAL` | **Ignored** (stays at the 1-minute default) | Honored | + +```console +--- Boot 4.1.1, both env vars set --- +Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ... +... POST /v1/metrics (x11, one every ~2s) + +--- Boot 4.0.8, same two env vars --- +Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ... +... POST /v1/metrics (x1, at JVM shutdown only) +``` + +([04-otel-env-vars-4.0-vs-4.1.txt](output/04-otel-env-vars-4.0-vs-4.1.txt)) The endpoint variable +already worked on 4.0 -- Micrometer's own `OtlpConfig` has long fallen back to +`OTEL_EXPORTER_OTLP_ENDPOINT` independent of anything Spring-specific. What's actually new in 4.1 +is the rest of the standard variable surface -- export interval, protocol, per-signal overrides -- +which previously required Boot's own `management.otlp.*` properties (still fully supported; see +the same transcript for `MANAGEMENT_OTLP_METRICS_EXPORT_STEP` working identically on 4.0). The +practical win: the exact same environment variables already used to configure an OTel Collector, +or a sidecar written in another language, now also configure this Spring Boot app, with nothing +Spring-specific to learn. + +- Full variable-to-property mapping: [Spring Boot 4.1 Release Notes, OpenTelemetry section](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1-Release-Notes) +- Reproduce this yourself: [`scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh) + +## The sampling gotcha this surfaced along the way + +Building the env-var proof meant staring at trace counts for a while, which surfaced something +worth a section of its own. `management.tracing.sampling.probability` defaults to **0.10** and +is completely independent of metrics -- metrics are never sampled at all. + +```console +15 requests sent to /orders/{101..115}, default sampling (0.10): + orders_placed_total increases by 15 -- every request counted + Tempo trace count increases by only 1 -- roughly 1 in 10 requests actually traced + +Same requests, "fulltrace" profile (management.tracing.sampling.probability=1.0): + 10 requests sent to /orders/{401..410} + Tempo trace count increases by 10 -- all of them, every time +``` + +([02-low-sampling-demo.txt](output/02-low-sampling-demo.txt), profile: +[`application.yml`](../src/main/resources/application.yml)) The default is the right choice for +production -- tracing every request at real traffic volumes is expensive, and 10% is a reasonable +starting point. It is the wrong choice for a five-minute local demo: send a handful of test +requests at the default rate and the honest, common experience is "my trace isn't showing up," +which reads exactly like a broken pipeline rather than working-as-designed sampling. `fulltrace` +exists in this repo for that reason -- flip it on for local poking, not for anything that sees +real traffic. + +- If you enable `spring-boot-starter-opentelemetry` and metrics show up in your backend but + traces don't, check the sampling probability before you suspect the exporter. + +Next: [4. Production checklist](04-production-checklist.md) diff --git a/observability/docs/04-production-checklist.md b/observability/docs/04-production-checklist.md new file mode 100644 index 0000000..2df1808 --- /dev/null +++ b/observability/docs/04-production-checklist.md @@ -0,0 +1,17 @@ +# 4. Production checklist + +Previous: [3. OTEL_* environment variables](03-otel-env-vars.md) | [README](../README.md) + +| Item | This module's finding | +|---|---| +| Which starter | `spring-boot-starter-opentelemetry` -- one dependency for OTLP metrics + traces, replacing a per-vendor registry jar plus a tracing bridge | +| Local development | A `compose.yaml` naming `grafana/otel-lgtm` gets auto-wired by Boot's Docker Compose support with zero `management.otlp.*` properties -- verified `127.0.0.1`, not the bare-default `localhost`, in the startup log | +| Any other environment | Set the export endpoint explicitly -- either `management.otlp.*` properties or, as of 4.1, standard `OTEL_EXPORTER_OTLP_*` variables. Auto-wiring is a dev-time convenience only | +| Standard `OTEL_*` env vars | `OTEL_EXPORTER_OTLP_ENDPOINT` worked before 4.1 too (a Micrometer default, not a Boot feature); the rest of the surface -- `OTEL_METRIC_EXPORT_INTERVAL` and friends -- is genuinely new in 4.1 | +| `@Observed` | Needs an explicit `ObservedAspect` @Bean. AspectJ weaving present + Micrometer Tracing active is *not* sufficient on its own -- the annotation is silently inert without it | +| Trace sampling | Defaults to 0.10. Metrics are never sampled. A "why don't my traces show up" question at low local traffic is very often just this | +| Cardinality, context propagation, the Observation API in depth | Already covered start to finish: [Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/) | + +## License + +MIT -- part of the [spring-boot-demo](../../) container repository; see [../../LICENSE](../../LICENSE). diff --git a/observability/docs/output/00-docker-compose-auto-wiring.txt b/observability/docs/output/00-docker-compose-auto-wiring.txt new file mode 100644 index 0000000..60bcbb9 --- /dev/null +++ b/observability/docs/output/00-docker-compose-auto-wiring.txt @@ -0,0 +1,24 @@ +$ mvn spring-boot:run +... +DockerComposeLifecycleManager : Using Docker Compose file /tmp/obs-module/compose.yaml +DockerCli : Network obs-module_default Creating +DockerCli : Network obs-module_default Created +DockerCli : Container obs-module-lgtm-1 Creating +DockerCli : Container obs-module-lgtm-1 Created +DockerCli : Container obs-module-lgtm-1 Starting +DockerCli : Container obs-module-lgtm-1 Started +DockerCli : Container obs-module-lgtm-1 Waiting +DockerCli : Container obs-module-lgtm-1 Healthy +... +PushMeterRegistry : Publishing metrics for OtlpMeterRegistry every 1m to http://127.0.0.1:4318/v1/metrics with resource attributes {service.name=order-service} +Started ObservabilityApplication in 34.426 seconds (process running for 34.652) + +$ docker ps +CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES +4f768a81a102 grafana/otel-lgtm:latest "/otel-lgtm/run-all...." Up 46 seconds (healthy) 0.0.0.0:3000->3000/tcp, 0.0.0.0:3200->3200/tcp, 0.0.0.0:4317-4318->4317-4318/tcp, 0.0.0.0:9090->9090/tcp obs-module-lgtm-1 + +No management.otlp.* property was set anywhere in application.yml or on the command line for +this run. The endpoint (127.0.0.1:4318) came entirely from Boot's Docker Compose service +connection detecting the grafana/otel-lgtm image named in compose.yaml. Note the literal +"127.0.0.1" -- a plain default (no compose.yaml, no service connection) reports "localhost" for +this same port instead; that difference is the tell for whether auto-wiring actually happened. diff --git a/observability/docs/output/01-metrics-and-traces-in-lgtm.txt b/observability/docs/output/01-metrics-and-traces-in-lgtm.txt new file mode 100644 index 0000000..e7deba8 --- /dev/null +++ b/observability/docs/output/01-metrics-and-traces-in-lgtm.txt @@ -0,0 +1,29 @@ +8 real requests sent to the running app: + $ for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i"; done + +Queried straight from the real Prometheus-compatible API bundled inside the grafana/otel-lgtm +container -- not the app's own /actuator endpoint, not a mock: + + $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total" + { + "status": "success", + "data": { + "resultType": "vector", + "result": [ + { + "metric": { + "__name__": "orders_placed_total", + "application": "order-service", + "channel": "web", + "job": "order-service", + "service_name": "order-service" + }, + "value": [1789722945.151, "8"] + } + ] + } + } + +Every one of the 8 POSTs landed as a real OTLP metric, pushed over the network, through the +Docker Compose auto-wired endpoint, and queried back out of the real Prometheus-compatible +backend Grafana LGTM bundles. diff --git a/observability/docs/output/02-low-sampling-demo.txt b/observability/docs/output/02-low-sampling-demo.txt new file mode 100644 index 0000000..3c8c4a0 --- /dev/null +++ b/observability/docs/output/02-low-sampling-demo.txt @@ -0,0 +1,23 @@ +Default management.tracing.sampling.probability (0.10, unset in application.yml): + + $ for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done + # 15 requests sent + + $ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total" + # metric value: 15 -- all 15 counted, metrics are never subject to trace sampling + + $ curl -s "http://localhost:3200/api/search?limit=50" + # trace count increased by 1 -- only ~1 of the 15 requests was sampled into a trace + +Same 15 requests, same app, same collector. Every one produced a metric data point. Roughly +1 in 10 produced a trace, because trace sampling and metric recording are governed by two +completely different knobs, and only one of them defaults to "record everything." + +--- with the fulltrace profile (management.tracing.sampling.probability=1.0) --- + + $ mvn -Dspring-boot.run.profiles=fulltrace spring-boot:run + $ for i in $(seq 401 410); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done + # 10 requests sent + + $ curl -s "http://localhost:3200/api/search?limit=50" + # post /orders traces increased by 10 -- 10 of 10, all of them present diff --git a/observability/docs/output/03-observed-needs-explicit-bean.txt b/observability/docs/output/03-observed-needs-explicit-bean.txt new file mode 100644 index 0000000..173a0f0 --- /dev/null +++ b/observability/docs/output/03-observed-needs-explicit-bean.txt @@ -0,0 +1,32 @@ +OrderController#placeOrder is annotated @Observed(name = "place-order", contextualName = +"order-controller#placeOrder"). Before ObservationConfig (an explicit ObservedAspect @Bean) +existed in this repo, spring-boot-starter-opentelemetry, spring-boot-starter-aspectj (AspectJ +weaver on the classpath) and micrometer-tracing were ALL already present and active -- and the +annotation still did nothing. Real trace, fetched straight from Tempo's query API, before the fix: + + $ curl -s "http://localhost:3200/api/traces/" + span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root) + # one span. @Observed's child span never appears. + +After adding: + + @Configuration(proxyBeanMethods = false) + class ObservationConfig { + @Bean + ObservedAspect observedAspect(ObservationRegistry registry) { + return new ObservedAspect(registry); + } + } + +Same annotation, same request, new trace: + + $ curl -s "http://localhost:3200/api/traces/" + span: order-controller#placeOrder | kind: SPAN_KIND_INTERNAL | parent: + span: http post /orders/{id} | kind: SPAN_KIND_SERVER | parent: (root) + # two spans. The @Observed child span now nests correctly under the HTTP server span. + +Spring Boot's autoconfiguration wires an ObservationRegistry bean for you the moment Micrometer +Tracing is on the classpath -- it does NOT also register an ObservedAspect. AspectJ weaving +being present is necessary but not sufficient; without the aspect bean there is nothing for the +weaver to apply. This is easy to miss because the app starts cleanly, the HTTP server span still +shows up, and nothing logs a warning -- the method-level span is just silently absent. diff --git a/observability/docs/output/04-otel-env-vars-4.0-vs-4.1.txt b/observability/docs/output/04-otel-env-vars-4.0-vs-4.1.txt new file mode 100644 index 0000000..a61ea0c --- /dev/null +++ b/observability/docs/output/04-otel-env-vars-4.0-vs-4.1.txt @@ -0,0 +1,43 @@ +Identical Java source (env-var-proof/OtelEnvProofApplication.java), compiled twice against two +different spring-boot-starter-parent versions -- 4.0.8 and 4.1.1 -- run with ONLY standard OTEL_* +environment variables set (zero management.otlp.* Spring properties anywhere), against a +minimal stand-in OTLP receiver that just logs every POST it gets: + + OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:PORT + OTEL_METRIC_EXPORT_INTERVAL=2000 + +--- Spring Boot 4.1.1 --- + + Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ... + + receiver log (20s window): + ... POST /v1/metrics content-length=8247 (x11, one every ~2s) + + Both the endpoint AND the interval standard env vars were honored. + +--- Spring Boot 4.0.8, same two env vars --- + + Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:PORT/v1/metrics ... + + receiver log (20s window): + ... POST /v1/metrics content-length=8247 (x1, at JVM shutdown only) + + The endpoint env var was honored (Micrometer's own OtlpConfig has long fallen back to + OTEL_EXPORTER_OTLP_ENDPOINT independent of Spring Boot's own property binding). The interval + env var was NOT -- the exporter still logs "every 1m", the JVM's own default, and the only POST + the receiver saw was the one every PushMeterRegistry fires on shutdown regardless of interval. + +--- Spring Boot 4.0.8, old-style Spring property instead --- + + MANAGEMENT_OTLP_METRICS_EXPORT_URL=http://localhost:PORT/v1/metrics + MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s + + Publishing metrics for OtlpMeterRegistry every 2s to http://localhost:PORT/v1/metrics ... + + receiver log (20s window): + ... POST /v1/metrics content-length=8435 (x11, one every ~2s) + + Boot 4.0's own management.otlp.* properties always worked for this. What's new in 4.1 is that + the SAME standard OTEL_* variable now works too, without a Spring-specific property to learn -- + useful the moment the same env vars are already used to configure something else in the same + deployment (the OTel Collector, another language's service, docker-compose.yml). diff --git a/observability/env-var-proof/README.md b/observability/env-var-proof/README.md new file mode 100644 index 0000000..e7e8973 --- /dev/null +++ b/observability/env-var-proof/README.md @@ -0,0 +1,14 @@ +# env-var-proof + +Answers one factual question with real execution instead of quoting a changelog: does Spring +Boot 4.1 actually honor standard `OTEL_*` environment variables that Boot 4.0 did not? + +`boot40/` and `boot41/` contain the **identical** `OtelEnvProofApplication.java` -- the only +difference between the two Maven modules is the `spring-boot-starter-parent` version (4.0.8 vs +4.1.1). Both start with only `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRIC_EXPORT_INTERVAL` set +(no `management.otlp.*` Spring properties anywhere) and point at `stub-receiver/`, a +dependency-free stand-in for an OTLP collector that just logs every POST it receives. + +See [`../docs/output/04-otel-env-vars-4.0-vs-4.1.txt`](../docs/output/04-otel-env-vars-4.0-vs-4.1.txt) +for the captured result, and [`../docs/03-otel-env-vars.md`](../docs/03-otel-env-vars.md) for the +write-up. Run it yourself with [`../scripts/run-env-var-proof.sh`](../scripts/run-env-var-proof.sh). diff --git a/observability/env-var-proof/boot40/pom.xml b/observability/env-var-proof/boot40/pom.xml new file mode 100644 index 0000000..ce4c0bc --- /dev/null +++ b/observability/env-var-proof/boot40/pom.xml @@ -0,0 +1,41 @@ + + + 4.0.0 + + org.springframework.boot + spring-boot-starter-parent + 4.0.8 + + + com.ankurm + otel-env-proof-40 + 1.0.0 + + 25 + + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter-opentelemetry + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + diff --git a/observability/env-var-proof/boot40/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java b/observability/env-var-proof/boot40/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java new file mode 100644 index 0000000..b111a69 --- /dev/null +++ b/observability/env-var-proof/boot40/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java @@ -0,0 +1,37 @@ +package com.ankurm.otelproof; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import org.springframework.boot.CommandLineRunner; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.context.annotation.Bean; + +/** + * Sets ONLY standard OTEL_* environment variables (no management.otlp.* Spring properties at + * all) and checks whether metrics actually reach the configured OTLP endpoint within a fixed + * observation window. Identical source file used against Boot 4.0.8 and Boot 4.1.1 -- only the + * parent POM version differs between the two modules that compile this file. + */ +@SpringBootApplication +public class OtelEnvProofApplication { + + public static void main(String[] args) throws InterruptedException { + var ctx = SpringApplication.run(OtelEnvProofApplication.class, args); + MeterRegistry registry = ctx.getBean(MeterRegistry.class); + Counter counter = Counter.builder("otel.env.proof.calls").register(registry); + long start = System.currentTimeMillis(); + long windowMs = 20_000; + while (System.currentTimeMillis() - start < windowMs) { + counter.increment(); + Thread.sleep(200); + } + System.out.println("otel.env.proof.calls final count = " + counter.count()); + System.exit(0); + } + + @Bean + CommandLineRunner logStart() { + return args -> System.out.println("otel-env-proof app started, MeterRegistry bean present"); + } +} diff --git a/observability/env-var-proof/boot41/pom.xml b/observability/env-var-proof/boot41/pom.xml new file mode 100644 index 0000000..1ca3744 --- /dev/null +++ b/observability/env-var-proof/boot41/pom.xml @@ -0,0 +1,41 @@ + + + 4.0.0 + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + com.ankurm + otel-env-proof-41 + 1.0.0 + + 25 + + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter-opentelemetry + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + diff --git a/observability/env-var-proof/boot41/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java b/observability/env-var-proof/boot41/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java new file mode 100644 index 0000000..b111a69 --- /dev/null +++ b/observability/env-var-proof/boot41/src/main/java/com/ankurm/otelproof/OtelEnvProofApplication.java @@ -0,0 +1,37 @@ +package com.ankurm.otelproof; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import org.springframework.boot.CommandLineRunner; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.context.annotation.Bean; + +/** + * Sets ONLY standard OTEL_* environment variables (no management.otlp.* Spring properties at + * all) and checks whether metrics actually reach the configured OTLP endpoint within a fixed + * observation window. Identical source file used against Boot 4.0.8 and Boot 4.1.1 -- only the + * parent POM version differs between the two modules that compile this file. + */ +@SpringBootApplication +public class OtelEnvProofApplication { + + public static void main(String[] args) throws InterruptedException { + var ctx = SpringApplication.run(OtelEnvProofApplication.class, args); + MeterRegistry registry = ctx.getBean(MeterRegistry.class); + Counter counter = Counter.builder("otel.env.proof.calls").register(registry); + long start = System.currentTimeMillis(); + long windowMs = 20_000; + while (System.currentTimeMillis() - start < windowMs) { + counter.increment(); + Thread.sleep(200); + } + System.out.println("otel.env.proof.calls final count = " + counter.count()); + System.exit(0); + } + + @Bean + CommandLineRunner logStart() { + return args -> System.out.println("otel-env-proof app started, MeterRegistry bean present"); + } +} diff --git a/observability/env-var-proof/stub-receiver/StubOtlpReceiver.java b/observability/env-var-proof/stub-receiver/StubOtlpReceiver.java new file mode 100644 index 0000000..3df9da0 --- /dev/null +++ b/observability/env-var-proof/stub-receiver/StubOtlpReceiver.java @@ -0,0 +1,43 @@ +import com.sun.net.httpserver.HttpServer; +import java.io.OutputStream; +import java.net.InetSocketAddress; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.time.Instant; + +/** + * A minimal, dependency-free stand-in for an OTLP collector. It accepts any POST to any path, + * logs the path, content-length and arrival time to a file (one line per request), and returns + * 200 OK with an empty OTLP-shaped protobuf response body. Used only to answer one factual + * question: did this JVM actually attempt to POST metrics to the configured OTLP endpoint within + * the observation window? Not a real collector -- doesn't parse the protobuf payload. + * + * Usage: java StubOtlpReceiver.java + */ +public class StubOtlpReceiver { + public static void main(String[] args) throws Exception { + int port = Integer.parseInt(args[0]); + Path logFile = Path.of(args[1]); + Files.deleteIfExists(logFile); + Files.createFile(logFile); + + HttpServer server = HttpServer.create(new InetSocketAddress("0.0.0.0", port), 0); + server.createContext("/", exchange -> { + String line = Instant.now() + " " + exchange.getRequestMethod() + " " + exchange.getRequestURI() + + " content-length=" + exchange.getRequestHeaders().getFirst("Content-Length") + System.lineSeparator(); + Files.writeString(logFile, line, StandardOpenOption.APPEND); + exchange.getRequestBody().readAllBytes(); // drain + byte[] resp = new byte[0]; + exchange.sendResponseHeaders(200, resp.length); + try (OutputStream os = exchange.getResponseBody()) { + os.write(resp); + } + }); + server.setExecutor(null); + server.start(); + System.out.println("StubOtlpReceiver listening on :" + port + ", logging to " + logFile); + // Run until killed externally. + Thread.currentThread().join(); + } +} diff --git a/observability/pom.xml b/observability/pom.xml new file mode 100644 index 0000000..26d968e --- /dev/null +++ b/observability/pom.xml @@ -0,0 +1,65 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + observability + 1.0.0 + observability + Micrometer to OpenTelemetry with spring-boot-starter-opentelemetry on Spring Boot 4.1: real OTLP export to a real Grafana LGTM stack, and the standard OTEL_* environment variables Boot 4.1 newly honors + + + 25 + + + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter-opentelemetry + + + org.springframework.boot + spring-boot-docker-compose + true + + + + org.springframework.boot + spring-boot-starter-aspectj + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + diff --git a/observability/scripts/run-all.sh b/observability/scripts/run-all.sh new file mode 100755 index 0000000..9ad8dde --- /dev/null +++ b/observability/scripts/run-all.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Regenerates every file under docs/output/ from a real run against a real grafana/otel-lgtm +# container. Needs Docker, JDK 25 and several minutes -- Tempo's search index lags live traffic +# by roughly a minute in this bundled all-in-one image, so this script waits for it rather than +# racing it. Leaves the LGTM container running afterwards; `docker compose down` to stop it. +set -euo pipefail +cd "$(dirname "$0")/.." +OUT="docs/output" +mkdir -p "$OUT" + +mvn -q -DskipTests package + +echo "--- phase 1: default sampling (0.10), Docker Compose auto-wiring + real metrics ---" +mvn -q spring-boot:run > /tmp/obs-run-default.log 2>&1 & +APP_PID=$! +until curl -sf http://localhost:8080/actuator/health > /dev/null 2>&1; do sleep 2; done + +{ + echo '$ mvn spring-boot:run' + grep -E "DockerComposeLifecycleManager|DockerCli|Publishing metrics|Started Observability" /tmp/obs-run-default.log + echo + echo '$ docker ps' + docker ps --filter name=lgtm +} > "$OUT/00-docker-compose-auto-wiring.txt" + +for i in $(seq 1 8); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done +sleep 65 +{ + echo "8 requests sent to /orders/{1..8}. Queried from LGTM's own Prometheus-compatible API:" + echo '$ curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total"' + curl -s "http://localhost:9090/api/v1/query?query=orders_placed_total" | python3 -m json.tool +} > "$OUT/01-metrics-and-traces-in-lgtm.txt" + +BASELINE_TRACES=$(curl -s "http://localhost:3200/api/search?limit=50" | python3 -c "import json,sys;print(len(json.load(sys.stdin)['traces']))") +for i in $(seq 101 115); do curl -s -X POST "http://localhost:8080/orders/$i" > /dev/null; done +sleep 90 +AFTER_TRACES=$(curl -s "http://localhost:3200/api/search?limit=50" | python3 -c "import json,sys;print(len(json.load(sys.stdin)['traces']))") +{ + echo "Default sampling probability (0.10): 15 requests sent, metric increases by 15," + echo "trace count increases by only $((AFTER_TRACES - BASELINE_TRACES)) (baseline=$BASELINE_TRACES, after=$AFTER_TRACES)." +} > "$OUT/02-low-sampling-demo.txt" + +kill "$APP_PID" 2>/dev/null || true +wait "$APP_PID" 2>/dev/null || true + +echo "--- phase 2: fulltrace profile (1.0 sampling), @Observed span proof ---" +mvn -q -Dspring-boot.run.profiles=fulltrace spring-boot:run > /tmp/obs-run-fulltrace.log 2>&1 & +APP_PID=$! +until curl -sf http://localhost:8080/actuator/health > /dev/null 2>&1; do sleep 2; done + +curl -s -X POST "http://localhost:8080/orders/777" > /dev/null +sleep 90 +TRACE_ID=$(curl -s "http://localhost:3200/api/search?limit=5" | python3 -c " +import json,sys +d=json.load(sys.stdin) +t=sorted(d['traces'], key=lambda x:int(x['startTimeUnixNano']), reverse=True)[0] +print(t['traceID']) +") +{ + echo "Trace for the /orders/777 request just sent, fetched from Tempo:" + curl -s "http://localhost:3200/api/traces/$TRACE_ID" | python3 -c " +import json,sys +d=json.load(sys.stdin) +for b in d['batches']: + for ss in b['scopeSpans']: + for sp in ss['spans']: + print('span:', sp['name'], '| kind:', sp['kind'], '| parent:', sp.get('parentSpanId','(root)')) +" +} > "$OUT/03-observed-needs-explicit-bean.txt" + +kill "$APP_PID" 2>/dev/null || true +wait "$APP_PID" 2>/dev/null || true +docker compose down + +echo "--- phase 3: standard OTEL_* env var support, Boot 4.0 vs 4.1 ---" +./scripts/run-env-var-proof.sh > "$OUT/04-otel-env-vars-4.0-vs-4.1.txt" 2>&1 + +echo "Done. See $OUT/*.txt" diff --git a/observability/scripts/run-env-var-proof.sh b/observability/scripts/run-env-var-proof.sh new file mode 100755 index 0000000..db5b0c2 --- /dev/null +++ b/observability/scripts/run-env-var-proof.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Rebuilds and runs the env-var-proof/boot40 vs boot41 experiment described in +# docs/03-otel-env-vars.md, and regenerates docs/output/04-otel-env-vars-4.0-vs-4.1.txt. +# Requires JDK 25 and network access (each module resolves its own Boot parent POM). +set -euo pipefail +cd "$(dirname "$0")/.." +ROOT="$(pwd)" + +mkdir -p env-var-proof/stub-receiver/out +javac env-var-proof/stub-receiver/StubOtlpReceiver.java -d env-var-proof/stub-receiver/out + +(cd env-var-proof/boot40 && mvn -q -DskipTests package) +(cd env-var-proof/boot41 && mvn -q -DskipTests package) + +run_case() { + local label="$1" jar="$2" port="$3" + shift 3 + local log="/tmp/otel-proof-${label}.log" + rm -f "$log" + java -cp env-var-proof/stub-receiver/out StubOtlpReceiver "$port" "$log" & + local receiver_pid=$! + sleep 1 + echo "=== $label ===" + env "$@" timeout 30 java -jar "$jar" | grep -E "Publishing metrics" || true + kill "$receiver_pid" 2>/dev/null || true + echo "--- receiver log ($label) ---" + cat "$log" +} + +run_case "boot41-env-vars" "env-var-proof/boot41/target/otel-env-proof-41-1.0.0.jar" 24318 \ + OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:24318" OTEL_METRIC_EXPORT_INTERVAL=2000 +run_case "boot40-env-vars" "env-var-proof/boot40/target/otel-env-proof-40-1.0.0.jar" 24319 \ + OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:24319" OTEL_METRIC_EXPORT_INTERVAL=2000 +run_case "boot40-management-props" "env-var-proof/boot40/target/otel-env-proof-40-1.0.0.jar" 24320 \ + MANAGEMENT_OTLP_METRICS_EXPORT_URL="http://localhost:24320/v1/metrics" MANAGEMENT_OTLP_METRICS_EXPORT_STEP=2s + +echo "Compare the three receiver logs above against docs/output/04-otel-env-vars-4.0-vs-4.1.txt" diff --git a/observability/scripts/run.sh b/observability/scripts/run.sh new file mode 100755 index 0000000..548098e --- /dev/null +++ b/observability/scripts/run.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Starts the app on :8080. Boot's Docker Compose support auto-starts compose.yaml's +# grafana/otel-lgtm container the first time this runs (needs Docker) and wires the OTLP +# metrics/traces/logs exporters to it -- no management.otlp.* properties needed. +# ./scripts/run.sh # default 0.10 trace sampling +# ./scripts/run.sh fulltrace # 1.0 trace sampling -- see docs/03-otel-env-vars.md +set -euo pipefail +cd "$(dirname "$0")/.." +PROFILE="${1:-}" +if [ -n "$PROFILE" ]; then + mvn -q -Dspring-boot.run.profiles="$PROFILE" spring-boot:run +else + mvn -q spring-boot:run +fi diff --git a/observability/src/main/java/com/ankurm/observability/ObservabilityApplication.java b/observability/src/main/java/com/ankurm/observability/ObservabilityApplication.java new file mode 100644 index 0000000..4cf49d1 --- /dev/null +++ b/observability/src/main/java/com/ankurm/observability/ObservabilityApplication.java @@ -0,0 +1,16 @@ +package com.ankurm.observability; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +/** + * Companion app for docs/00-migrating-to-opentelemetry-starter.md. Run with `scripts/run.sh` -- + * Boot's Docker Compose support (compose.yaml, grafana/otel-lgtm) auto-wires the OTLP metrics, + * traces and logs exporters with zero management.otlp.* properties in application.yml. + */ +@SpringBootApplication +public class ObservabilityApplication { + public static void main(String[] args) { + SpringApplication.run(ObservabilityApplication.class, args); + } +} diff --git a/observability/src/main/java/com/ankurm/observability/ObservationConfig.java b/observability/src/main/java/com/ankurm/observability/ObservationConfig.java new file mode 100644 index 0000000..1a26a14 --- /dev/null +++ b/observability/src/main/java/com/ankurm/observability/ObservationConfig.java @@ -0,0 +1,22 @@ +package com.ankurm.observability; + +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.observation.aop.ObservedAspect; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +/** + * Without this bean, @Observed is inert: Spring Boot's autoconfiguration wires up an + * ObservationRegistry, but it does NOT register an ObservedAspect for you, even with + * spring-boot-starter-aspectj (AspectJ weaver) on the classpath and Micrometer Tracing active. + * See docs/output/03-observed-needs-explicit-bean.txt for the trace showing only the HTTP server + * span before this bean existed, and docs/02-observation-api.md for the correction. + */ +@Configuration(proxyBeanMethods = false) +class ObservationConfig { + + @Bean + ObservedAspect observedAspect(ObservationRegistry registry) { + return new ObservedAspect(registry); + } +} diff --git a/observability/src/main/java/com/ankurm/observability/OrderController.java b/observability/src/main/java/com/ankurm/observability/OrderController.java new file mode 100644 index 0000000..e49f298 --- /dev/null +++ b/observability/src/main/java/com/ankurm/observability/OrderController.java @@ -0,0 +1,39 @@ +package com.ankurm.observability; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.observation.annotation.Observed; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RestController; + +import java.util.Map; +import java.util.concurrent.ThreadLocalRandom; + +/** + * Real traffic source for docs/output/01-metrics-and-traces-in-lgtm.txt. Every call increments + * a Micrometer counter (exported as an OTLP metric) and, via @Observed, opens a trace span -- + * one recording, two signals, exactly as described in + * https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/observability/docs/02-observation-api.md + */ +@RestController +class OrderController { + + private final Counter ordersPlaced; + + OrderController(MeterRegistry registry) { + this.ordersPlaced = Counter.builder("orders.placed") + .description("Number of orders placed") + .tag("channel", "web") + .register(registry); + } + + @Observed(name = "place-order", contextualName = "order-controller#placeOrder") + @PostMapping("/orders/{id}") + Map placeOrder(@PathVariable String id) throws InterruptedException { + ordersPlaced.increment(); + // Simulate a bit of real work so the trace span has non-zero duration. + Thread.sleep(ThreadLocalRandom.current().nextInt(5, 40)); + return Map.of("id", id, "status", "placed"); + } +} diff --git a/observability/src/main/resources/application.yml b/observability/src/main/resources/application.yml new file mode 100644 index 0000000..6387b3c --- /dev/null +++ b/observability/src/main/resources/application.yml @@ -0,0 +1,31 @@ +server: + port: 8080 + +spring: + application: + name: order-service + +management: + endpoints: + web: + exposure: + include: health + metrics: + tags: + application: ${spring.application.name} + # No management.otlp.* export properties here on purpose -- see docs/03-otel-env-vars.md. + # In development, Boot's Docker Compose support (see compose.yaml) auto-configures the OTLP + # export endpoints for metrics, traces and logs the moment it detects the grafana/otel-lgtm + # image, without a single management.otlp.* property in this file. + # management.tracing.sampling.probability defaults to 0.10 -- see docs/output/02-low-sampling-demo.txt + # for what that does to a short burst of local traffic, and the "fulltrace" profile below for the fix. + +--- +spring: + config: + activate: + on-profile: fulltrace +management: + tracing: + sampling: + probability: 1.0 diff --git a/observability/src/test/java/com/ankurm/observability/ObservabilityApplicationTests.java b/observability/src/test/java/com/ankurm/observability/ObservabilityApplicationTests.java new file mode 100644 index 0000000..7fc1c2a --- /dev/null +++ b/observability/src/test/java/com/ankurm/observability/ObservabilityApplicationTests.java @@ -0,0 +1,20 @@ +package com.ankurm.observability; + +import org.junit.jupiter.api.Test; +import org.springframework.boot.test.context.SpringBootTest; + +/** + * Docker Compose support is off by default in tests (Boot skips it under `mvn test`), so this + * just confirms the application context -- including the OpenTelemetry starter's autoconfiguration + * and the explicit ObservedAspect bean in ObservationConfig -- wires up cleanly without a running + * collector. The real, docker-backed proof (metrics and traces actually landing in a live Grafana + * LGTM stack) is in scripts/run-all.sh and docs/output/, which need Docker and take real wall time + * -- not something to run on every `mvn test`. + */ +@SpringBootTest +class ObservabilityApplicationTests { + + @Test + void contextLoads() { + } +}