# observability

Companion module for the ankurm.com post [**"A Practical Guide to Monitoring Spring Boot
Microservices: Prometheus, Grafana, and Boot 4.1's OpenTelemetry Starter"**](https://ankurm.com/a-practical-guide-to-monitoring-spring-boot-microservices-with-prometheus-grafana/)
(the original 2025 Prometheus + Grafana walkthrough, rewritten around
`spring-boot-starter-opentelemetry`). Every metric and trace quoted in the post came out of a
real `grafana/otel-lgtm` container, driven by real HTTP traffic against a real running app --
not a diagram, not an invented log line.

For the deep, start-to-finish Micrometer/OpenTelemetry treatment -- what `spring-boot-starter-opentelemetry`
actually changes, the Observation API, cardinality, context propagation -- see
[**Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide**](https://ankurm.com/micrometer-opentelemetry-spring-boot-4-observability-guide/),
already on this blog. This module exists to *run* a handful of that guide's claims against real
infrastructure rather than just restate them, and it found two gaps along the way: `@Observed`
needs a bean the guide doesn't mention, and the OTLP endpoint env var it demonstrates was not, in
fact, new in Boot 4.1 -- only the rest of the standard variable surface was.

Lives in this container repo (not as its own top-level repository) alongside
[`../actuator-in-production`](../actuator-in-production), the companion module for Boot 4's
Actuator endpoints generally.

## Versions (verified against `repo1.maven.org` maven-metadata.xml and primary-source release notes)

| Component | Version | Notes |
|---|---|---|
| JDK | 25 (Temurin 25.0.4.1+1) | latest LTS |
| Spring Boot | 4.1.1 | latest GA at time of writing |
| Spring Framework | 7.0.9 | latest GA |
| `grafana/otel-lgtm` | latest (LGTM_VERSION v0.33.0 at time of writing) | Loki + Grafana + Tempo + Prometheus-compatible backend + OTLP collector, one image |

`env-var-proof/` additionally builds against `spring-boot-starter-parent` 4.0.8, on purpose --
see [docs/03-otel-env-vars.md](docs/03-otel-env-vars.md).

## Quickstart

```bash
./scripts/run.sh              # starts on :8080; auto-starts grafana/otel-lgtm via Docker Compose
./scripts/run.sh fulltrace    # same, with 100% trace sampling instead of the 10% default
./scripts/run-all.sh          # regenerates every file in docs/output/ from a real run (Docker, ~6 min)
./scripts/run-env-var-proof.sh   # the Boot 4.0 vs 4.1 OTEL_* env var experiment on its own

curl -X POST localhost:8080/orders/1
open http://localhost:3000          # Grafana, admin/admin
curl "http://localhost:9090/api/v1/query?query=orders_placed_total"
curl "http://localhost:3200/api/search?limit=10"
```

Requires JDK 25, Maven, and Docker with network access (to pull `grafana/otel-lgtm`, roughly
1.5GB). First run must be online for Maven too.

## What's demonstrated where

| Area | Source | Transcript |
|---|---|---|
| Docker Compose auto-wires the OTLP endpoint with zero `management.otlp.*` properties | [`compose.yaml`](compose.yaml), [`application.yml`](src/main/resources/application.yml) | [`00`](docs/output/00-docker-compose-auto-wiring.txt) |
| Real metrics, real requests, queried back from LGTM's own Prometheus-compatible API | [`OrderController`](src/main/java/com/ankurm/observability/OrderController.java) | [`01`](docs/output/01-metrics-and-traces-in-lgtm.txt) |
| Default 0.10 trace sampling vs `fulltrace` profile (1.0) | [`application.yml`](src/main/resources/application.yml) | [`02`](docs/output/02-low-sampling-demo.txt) |
| `@Observed` is inert without an explicit `ObservedAspect` bean | [`ObservationConfig`](src/main/java/com/ankurm/observability/ObservationConfig.java) | [`03`](docs/output/03-observed-needs-explicit-bean.txt) |
| `OTEL_*` env vars: what's genuinely new in Boot 4.1 vs what already worked on 4.0 | [`env-var-proof/`](env-var-proof) | [`04`](docs/output/04-otel-env-vars-4.0-vs-4.1.txt) |

## Documentation chapters

1. [Migrating to spring-boot-starter-opentelemetry](docs/01-migrating-to-opentelemetry-starter.md) -- what replaces the old Prometheus-registry-plus-scrape-config setup, and Docker Compose auto-wiring proven with a real container
2. [The Observation API and @Observed](docs/02-observation-api.md) -- the missing `ObservedAspect` bean
3. [OTEL_* environment variables](docs/03-otel-env-vars.md) -- Boot 4.0 vs 4.1, dual-version proof, plus the trace-sampling gotcha it surfaced
4. [Production checklist](docs/04-production-checklist.md)

## License

MIT -- see [../LICENSE](../LICENSE).