Spring Boot 4 Actuator in production: endpoints, security, custom health indicators

Companion repository for the ankurm.com article. Every transcript in docs/output/
was produced by running this project; scripts/run-all.sh regenerates all of them.

Verified against Spring Boot 4.1.1 / Framework 7.0.9 / Security 7.1.1 /
Micrometer 1.17.1 / kafka-clients 4.2.1 on Temurin JDK 25.0.4.1+1.

docs/06-health-indicator-failure-modes.md

@@ -0,0 +1,100 @@
+[← 05 Custom health indicators](05-custom-health-indicators.md) · **06 · Health indicator failure modes** · [07 Groups and probes →](07-groups-and-probes.md)
+
+# 06 — Health indicator failure modes
+
+## The 60-second Kafka health check
+
+This is the most useful number in the repository. The textbook Kafka health check:
+
+```java
+AdminClient admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"));
+admin.describeCluster().nodes().get();
+```
+
+Against an unreachable broker, from
+[`output/09-kafka-timeout.txt.naive`](output/09-kafka-timeout.txt.naive):
+
+```json
+{"details":{"tuned":false,"error":"TimeoutException: Timed out waiting for a node assignment. Call: listNodes","probeMs":60002,"budgetMs":180000},"status":"DOWN"}
+
+wall clock: 60.2 s
+```
+
+**60.002 seconds.** The outer budget was 180 s, so it was not the binding constraint — Kafka's
+own `default.api.timeout.ms` was. These are the `AdminClientConfig` defaults, read from
+`AdminClientConfig.configDef()` on kafka-clients 4.2.1:
+
+| Property | Default |
+|---|---|
+| `request.timeout.ms` | 30000 |
+| `default.api.timeout.ms` | **60000** |
+| `socket.connection.setup.timeout.ms` | 10000 |
+| `retries` | 2147483647 |
+| `metadata.recovery.strategy` | `rebootstrap` |
+
+The trap is that `request.timeout.ms` is the one everybody sets. It bounds a single attempt, and
+with `retries` at `Integer.MAX_VALUE` the client simply attempts again until
+`default.api.timeout.ms` fires. Set only `request.timeout.ms` and your health check still blocks
+for a minute.
+
+The tuned version, same broker, same absence of it —
+[`output/09-kafka-timeout.txt.tuned`](output/09-kafka-timeout.txt.tuned):
+
+```json
+{"details":{"tuned":true,"error":"TimeoutException: null","probeMs":1500,"budgetMs":1500},"status":"DOWN"}
+
+wall clock: 1.6 s
+```
+
+Forty times faster, from setting four properties and one `KafkaFuture.get(timeout)`.
+
+Why this matters beyond the number: your probe interval is probably 10 seconds. A check that
+takes 60 seconds means six probes are in flight at once, each holding a container worker thread,
+for as long as the broker is down. That is how a Kafka outage becomes an application outage.
+
+## The rebootstrap storm
+
+Kafka 4 defaults `metadata.recovery.strategy` to `rebootstrap`. A long-lived `AdminClient`
+pointed at a dead broker keeps a background thread retrying, and it logs. The first run of this
+repository produced **192** `Rebootstrapping with Cluster(id = null, ...)` lines in a few
+seconds, and the sandbox ran out of memory shortly after.
+
+`metadata.recovery.strategy: none` plus a sane `reconnect.backoff.max.ms` stops it. Whatever you
+choose, know that the AdminClient's background thread is doing work whether or not anyone is
+calling your health endpoint.
+
+## The unbounded HTTP call
+
+`ExternalApiHealthIndicator` sets connect and read timeouts of 750 ms. Against a stub that
+sleeps 30 seconds — [`output/10-slow-upstream.txt`](output/10-slow-upstream.txt):
+
+```json
+{"details":{"error":"ResourceAccessException: I/O error on GET request ...: Read timed out","latencyMs":753,"timeoutMs":750},"status":"DOWN"}
+
+wall clock: 0.84 s
+```
+
+`latencyMs: 753` against `timeoutMs: 750`. Remove `setReadTimeout` and that becomes 30 seconds,
+holding a Tomcat worker the entire time.
+
+The default for `SimpleClientHttpRequestFactory` — and for `HttpURLConnection` underneath it —
+is no timeout at all. Not a long timeout. None.
+
+## Rules of thumb
+
+- **Every remote call in a health indicator needs an explicit timeout**, and the total of all of
+  them should be comfortably under your probe interval.
+- **Set the API-level timeout, not just the request-level one.** Kafka is the sharpest example
+  but the pattern recurs — clients with internal retry loops need an outer bound.
+- **Fail fast and fail loudly.** A `DOWN` with a useful `error` detail beats a probe that hangs.
+- **Health checks should be cheap.** They run on every instance on every probe interval, forever.
+- **Do not check dependencies you cannot act on.** If a third party being down does not change
+  what this instance should do, do not put it in a health indicator at all.
+
+There is a cache if you need it: `management.endpoint.health.cache.time-to-live`. Reach for it
+only after you have fixed the timeouts — caching a 60-second check gives you a fast endpoint
+serving a stale answer, which is worse than a slow honest one.
+
+---
+
+[← 05](05-custom-health-indicators.md) · **06** · [07 Groups and probes →](07-groups-and-probes.md)