Ankur Mhatre
4b6cefa60a
Companion repository for the ankurm.com article. Every transcript in docs/output/ was produced by running this project; scripts/run-all.sh regenerates all of them. Verified against Spring Boot 4.1.1 / Framework 7.0.9 / Security 7.1.1 / Micrometer 1.17.1 / kafka-clients 4.2.1 on Temurin JDK 25.0.4.1+1.
101 lines
4.4 KiB
Markdown
101 lines
4.4 KiB
Markdown
[← 05 Custom health indicators](05-custom-health-indicators.md) · **06 · Health indicator failure modes** · [07 Groups and probes →](07-groups-and-probes.md)
|
|
|
|
# 06 — Health indicator failure modes
|
|
|
|
## The 60-second Kafka health check
|
|
|
|
This is the most useful number in the repository. The textbook Kafka health check:
|
|
|
|
```java
|
|
AdminClient admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"));
|
|
admin.describeCluster().nodes().get();
|
|
```
|
|
|
|
Against an unreachable broker, from
|
|
[`output/09-kafka-timeout.txt.naive`](output/09-kafka-timeout.txt.naive):
|
|
|
|
```json
|
|
{"details":{"tuned":false,"error":"TimeoutException: Timed out waiting for a node assignment. Call: listNodes","probeMs":60002,"budgetMs":180000},"status":"DOWN"}
|
|
|
|
wall clock: 60.2 s
|
|
```
|
|
|
|
**60.002 seconds.** The outer budget was 180 s, so it was not the binding constraint — Kafka's
|
|
own `default.api.timeout.ms` was. These are the `AdminClientConfig` defaults, read from
|
|
`AdminClientConfig.configDef()` on kafka-clients 4.2.1:
|
|
|
|
| Property | Default |
|
|
|---|---|
|
|
| `request.timeout.ms` | 30000 |
|
|
| `default.api.timeout.ms` | **60000** |
|
|
| `socket.connection.setup.timeout.ms` | 10000 |
|
|
| `retries` | 2147483647 |
|
|
| `metadata.recovery.strategy` | `rebootstrap` |
|
|
|
|
The trap is that `request.timeout.ms` is the one everybody sets. It bounds a single attempt, and
|
|
with `retries` at `Integer.MAX_VALUE` the client simply attempts again until
|
|
`default.api.timeout.ms` fires. Set only `request.timeout.ms` and your health check still blocks
|
|
for a minute.
|
|
|
|
The tuned version, same broker, same absence of it —
|
|
[`output/09-kafka-timeout.txt.tuned`](output/09-kafka-timeout.txt.tuned):
|
|
|
|
```json
|
|
{"details":{"tuned":true,"error":"TimeoutException: null","probeMs":1500,"budgetMs":1500},"status":"DOWN"}
|
|
|
|
wall clock: 1.6 s
|
|
```
|
|
|
|
Forty times faster, from setting four properties and one `KafkaFuture.get(timeout)`.
|
|
|
|
Why this matters beyond the number: your probe interval is probably 10 seconds. A check that
|
|
takes 60 seconds means six probes are in flight at once, each holding a container worker thread,
|
|
for as long as the broker is down. That is how a Kafka outage becomes an application outage.
|
|
|
|
## The rebootstrap storm
|
|
|
|
Kafka 4 defaults `metadata.recovery.strategy` to `rebootstrap`. A long-lived `AdminClient`
|
|
pointed at a dead broker keeps a background thread retrying, and it logs. The first run of this
|
|
repository produced **192** `Rebootstrapping with Cluster(id = null, ...)` lines in a few
|
|
seconds, and the sandbox ran out of memory shortly after.
|
|
|
|
`metadata.recovery.strategy: none` plus a sane `reconnect.backoff.max.ms` stops it. Whatever you
|
|
choose, know that the AdminClient's background thread is doing work whether or not anyone is
|
|
calling your health endpoint.
|
|
|
|
## The unbounded HTTP call
|
|
|
|
`ExternalApiHealthIndicator` sets connect and read timeouts of 750 ms. Against a stub that
|
|
sleeps 30 seconds — [`output/10-slow-upstream.txt`](output/10-slow-upstream.txt):
|
|
|
|
```json
|
|
{"details":{"error":"ResourceAccessException: I/O error on GET request ...: Read timed out","latencyMs":753,"timeoutMs":750},"status":"DOWN"}
|
|
|
|
wall clock: 0.84 s
|
|
```
|
|
|
|
`latencyMs: 753` against `timeoutMs: 750`. Remove `setReadTimeout` and that becomes 30 seconds,
|
|
holding a Tomcat worker the entire time.
|
|
|
|
The default for `SimpleClientHttpRequestFactory` — and for `HttpURLConnection` underneath it —
|
|
is no timeout at all. Not a long timeout. None.
|
|
|
|
## Rules of thumb
|
|
|
|
- **Every remote call in a health indicator needs an explicit timeout**, and the total of all of
|
|
them should be comfortably under your probe interval.
|
|
- **Set the API-level timeout, not just the request-level one.** Kafka is the sharpest example
|
|
but the pattern recurs — clients with internal retry loops need an outer bound.
|
|
- **Fail fast and fail loudly.** A `DOWN` with a useful `error` detail beats a probe that hangs.
|
|
- **Health checks should be cheap.** They run on every instance on every probe interval, forever.
|
|
- **Do not check dependencies you cannot act on.** If a third party being down does not change
|
|
what this instance should do, do not put it in a health indicator at all.
|
|
|
|
There is a cache if you need it: `management.endpoint.health.cache.time-to-live`. Reach for it
|
|
only after you have fixed the timeouts — caching a 60-second check gives you a fast endpoint
|
|
serving a stale answer, which is worse than a slow honest one.
|
|
|
|
---
|
|
|
|
[← 05](05-custom-health-indicators.md) · **06** · [07 Groups and probes →](07-groups-and-probes.md)
|