[← 05 Custom health indicators](05-custom-health-indicators.md) · **06 · Health indicator failure modes** · [07 Groups and probes →](07-groups-and-probes.md)

# 06 — Health indicator failure modes

## The 60-second Kafka health check

This is the most useful number in the repository. The textbook Kafka health check:

```java
AdminClient admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"));
admin.describeCluster().nodes().get();
```

Against an unreachable broker, from
[`output/09-kafka-timeout.txt.naive`](output/09-kafka-timeout.txt.naive):

```json
{"details":{"tuned":false,"error":"TimeoutException: Timed out waiting for a node assignment. Call: listNodes","probeMs":60002,"budgetMs":180000},"status":"DOWN"}

wall clock: 60.2 s
```

**60.002 seconds.** The outer budget was 180 s, so it was not the binding constraint — Kafka's
own `default.api.timeout.ms` was. These are the `AdminClientConfig` defaults, read from
`AdminClientConfig.configDef()` on kafka-clients 4.2.1:

| Property | Default |
|---|---|
| `request.timeout.ms` | 30000 |
| `default.api.timeout.ms` | **60000** |
| `socket.connection.setup.timeout.ms` | 10000 |
| `retries` | 2147483647 |
| `metadata.recovery.strategy` | `rebootstrap` |

The trap is that `request.timeout.ms` is the one everybody sets. It bounds a single attempt, and
with `retries` at `Integer.MAX_VALUE` the client simply attempts again until
`default.api.timeout.ms` fires. Set only `request.timeout.ms` and your health check still blocks
for a minute.

The tuned version, same broker, same absence of it —
[`output/09-kafka-timeout.txt.tuned`](output/09-kafka-timeout.txt.tuned):

```json
{"details":{"tuned":true,"error":"TimeoutException: null","probeMs":1500,"budgetMs":1500},"status":"DOWN"}

wall clock: 1.6 s
```

Forty times faster, from setting four properties and one `KafkaFuture.get(timeout)`.

Why this matters beyond the number: your probe interval is probably 10 seconds. A check that
takes 60 seconds means six probes are in flight at once, each holding a container worker thread,
for as long as the broker is down. That is how a Kafka outage becomes an application outage.

## The rebootstrap storm

Kafka 4 defaults `metadata.recovery.strategy` to `rebootstrap`. A long-lived `AdminClient`
pointed at a dead broker keeps a background thread retrying, and it logs. The first run of this
repository produced **192** `Rebootstrapping with Cluster(id = null, ...)` lines in a few
seconds, and the sandbox ran out of memory shortly after.

`metadata.recovery.strategy: none` plus a sane `reconnect.backoff.max.ms` stops it. Whatever you
choose, know that the AdminClient's background thread is doing work whether or not anyone is
calling your health endpoint.

## The unbounded HTTP call

`ExternalApiHealthIndicator` sets connect and read timeouts of 750 ms. Against a stub that
sleeps 30 seconds — [`output/10-slow-upstream.txt`](output/10-slow-upstream.txt):

```json
{"details":{"error":"ResourceAccessException: I/O error on GET request ...: Read timed out","latencyMs":753,"timeoutMs":750},"status":"DOWN"}

wall clock: 0.84 s
```

`latencyMs: 753` against `timeoutMs: 750`. Remove `setReadTimeout` and that becomes 30 seconds,
holding a Tomcat worker the entire time.

The default for `SimpleClientHttpRequestFactory` — and for `HttpURLConnection` underneath it —
is no timeout at all. Not a long timeout. None.

## Rules of thumb

- **Every remote call in a health indicator needs an explicit timeout**, and the total of all of
  them should be comfortably under your probe interval.
- **Set the API-level timeout, not just the request-level one.** Kafka is the sharpest example
  but the pattern recurs — clients with internal retry loops need an outer bound.
- **Fail fast and fail loudly.** A `DOWN` with a useful `error` detail beats a probe that hangs.
- **Health checks should be cheap.** They run on every instance on every probe interval, forever.
- **Do not check dependencies you cannot act on.** If a third party being down does not change
  what this instance should do, do not put it in a health indicator at all.

There is a cache if you need it: `management.endpoint.health.cache.time-to-live`. Reach for it
only after you have fixed the timeouts — caching a 60-second check gives you a fast endpoint
serving a stale answer, which is worse than a slow honest one.

---

[← 05](05-custom-health-indicators.md) · **06** · [07 Groups and probes →](07-groups-and-probes.md)