spring-boot-demo/docs/06-health-indicator-failure-modes.md

← 05 Custom health indicators · 06 · Health indicator failure modes · 07 Groups and probes →

06 — Health indicator failure modes

The 60-second Kafka health check

This is the most useful number in the repository. The textbook Kafka health check:

AdminClient admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"));
admin.describeCluster().nodes().get();

Against an unreachable broker, from output/09-kafka-timeout.txt.naive:

{"details":{"tuned":false,"error":"TimeoutException: Timed out waiting for a node assignment. Call: listNodes","probeMs":60002,"budgetMs":180000},"status":"DOWN"}

wall clock: 60.2 s

60.002 seconds. The outer budget was 180 s, so it was not the binding constraint — Kafka's own default.api.timeout.ms was. These are the AdminClientConfig defaults, read from AdminClientConfig.configDef() on kafka-clients 4.2.1:

Property	Default
request.timeout.ms	30000
default.api.timeout.ms	60000
socket.connection.setup.timeout.ms	10000
retries	2147483647
metadata.recovery.strategy	rebootstrap

The trap is that request.timeout.ms is the one everybody sets. It bounds a single attempt, and with retries at Integer.MAX_VALUE the client simply attempts again until default.api.timeout.ms fires. Set only request.timeout.ms and your health check still blocks for a minute.

The tuned version, same broker, same absence of it — output/09-kafka-timeout.txt.tuned:

{"details":{"tuned":true,"error":"TimeoutException: null","probeMs":1500,"budgetMs":1500},"status":"DOWN"}

wall clock: 1.6 s

Forty times faster, from setting four properties and one KafkaFuture.get(timeout).

Why this matters beyond the number: your probe interval is probably 10 seconds. A check that takes 60 seconds means six probes are in flight at once, each holding a container worker thread, for as long as the broker is down. That is how a Kafka outage becomes an application outage.

The rebootstrap storm

Kafka 4 defaults metadata.recovery.strategy to rebootstrap. A long-lived AdminClient pointed at a dead broker keeps a background thread retrying, and it logs. The first run of this repository produced 192 Rebootstrapping with Cluster(id = null, ...) lines in a few seconds, and the sandbox ran out of memory shortly after.

metadata.recovery.strategy: none plus a sane reconnect.backoff.max.ms stops it. Whatever you choose, know that the AdminClient's background thread is doing work whether or not anyone is calling your health endpoint.

The unbounded HTTP call

ExternalApiHealthIndicator sets connect and read timeouts of 750 ms. Against a stub that sleeps 30 seconds — output/10-slow-upstream.txt:

{"details":{"error":"ResourceAccessException: I/O error on GET request ...: Read timed out","latencyMs":753,"timeoutMs":750},"status":"DOWN"}

wall clock: 0.84 s

latencyMs: 753 against timeoutMs: 750. Remove setReadTimeout and that becomes 30 seconds, holding a Tomcat worker the entire time.

The default for SimpleClientHttpRequestFactory — and for HttpURLConnection underneath it — is no timeout at all. Not a long timeout. None.

Rules of thumb

• Every remote call in a health indicator needs an explicit timeout, and the total of all of them should be comfortably under your probe interval.
• Set the API-level timeout, not just the request-level one. Kafka is the sharpest example but the pattern recurs — clients with internal retry loops need an outer bound.
• Fail fast and fail loudly. A DOWN with a useful error detail beats a probe that hangs.
• Health checks should be cheap. They run on every instance on every probe interval, forever.
• Do not check dependencies you cannot act on. If a third party being down does not change what this instance should do, do not put it in a health indicator at all.

There is a cache if you need it: management.endpoint.health.cache.time-to-live. Reach for it only after you have fixed the timeouts — caching a 60-second check gives you a fast endpoint serving a stale answer, which is worse than a slow honest one.

---

← 05 · 06 · 07 Groups and probes →