Ankur Mhatre
4b6cefa60a
Companion repository for the ankurm.com article. Every transcript in docs/output/ was produced by running this project; scripts/run-all.sh regenerates all of them. Verified against Spring Boot 4.1.1 / Framework 7.0.9 / Security 7.1.1 / Micrometer 1.17.1 / kafka-clients 4.2.1 on Temurin JDK 25.0.4.1+1.
131 lines
4.7 KiB
Markdown
131 lines
4.7 KiB
Markdown
[← 06 Failure modes](06-health-indicator-failure-modes.md) · **07 · Groups, probes and Kubernetes** · [08 The diagnostics endpoint →](08-diagnostics.md)
|
|
|
|
# 07 — Groups, probes and Kubernetes
|
|
|
|
## The failure this prevents
|
|
|
|
A third-party API goes down. Your `externalApi` indicator reports `DOWN`. `/actuator/health`
|
|
aggregates to `DOWN` and answers 503. Your Kubernetes manifest points **both** probes at it:
|
|
|
|
```yaml
|
|
livenessProbe: { httpGet: { path: /actuator/health, port: 8080 } }
|
|
readinessProbe: { httpGet: { path: /actuator/health, port: 8080 } }
|
|
```
|
|
|
|
The readiness failure is correct — this instance cannot serve. The liveness failure is a
|
|
catastrophe: the kubelet kills the container, every replica fails the same probe at the same
|
|
time, and the whole deployment enters a restart loop. A partial outage in someone else's system
|
|
has become a total outage in yours, and the restarts make recovery slower because every fresh
|
|
JVM has to warm up while the upstream is still failing.
|
|
|
|
**Liveness answers "is this process broken beyond recovery?"** Almost nothing external belongs
|
|
in it, because restarting cannot fix an external problem.
|
|
|
|
**Readiness answers "should this instance receive traffic right now?"** Dependencies belong here.
|
|
|
|
## The configuration
|
|
|
|
From `application-groups.yaml`:
|
|
|
|
```yaml
|
|
management:
|
|
endpoint:
|
|
health:
|
|
group:
|
|
liveness:
|
|
include: livenessState,diskSpace
|
|
readiness:
|
|
include: readinessState,ordersDatabase,externalApi
|
|
startup:
|
|
include: ordersDatabase
|
|
probes:
|
|
enabled: true
|
|
```
|
|
|
|
Each group gets its own path: `/actuator/health/liveness`, `/actuator/health/readiness`,
|
|
`/actuator/health/startup`. Groups can carry their own `show-details` too.
|
|
|
|
## The evidence
|
|
|
|
[`output/08-groups-and-probes.txt`](output/08-groups-and-probes.txt):
|
|
|
|
```
|
|
--- baseline: upstream UP ---
|
|
GET /actuator/health HTTP 503
|
|
GET /actuator/health/liveness HTTP 200
|
|
GET /actuator/health/readiness HTTP 200
|
|
GET /actuator/health/startup HTTP 200
|
|
|
|
--- upstream DOWN ---
|
|
GET /actuator/health HTTP 503
|
|
GET /actuator/health/liveness HTTP 200
|
|
GET /actuator/health/readiness HTTP 503
|
|
GET /actuator/health/startup HTTP 200
|
|
```
|
|
|
|
Liveness stayed 200 across the outage. Readiness moved. Kubernetes removes this instance from
|
|
the Service endpoints and leaves the process alone; when the upstream recovers, readiness goes
|
|
green and traffic returns without a single restart.
|
|
|
|
Note the aggregate `/actuator/health` was 503 in *both* columns — the Kafka indicator is down
|
|
throughout, since this repository runs no broker. That is exactly why you should not point a
|
|
probe at the aggregate: it is the union of everything, and it tells the orchestrator nothing
|
|
actionable.
|
|
|
|
## The manifest
|
|
|
|
```yaml
|
|
livenessProbe:
|
|
httpGet: { path: /actuator/health/liveness, port: 9001 }
|
|
periodSeconds: 10
|
|
failureThreshold: 3
|
|
readinessProbe:
|
|
httpGet: { path: /actuator/health/readiness, port: 9001 }
|
|
periodSeconds: 5
|
|
failureThreshold: 2
|
|
startupProbe:
|
|
httpGet: { path: /actuator/health/startup, port: 9001 }
|
|
periodSeconds: 5
|
|
failureThreshold: 30
|
|
```
|
|
|
|
Port 9001 is the management port from [chapter 04](04-securing-actuator.md). The kubelet reaches
|
|
it inside the pod network; the ingress does not.
|
|
|
|
The `startupProbe` matters more than people think. Without one, a JVM that takes 45 seconds to
|
|
warm up under `failureThreshold: 3` and `periodSeconds: 10` gets killed at 30 seconds, forever,
|
|
and the symptom is a crash-loop with no error in the logs.
|
|
|
|
## `readinessState`, and shutting down cleanly
|
|
|
|
`readinessState` is Boot's own indicator, driven by `ApplicationAvailability`. During graceful
|
|
shutdown Boot flips it to `REFUSING_TRAFFIC` before the server stops accepting, so readiness
|
|
goes red while in-flight requests finish. Pair it with:
|
|
|
|
```yaml
|
|
server:
|
|
shutdown: graceful
|
|
spring:
|
|
lifecycle:
|
|
timeout-per-shutdown-phase: 30s
|
|
```
|
|
|
|
You can also drive it yourself by publishing an `AvailabilityChangeEvent` — useful for taking an
|
|
instance out of rotation before a risky migration.
|
|
|
|
## Which dependencies belong in readiness
|
|
|
|
Ask: *if this is down, can this instance still serve any useful request?*
|
|
|
|
- Primary database → yes, readiness.
|
|
- Kafka, for a service whose only job is consuming → yes, readiness.
|
|
- Kafka, for a service that also serves reads → probably not; degrade rather than withdraw.
|
|
- A recommendations API you fall back to a static list for → no. Do not check it at all.
|
|
|
|
Every dependency in readiness is a dependency that can take your service out of rotation. That
|
|
list should be shorter than your instinct suggests.
|
|
|
|
---
|
|
|
|
[← 06](06-health-indicator-failure-modes.md) · **07** · [08 The diagnostics endpoint →](08-diagnostics.md)
|