[← 06 Failure modes](06-health-indicator-failure-modes.md) · **07 · Groups, probes and Kubernetes** · [08 The diagnostics endpoint →](08-diagnostics.md)

# 07 — Groups, probes and Kubernetes

## The failure this prevents

A third-party API goes down. Your `externalApi` indicator reports `DOWN`. `/actuator/health`
aggregates to `DOWN` and answers 503. Your Kubernetes manifest points **both** probes at it:

```yaml
livenessProbe:  { httpGet: { path: /actuator/health, port: 8080 } }
readinessProbe: { httpGet: { path: /actuator/health, port: 8080 } }
```

The readiness failure is correct — this instance cannot serve. The liveness failure is a
catastrophe: the kubelet kills the container, every replica fails the same probe at the same
time, and the whole deployment enters a restart loop. A partial outage in someone else's system
has become a total outage in yours, and the restarts make recovery slower because every fresh
JVM has to warm up while the upstream is still failing.

**Liveness answers "is this process broken beyond recovery?"** Almost nothing external belongs
in it, because restarting cannot fix an external problem.

**Readiness answers "should this instance receive traffic right now?"** Dependencies belong here.

## The configuration

From `application-groups.yaml`:

```yaml
management:
  endpoint:
    health:
      group:
        liveness:
          include: livenessState,diskSpace
        readiness:
          include: readinessState,ordersDatabase,externalApi
        startup:
          include: ordersDatabase
      probes:
        enabled: true
```

Each group gets its own path: `/actuator/health/liveness`, `/actuator/health/readiness`,
`/actuator/health/startup`. Groups can carry their own `show-details` too.

## The evidence

[`output/08-groups-and-probes.txt`](output/08-groups-and-probes.txt):

```
--- baseline: upstream UP ---
  GET /actuator/health              HTTP 503
  GET /actuator/health/liveness     HTTP 200
  GET /actuator/health/readiness    HTTP 200
  GET /actuator/health/startup      HTTP 200

--- upstream DOWN ---
  GET /actuator/health              HTTP 503
  GET /actuator/health/liveness     HTTP 200
  GET /actuator/health/readiness    HTTP 503
  GET /actuator/health/startup      HTTP 200
```

Liveness stayed 200 across the outage. Readiness moved. Kubernetes removes this instance from
the Service endpoints and leaves the process alone; when the upstream recovers, readiness goes
green and traffic returns without a single restart.

Note the aggregate `/actuator/health` was 503 in *both* columns — the Kafka indicator is down
throughout, since this repository runs no broker. That is exactly why you should not point a
probe at the aggregate: it is the union of everything, and it tells the orchestrator nothing
actionable.

## The manifest

```yaml
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 9001 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 9001 }
  periodSeconds: 5
  failureThreshold: 2
startupProbe:
  httpGet: { path: /actuator/health/startup, port: 9001 }
  periodSeconds: 5
  failureThreshold: 30
```

Port 9001 is the management port from [chapter 04](04-securing-actuator.md). The kubelet reaches
it inside the pod network; the ingress does not.

The `startupProbe` matters more than people think. Without one, a JVM that takes 45 seconds to
warm up under `failureThreshold: 3` and `periodSeconds: 10` gets killed at 30 seconds, forever,
and the symptom is a crash-loop with no error in the logs.

## `readinessState`, and shutting down cleanly

`readinessState` is Boot's own indicator, driven by `ApplicationAvailability`. During graceful
shutdown Boot flips it to `REFUSING_TRAFFIC` before the server stops accepting, so readiness
goes red while in-flight requests finish. Pair it with:

```yaml
server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
```

You can also drive it yourself by publishing an `AvailabilityChangeEvent` — useful for taking an
instance out of rotation before a risky migration.

## Which dependencies belong in readiness

Ask: *if this is down, can this instance still serve any useful request?*

- Primary database → yes, readiness.
- Kafka, for a service whose only job is consuming → yes, readiness.
- Kafka, for a service that also serves reads → probably not; degrade rather than withdraw.
- A recommendations API you fall back to a static list for → no. Do not check it at all.

Every dependency in readiness is a dependency that can take your service out of rotation. That
list should be shorter than your instinct suggests.

---

[← 06](06-health-indicator-failure-modes.md) · **07** · [08 The diagnostics endpoint →](08-diagnostics.md)