asmhatreandClaude Opus 5
a065696478
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
48 lines
2.8 KiB
Markdown
48 lines
2.8 KiB
Markdown
# 1. The lab
|
|
|
|
[Index](../README.md) · Next: [2. Probes →](02-probes.md)
|
|
|
|
Every transcript in [`output/`](output) came from a single-node **k3s v1.36.4+k3s1** cluster on a
|
|
2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and
|
|
on a 64-core node it is much harder to see.
|
|
|
|
## Reproducing it
|
|
|
|
Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:
|
|
|
|
- namespace `demo` ([`k8s/00-namespace.yaml`](../k8s/00-namespace.yaml))
|
|
- the image `sbd/k8s-demo:1` loaded into the cluster's container runtime, pulled with
|
|
`imagePullPolicy: Never`. Build it with `mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .`,
|
|
then `k3s ctr -n k8s.io images import`, `kind load docker-image`, or `minikube image load`
|
|
- `busybox:1.37` available the same way (the `downstream` service and the `client` pod)
|
|
- for [chapter 6](06-hpa-custom-metrics.md): `quay.io/prometheus/prometheus:v3.14.0` and
|
|
`registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0`
|
|
|
|
The image is the layered, distroless one recommended in the
|
|
[Docker article](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/):
|
|
no shell, which matters in [chapter 5](05-graceful-shutdown.md).
|
|
|
|
## Two things specific to this lab
|
|
|
|
**The node runs cgroup v1.** Kubernetes 1.35 changed the kubelet's `failCgroupV1` default to `true`
|
|
(KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes
|
|
`--kubelet-arg=fail-cgroupv1=false`. Production nodes should be cgroup v2; [chapter 3](03-jvm-ergonomics.md)
|
|
lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.
|
|
|
|
**The VM's process had no `CAP_SYS_RESOURCE`**, so runc could not lower `oom_score_adj` for the pod
|
|
sandboxes and every pod failed with `failed to update /proc/self/oom_score_adj: Permission denied`.
|
|
containerd's `restrict_oom_score_adj = true` (a drop-in under
|
|
`/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/`) fixes it - the same setting rootless
|
|
setups use.
|
|
|
|
**The kubelet garbage-collected images it was about to need.** The VM's disk is shared with its
|
|
host and reported 94 % used, above the kubelet's default `image-gc-high-threshold` of 85 %. The
|
|
kubelet deleted every image no running container used - the Prometheus and adapter images among
|
|
them - and their pods failed with `ErrImageNeverPull`. With `imagePullPolicy: Never` there is no
|
|
registry to fall back on. The lab now passes `image-gc-high-threshold=100`,
|
|
`image-gc-low-threshold=99` and a 1 GiB `eviction-hard` threshold; on a real node, fix the disk.
|
|
|
|
And one mistake worth not repeating: starting k3s from a shell with `HTTPS_PROXY` set makes the API
|
|
server route its own connections to the kubelet (`kubectl logs`, `exec`) through that proxy, where
|
|
they fail. Start it with the proxy variables unset.
|