Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA

Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
2026-09-11 17:12:10 +00:00
parent 644da9e65e
commit a065696478
72 changed files with 28415 additions and 2 deletions
@@ -0,0 +1,47 @@
+# 1. The lab
+
+[Index](../README.md) · Next: [2. Probes →](02-probes.md)
+
+Every transcript in [`output/`](output) came from a single-node **k3s v1.36.4+k3s1** cluster on a
+2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and
+on a 64-core node it is much harder to see.
+
+## Reproducing it
+
+Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:
+
+- namespace `demo` ([`k8s/00-namespace.yaml`](../k8s/00-namespace.yaml))
+- the image `sbd/k8s-demo:1` loaded into the cluster's container runtime, pulled with
+  `imagePullPolicy: Never`. Build it with `mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .`,
+  then `k3s ctr -n k8s.io images import`, `kind load docker-image`, or `minikube image load`
+- `busybox:1.37` available the same way (the `downstream` service and the `client` pod)
+- for [chapter 6](06-hpa-custom-metrics.md): `quay.io/prometheus/prometheus:v3.14.0` and
+  `registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0`
+
+The image is the layered, distroless one recommended in the
+[Docker article](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/):
+no shell, which matters in [chapter 5](05-graceful-shutdown.md).
+
+## Two things specific to this lab
+
+**The node runs cgroup v1.** Kubernetes 1.35 changed the kubelet's `failCgroupV1` default to `true`
+(KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes
+`--kubelet-arg=fail-cgroupv1=false`. Production nodes should be cgroup v2; [chapter 3](03-jvm-ergonomics.md)
+lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.
+
+**The VM's process had no `CAP_SYS_RESOURCE`**, so runc could not lower `oom_score_adj` for the pod
+sandboxes and every pod failed with `failed to update /proc/self/oom_score_adj: Permission denied`.
+containerd's `restrict_oom_score_adj = true` (a drop-in under
+`/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/`) fixes it - the same setting rootless
+setups use.
+
+**The kubelet garbage-collected images it was about to need.** The VM's disk is shared with its
+host and reported 94 % used, above the kubelet's default `image-gc-high-threshold` of 85 %. The
+kubelet deleted every image no running container used - the Prometheus and adapter images among
+them - and their pods failed with `ErrImageNeverPull`. With `imagePullPolicy: Never` there is no
+registry to fall back on. The lab now passes `image-gc-high-threshold=100`,
+`image-gc-low-threshold=99` and a 1 GiB `eviction-hard` threshold; on a real node, fix the disk.
+
+And one mistake worth not repeating: starting k3s from a shell with `HTTPS_PROXY` set makes the API
+server route its own connections to the kubelet (`kubectl logs`, `exec`) through that proxy, where
+they fail. Start it with the proxy variables unset.