asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity
Files
66208bcd97c27b4f323b253173c4f5c6380609d0
spring-boot-demo/kubernetes-deployment/docs/01-the-lab.md
T
asmhatreandClaude Opus 5 a065696478 Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA 
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful
Shutdown, Limits and JVM Ergonomics". A dependency outage under three
probe-group setups, a rolling restart under load four ways (three runs
each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load
under five CPU limits with throttling counters, and an HPA driven by a
Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
2026-09-11 17:12:10 +00:00

2.8 KiB
Raw Blame History

1. The lab

Index · Next: 2. Probes →

Every transcript in output/ came from a single-node k3s v1.36.4+k3s1 cluster on a 2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and on a 64-core node it is much harder to see.

Reproducing it

Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:

  • namespace demo (k8s/00-namespace.yaml)
  • the image sbd/k8s-demo:1 loaded into the cluster's container runtime, pulled with imagePullPolicy: Never. Build it with mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 ., then k3s ctr -n k8s.io images import, kind load docker-image, or minikube image load
  • busybox:1.37 available the same way (the downstream service and the client pod)
  • for chapter 6: quay.io/prometheus/prometheus:v3.14.0 and registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0

The image is the layered, distroless one recommended in the Docker article: no shell, which matters in chapter 5.

Two things specific to this lab

The node runs cgroup v1. Kubernetes 1.35 changed the kubelet's failCgroupV1 default to true (KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes --kubelet-arg=fail-cgroupv1=false. Production nodes should be cgroup v2; chapter 3 lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.

The VM's process had no CAP_SYS_RESOURCE, so runc could not lower oom_score_adj for the pod sandboxes and every pod failed with failed to update /proc/self/oom_score_adj: Permission denied. containerd's restrict_oom_score_adj = true (a drop-in under /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/) fixes it - the same setting rootless setups use.

The kubelet garbage-collected images it was about to need. The VM's disk is shared with its host and reported 94 % used, above the kubelet's default image-gc-high-threshold of 85 %. The kubelet deleted every image no running container used - the Prometheus and adapter images among them - and their pods failed with ErrImageNeverPull. With imagePullPolicy: Never there is no registry to fall back on. The lab now passes image-gc-high-threshold=100, image-gc-low-threshold=99 and a 1 GiB eviction-hard threshold; on a real node, fix the disk.

And one mistake worth not repeating: starting k3s from a shell with HTTPS_PROXY set makes the API server route its own connections to the kubelet (kubectl logs, exec) through that proxy, where they fail. Start it with the proxy variables unset.

View Git Blame Copy Permalink