# 1. The lab

[Index](../README.md) · Next: [2. Probes →](02-probes.md)

Every transcript in [`output/`](output) came from a single-node **k3s v1.36.4+k3s1** cluster on a
2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and
on a 64-core node it is much harder to see.

## Reproducing it

Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:

- namespace `demo` ([`k8s/00-namespace.yaml`](../k8s/00-namespace.yaml))
- the image `sbd/k8s-demo:1` loaded into the cluster's container runtime, pulled with
  `imagePullPolicy: Never`. Build it with `mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .`,
  then `k3s ctr -n k8s.io images import`, `kind load docker-image`, or `minikube image load`
- `busybox:1.37` available the same way (the `downstream` service and the `client` pod)
- for [chapter 6](06-hpa-custom-metrics.md): `quay.io/prometheus/prometheus:v3.14.0` and
  `registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0`

The image is the layered, distroless one recommended in the
[Docker article](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/):
no shell, which matters in [chapter 5](05-graceful-shutdown.md).

## Two things specific to this lab

**The node runs cgroup v1.** Kubernetes 1.35 changed the kubelet's `failCgroupV1` default to `true`
(KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes
`--kubelet-arg=fail-cgroupv1=false`. Production nodes should be cgroup v2; [chapter 3](03-jvm-ergonomics.md)
lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.

**The VM's process had no `CAP_SYS_RESOURCE`**, so runc could not lower `oom_score_adj` for the pod
sandboxes and every pod failed with `failed to update /proc/self/oom_score_adj: Permission denied`.
containerd's `restrict_oom_score_adj = true` (a drop-in under
`/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/`) fixes it - the same setting rootless
setups use.

**The kubelet garbage-collected images it was about to need.** The VM's disk is shared with its
host and reported 94 % used, above the kubelet's default `image-gc-high-threshold` of 85 %. The
kubelet deleted every image no running container used - the Prometheus and adapter images among
them - and their pods failed with `ErrImageNeverPull`. With `imagePullPolicy: Never` there is no
registry to fall back on. The lab now passes `image-gc-high-threshold=100`,
`image-gc-low-threshold=99` and a 1 GiB `eviction-hard` threshold; on a real node, fix the disk.

And one mistake worth not repeating: starting k3s from a shell with `HTTPS_PROXY` set makes the API
server route its own connections to the kubelet (`kubectl logs`, `exec`) through that proxy, where
they fail. Start it with the proxy variables unset.