Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA

Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
2026-09-11 17:12:10 +00:00
parent 644da9e65e
commit a065696478
72 changed files with 28415 additions and 2 deletions
@@ -0,0 +1,74 @@
+# 6. HPA on a custom Micrometer metric
+
+[← 5. Graceful shutdown](05-graceful-shutdown.md) · [Index](../README.md)
+
+CPU is a poor scaling signal for a JVM service: startup and JIT compilation burn CPU while serving
+nothing, GC competes with requests for the same quota ([chapter 4](04-cpu-limits-and-gc.md)), and a
+service waiting on I/O can be saturated at 20 % CPU. What you usually want to scale on is work in
+progress. The demo scales on in-flight requests.
+
+## The pipeline
+
+1. **Micrometer.** [`WorkController`](../src/main/java/com/ankurm/k8s/web/WorkController.java)
+   registers a gauge `app.inflight.requests`. The Prometheus registry renames it: dots become
+   underscores, a gauge gets no suffix (a counter would get `_total`).
+2. **Prometheus** scrapes `/actuator/prometheus` every 5 s ([`k8s/hpa/prometheus.yaml`](../k8s/hpa/prometheus.yaml))
+   and attaches `namespace` and `pod` labels from Kubernetes service discovery.
+3. **prometheus-adapter** turns the series into the `custom.metrics.k8s.io` API with one rule
+   ([`k8s/hpa/prometheus-adapter.yaml`](../k8s/hpa/prometheus-adapter.yaml)). The
+   `resources.overrides` block is what maps the `pod` label to a Kubernetes Pod - without it the
+   metric exists in Prometheus and never appears in the API.
+4. **The HPA** ([`k8s/hpa/hpa.yaml`](../k8s/hpa/hpa.yaml)) targets an average of 5 per pod.
+
+What each end sees ([`hpa-custom-metrics-api.txt`](output/hpa-custom-metrics-api.txt)):
+
+```
+app_inflight_requests{application="orders"} 0.0
+```
+
+```
+            "metricName": "app_inflight_requests",
+            "timestamp": "2026-09-11T16:33:28Z",
+            "value": "0",
+```
+
+## The run
+
+2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms
+([`hpa-custom-metric.txt`](output/hpa-custom-metric.txt)):
+
+```
+t      clients    metric    desired  ready pods
+0s     2          0         1        1
+22s    30         333m      1        1
+32s    30         6333m     2        2
+43s    30         10333m    4        2
+54s    30         10333m    4        4
+64s    30         7475m     4        4
+148s   0          6416m     4        4
+169s   0          2499m     4        4
+179s   0          0         2        2
+200s   0          0         1        1
+```
+
+- **Reaction: ~12 s** from load to the first scale-up - a 5 s scrape, the adapter's query, and the
+  HPA controller's 15 s sync period.
+- **It overshoots on purpose.** At t=43 the metric was 10.3 against a target of 5 on 2 pods, so the
+  HPA asked for 5 and was capped at `maxReplicas: 4`. 30 clients over 4 pods settles at ~7.5 - still
+  above target, which is what a ceiling looks like.
+- **Scale-down took 40-60 s after the load stopped**, because the adapter rule averages over 30 s
+  and the HPA's scale-down stabilization window (shortened to 30 s here; **the default is 300 s**)
+  holds the highest recommendation it saw.
+- **7,209 requests, 0 failures** through two scale-downs - the `preStop` sleep from
+  [chapter 5](05-graceful-shutdown.md) doing its job.
+
+## Choices in the rule worth knowing about
+
+- `metricsQuery: avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s])` smooths a gauge that jumps
+  with every request. Without it the HPA chases noise; with too long a window it reacts late.
+- For a **counter** (say `http_server_requests_seconds_count`) the query must be a `rate(...)`, and
+  the adapter's documented examples rename `..._total` series to `..._per_second` - so the name the
+  HPA asks for is not the name Micrometer exported.
+- prometheus-adapter 0.12.0 (May 2024) is still its latest release. KEDA's Prometheus scaler is the
+  common alternative: it ships its own metrics API server, reads PromQL directly, and can scale to
+  zero.