asmhatreandClaude Opus 5
a065696478
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
75 lines
3.6 KiB
Markdown
75 lines
3.6 KiB
Markdown
# 6. HPA on a custom Micrometer metric
|
|
|
|
[← 5. Graceful shutdown](05-graceful-shutdown.md) · [Index](../README.md)
|
|
|
|
CPU is a poor scaling signal for a JVM service: startup and JIT compilation burn CPU while serving
|
|
nothing, GC competes with requests for the same quota ([chapter 4](04-cpu-limits-and-gc.md)), and a
|
|
service waiting on I/O can be saturated at 20 % CPU. What you usually want to scale on is work in
|
|
progress. The demo scales on in-flight requests.
|
|
|
|
## The pipeline
|
|
|
|
1. **Micrometer.** [`WorkController`](../src/main/java/com/ankurm/k8s/web/WorkController.java)
|
|
registers a gauge `app.inflight.requests`. The Prometheus registry renames it: dots become
|
|
underscores, a gauge gets no suffix (a counter would get `_total`).
|
|
2. **Prometheus** scrapes `/actuator/prometheus` every 5 s ([`k8s/hpa/prometheus.yaml`](../k8s/hpa/prometheus.yaml))
|
|
and attaches `namespace` and `pod` labels from Kubernetes service discovery.
|
|
3. **prometheus-adapter** turns the series into the `custom.metrics.k8s.io` API with one rule
|
|
([`k8s/hpa/prometheus-adapter.yaml`](../k8s/hpa/prometheus-adapter.yaml)). The
|
|
`resources.overrides` block is what maps the `pod` label to a Kubernetes Pod - without it the
|
|
metric exists in Prometheus and never appears in the API.
|
|
4. **The HPA** ([`k8s/hpa/hpa.yaml`](../k8s/hpa/hpa.yaml)) targets an average of 5 per pod.
|
|
|
|
What each end sees ([`hpa-custom-metrics-api.txt`](output/hpa-custom-metrics-api.txt)):
|
|
|
|
```
|
|
app_inflight_requests{application="orders"} 0.0
|
|
```
|
|
|
|
```
|
|
"metricName": "app_inflight_requests",
|
|
"timestamp": "2026-09-11T16:33:28Z",
|
|
"value": "0",
|
|
```
|
|
|
|
## The run
|
|
|
|
2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms
|
|
([`hpa-custom-metric.txt`](output/hpa-custom-metric.txt)):
|
|
|
|
```
|
|
t clients metric desired ready pods
|
|
0s 2 0 1 1
|
|
22s 30 333m 1 1
|
|
32s 30 6333m 2 2
|
|
43s 30 10333m 4 2
|
|
54s 30 10333m 4 4
|
|
64s 30 7475m 4 4
|
|
148s 0 6416m 4 4
|
|
169s 0 2499m 4 4
|
|
179s 0 0 2 2
|
|
200s 0 0 1 1
|
|
```
|
|
|
|
- **Reaction: ~12 s** from load to the first scale-up - a 5 s scrape, the adapter's query, and the
|
|
HPA controller's 15 s sync period.
|
|
- **It overshoots on purpose.** At t=43 the metric was 10.3 against a target of 5 on 2 pods, so the
|
|
HPA asked for 5 and was capped at `maxReplicas: 4`. 30 clients over 4 pods settles at ~7.5 - still
|
|
above target, which is what a ceiling looks like.
|
|
- **Scale-down took 40-60 s after the load stopped**, because the adapter rule averages over 30 s
|
|
and the HPA's scale-down stabilization window (shortened to 30 s here; **the default is 300 s**)
|
|
holds the highest recommendation it saw.
|
|
- **7,209 requests, 0 failures** through two scale-downs - the `preStop` sleep from
|
|
[chapter 5](05-graceful-shutdown.md) doing its job.
|
|
|
|
## Choices in the rule worth knowing about
|
|
|
|
- `metricsQuery: avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s])` smooths a gauge that jumps
|
|
with every request. Without it the HPA chases noise; with too long a window it reacts late.
|
|
- For a **counter** (say `http_server_requests_seconds_count`) the query must be a `rate(...)`, and
|
|
the adapter's documented examples rename `..._total` series to `..._per_second` - so the name the
|
|
HPA asks for is not the name Micrometer exported.
|
|
- prometheus-adapter 0.12.0 (May 2024) is still its latest release. KEDA's Prometheus scaler is the
|
|
common alternative: it ships its own metrics API server, reads PromQL directly, and can scale to
|
|
zero.
|