spring-boot-demo/kubernetes-deployment/docs/06-hpa-custom-metrics.md

6. HPA on a custom Micrometer metric

← 5. Graceful shutdown · Index

CPU is a poor scaling signal for a JVM service: startup and JIT compilation burn CPU while serving nothing, GC competes with requests for the same quota (chapter 4), and a service waiting on I/O can be saturated at 20 % CPU. What you usually want to scale on is work in progress. The demo scales on in-flight requests.

The pipeline

1. Micrometer. WorkController registers a gauge app.inflight.requests. The Prometheus registry renames it: dots become underscores, a gauge gets no suffix (a counter would get _total).
2. Prometheus scrapes /actuator/prometheus every 5 s (k8s/hpa/prometheus.yaml) and attaches namespace and pod labels from Kubernetes service discovery.
3. prometheus-adapter turns the series into the custom.metrics.k8s.io API with one rule (k8s/hpa/prometheus-adapter.yaml). The resources.overrides block is what maps the pod label to a Kubernetes Pod - without it the metric exists in Prometheus and never appears in the API.
4. The HPA (k8s/hpa/hpa.yaml) targets an average of 5 per pod.

What each end sees (hpa-custom-metrics-api.txt):

app_inflight_requests{application="orders"} 0.0

            "metricName": "app_inflight_requests",
            "timestamp": "2026-09-11T16:33:28Z",
            "value": "0",

The run

2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms (hpa-custom-metric.txt):

t      clients    metric    desired  ready pods
0s     2          0         1        1
22s    30         333m      1        1
32s    30         6333m     2        2
43s    30         10333m    4        2
54s    30         10333m    4        4
64s    30         7475m     4        4
148s   0          6416m     4        4
169s   0          2499m     4        4
179s   0          0         2        2
200s   0          0         1        1

• Reaction: ~12 s from load to the first scale-up - a 5 s scrape, the adapter's query, and the HPA controller's 15 s sync period.
• It overshoots on purpose. At t=43 the metric was 10.3 against a target of 5 on 2 pods, so the HPA asked for 5 and was capped at maxReplicas: 4. 30 clients over 4 pods settles at ~7.5 - still above target, which is what a ceiling looks like.
• Scale-down took 40-60 s after the load stopped, because the adapter rule averages over 30 s and the HPA's scale-down stabilization window (shortened to 30 s here; the default is 300 s) holds the highest recommendation it saw.
• 7,209 requests, 0 failures through two scale-downs - the preStop sleep from chapter 5 doing its job.

Choices in the rule worth knowing about

• metricsQuery: avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s]) smooths a gauge that jumps with every request. Without it the HPA chases noise; with too long a window it reacts late.
• For a counter (say http_server_requests_seconds_count) the query must be a rate(...), and the adapter's documented examples rename ..._total series to ..._per_second - so the name the HPA asks for is not the name Micrometer exported.
• prometheus-adapter 0.12.0 (May 2024) is still its latest release. KEDA's Prometheus scaler is the common alternative: it ships its own metrics API server, reads PromQL directly, and can scale to zero.