Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,47 @@
|
||||
# 1. The lab
|
||||
|
||||
[Index](../README.md) · Next: [2. Probes →](02-probes.md)
|
||||
|
||||
Every transcript in [`output/`](output) came from a single-node **k3s v1.36.4+k3s1** cluster on a
|
||||
2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and
|
||||
on a 64-core node it is much harder to see.
|
||||
|
||||
## Reproducing it
|
||||
|
||||
Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:
|
||||
|
||||
- namespace `demo` ([`k8s/00-namespace.yaml`](../k8s/00-namespace.yaml))
|
||||
- the image `sbd/k8s-demo:1` loaded into the cluster's container runtime, pulled with
|
||||
`imagePullPolicy: Never`. Build it with `mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .`,
|
||||
then `k3s ctr -n k8s.io images import`, `kind load docker-image`, or `minikube image load`
|
||||
- `busybox:1.37` available the same way (the `downstream` service and the `client` pod)
|
||||
- for [chapter 6](06-hpa-custom-metrics.md): `quay.io/prometheus/prometheus:v3.14.0` and
|
||||
`registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0`
|
||||
|
||||
The image is the layered, distroless one recommended in the
|
||||
[Docker article](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/):
|
||||
no shell, which matters in [chapter 5](05-graceful-shutdown.md).
|
||||
|
||||
## Two things specific to this lab
|
||||
|
||||
**The node runs cgroup v1.** Kubernetes 1.35 changed the kubelet's `failCgroupV1` default to `true`
|
||||
(KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes
|
||||
`--kubelet-arg=fail-cgroupv1=false`. Production nodes should be cgroup v2; [chapter 3](03-jvm-ergonomics.md)
|
||||
lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.
|
||||
|
||||
**The VM's process had no `CAP_SYS_RESOURCE`**, so runc could not lower `oom_score_adj` for the pod
|
||||
sandboxes and every pod failed with `failed to update /proc/self/oom_score_adj: Permission denied`.
|
||||
containerd's `restrict_oom_score_adj = true` (a drop-in under
|
||||
`/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/`) fixes it - the same setting rootless
|
||||
setups use.
|
||||
|
||||
**The kubelet garbage-collected images it was about to need.** The VM's disk is shared with its
|
||||
host and reported 94 % used, above the kubelet's default `image-gc-high-threshold` of 85 %. The
|
||||
kubelet deleted every image no running container used - the Prometheus and adapter images among
|
||||
them - and their pods failed with `ErrImageNeverPull`. With `imagePullPolicy: Never` there is no
|
||||
registry to fall back on. The lab now passes `image-gc-high-threshold=100`,
|
||||
`image-gc-low-threshold=99` and a 1 GiB `eviction-hard` threshold; on a real node, fix the disk.
|
||||
|
||||
And one mistake worth not repeating: starting k3s from a shell with `HTTPS_PROXY` set makes the API
|
||||
server route its own connections to the kubelet (`kubectl logs`, `exec`) through that proxy, where
|
||||
they fail. Start it with the proxy variables unset.
|
||||
@@ -0,0 +1,104 @@
|
||||
# 2. Probes: which checks belong in which probe
|
||||
|
||||
[← 1. The lab](01-the-lab.md) · [Index](../README.md) · Next: [3. JVM ergonomics →](03-jvm-ergonomics.md)
|
||||
|
||||
Spring Boot exposes two health groups for Kubernetes, `/actuator/health/liveness` and
|
||||
`/actuator/health/readiness`. They are added automatically when Boot detects Kubernetes (the
|
||||
`*_SERVICE_HOST` / `*_SERVICE_PORT` variables), and explicitly here with
|
||||
`management.endpoint.health.probes.enabled=true` so they also exist on a laptop. By default each
|
||||
contains exactly one thing - the application's `LivenessState` and `ReadinessState` - and **nothing
|
||||
else**, not the database, not `diskSpace`, not your own indicators.
|
||||
|
||||
The three probes answer different questions, and the kubelet does something different on failure:
|
||||
|
||||
| Probe | Question | On failure |
|
||||
|---|---|---|
|
||||
| startup | has it finished starting? | kill after `failureThreshold × periodSeconds`; liveness/readiness wait until it passes |
|
||||
| liveness | is this process broken beyond repair? | **kill and restart the container** |
|
||||
| readiness | should it get traffic right now? | remove from the Service; no restart |
|
||||
|
||||
## One dependency outage, three configurations
|
||||
|
||||
[`DownstreamHealthIndicator`](../src/main/java/com/ankurm/k8s/health/DownstreamHealthIndicator.java)
|
||||
calls a `downstream` service. [`demo-probes.sh`](../scripts/demo-probes.sh) scales the downstream to
|
||||
zero for 60 s and watches the two `orders` pods, the ready endpoints, and whether `GET /work` - an
|
||||
endpoint that **does not use the downstream at all** - still works through the Service.
|
||||
|
||||
**In the liveness group** ([`probes-liveness-includes-downstream.txt`](output/probes-liveness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
Both replicas were killed and restarted together - three failed liveness probes, 15 s - and the
|
||||
restarted containers could not pass their startup probe (it checks the same group) until the
|
||||
downstream came back. A dependency outage became a full outage of a service that did not need the
|
||||
dependency, and the JVMs lost their warm caches and JIT state on top. Had the outage outlasted the
|
||||
startup probe's 120 s budget, the pods would have been restarted again, into `CrashLoopBackOff`.
|
||||
|
||||
**In the readiness group** ([`probes-readiness-includes-downstream.txt`](output/probes-readiness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
No restarts, and recovery within seconds of the downstream returning. **But still zero serving
|
||||
endpoints within 6 s** - every replica checks the same shared dependency, so every replica leaves
|
||||
the Service at once. Clients get connection refused instead of a meaningful error, including for
|
||||
the requests that never needed the downstream.
|
||||
|
||||
**In neither group** - Spring Boot's default ([`probes-default-groups.txt`](output/probes-default-groups.txt)):
|
||||
|
||||
```
|
||||
t= 23s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
The pods stay in service. The requests that need the downstream fail - with whatever error your
|
||||
code returns, which can be a precise 503 - and the rest keep working.
|
||||
|
||||
## The rule this gives
|
||||
|
||||
- **Liveness**: only things a restart would fix. Boot's default `livenessState` is usually exactly
|
||||
right. A shared dependency in liveness restarts every replica at once.
|
||||
- **Readiness**: only things specific to *this* instance - warming a local cache, a full local
|
||||
queue. A shared dependency in readiness takes every replica out at once.
|
||||
- **Shared dependencies** belong in the request path: timeouts, a circuit breaker, a fast 503.
|
||||
Keep them in `/actuator/health` for dashboards - just not in a probe group.
|
||||
|
||||
## Slow starts: the startup probe
|
||||
|
||||
[`demo-startup.sh`](../scripts/demo-startup.sh) makes the context take ~45 s to refresh
|
||||
(`DEMO_STARTUP_DELAY=40s`, a sleep in a `@PostConstruct`) and rolls it out to one replica, first
|
||||
without a startup probe ([`startup-probe.txt`](output/startup-probe.txt)):
|
||||
|
||||
```
|
||||
t= 44s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 50s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 88s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=118s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
```
|
||||
|
||||
Liveness (every 5 s, three failures) starts failing the moment the container starts, because
|
||||
Tomcat is not listening yet (`connect: connection refused` in the events). The new pod is killed
|
||||
before it can ever finish starting, over and over. The first kill only *shows* as a restart at
|
||||
t=50 s because the kill runs the 5 s `preStop` sleep and then waits for the JVM to exit.
|
||||
|
||||
The old pod `7ltj7` keeps serving throughout: with `maxUnavailable: 0` the rollout simply stalls
|
||||
until `progressDeadlineSeconds` (600 s) marks it failed. With more replicas and a
|
||||
`maxUnavailable` above zero, the same change takes capacity away with every attempt.
|
||||
|
||||
The same rollout with a startup probe:
|
||||
|
||||
```
|
||||
t= 45s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 51s [jthjg ready=true restarts=0]
|
||||
```
|
||||
|
||||
Ready at ~51 s, no restarts, and liveness only begins once the startup probe has passed. Size the
|
||||
startup budget (`failureThreshold × periodSeconds`, 120 s here) for your slowest start - a cold
|
||||
node, a slow config server - and keep liveness tight.
|
||||
@@ -0,0 +1,60 @@
|
||||
# 3. JVM ergonomics: what the JVM decides from your pod spec
|
||||
|
||||
[← 2. Probes](02-probes.md) · [Index](../README.md) · Next: [4. CPU limits and GC →](04-cpu-limits-and-gc.md)
|
||||
|
||||
With no JVM options at all, the JVM reads the container's cgroup limits and picks a processor
|
||||
count, a garbage collector, a heap size and thread counts. [`demo-ergonomics.sh`](../scripts/demo-ergonomics.sh)
|
||||
runs `java -XX:+PrintFlagsFinal -version` in a pod per resource shape
|
||||
([`jvm-ergonomics.txt`](output/jvm-ergonomics.txt)):
|
||||
|
||||
```
|
||||
cpu.req cpu.lim mem.req mem.lim | CPUs GC MaxHeap PGCThr CGCThr JITThr
|
||||
100m none 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 500m 256Mi 512Mi | 1 SerialGC 128M 0 0 2
|
||||
100m 1 256Mi 1Gi | 1 SerialGC 256M 0 0 2
|
||||
100m 1 256Mi 2Gi | 1 SerialGC 512M 0 0 2
|
||||
100m 1500m 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 2 256Mi 1Gi | 2 SerialGC 256M 0 0 2
|
||||
100m 2 256Mi 1700Mi | 2 SerialGC 426M 0 0 2
|
||||
100m 2 256Mi 1800Mi | 2 G1GC 450M 2 1 2
|
||||
100m 2 256Mi 4Gi | 2 G1GC 1024M 2 1 2
|
||||
```
|
||||
|
||||
## The rules the table follows
|
||||
|
||||
- **CPUs = the limit, rounded up.** 500m → 1, 1500m → 2. With no limit, the JVM sees every CPU on
|
||||
the node - here 2, on a production node perhaps 64.
|
||||
- **Requests are ignored.** Every row has a 100m request. JDK 19 removed CPU shares from the
|
||||
calculation (JDK-8281181), so on a modern JDK `requests.cpu` affects scheduling and nothing the
|
||||
JVM decides.
|
||||
- **G1 needs 2 CPUs *and* about 1792 MB.** Below either, the JVM is not a "server-class machine"
|
||||
and picks SerialGC: a single-threaded, stop-the-world collector. 2 CPUs with 1700Mi → Serial;
|
||||
1800Mi → G1. A typical Spring Boot pod sized 512Mi-1Gi therefore **runs SerialGC
|
||||
without anyone having chosen it**.
|
||||
- **Heap = 25 % of the memory limit** (`MaxRAMPercentage=25`). A 512Mi pod gets a 128 MB heap and
|
||||
384 MB for metaspace, code cache, thread stacks and native memory - usually far more than they use. `-XX:MaxRAMPercentage=75` is the usual
|
||||
correction; the manifests here set it through `JAVA_TOOL_OPTIONS`.
|
||||
- **1500m gives G1 two parallel GC threads on one and a half CPUs of quota.** That combination is
|
||||
what [chapter 4](04-cpu-limits-and-gc.md) measures.
|
||||
|
||||
## cgroup v1 here, v2 in production
|
||||
|
||||
The lab node is cgroup v1 (a limitation of the VM it ran in; Kubernetes 1.35+ refuses v1 unless
|
||||
`failCgroupV1=false`). JVM container detection works the same on both. The files differ:
|
||||
|
||||
| | cgroup v1 | cgroup v2 |
|
||||
|---|---|---|
|
||||
| CPU limit | `cpu/cpu.cfs_quota_us` / `cpu.cfs_period_us` | `cpu.max` (`150000 100000`) |
|
||||
| throttling | `cpu/cpu.stat`: `nr_throttled`, `throttled_time` (ns) | `cpu.stat`: `nr_throttled`, `throttled_usec` |
|
||||
| memory limit | `memory/memory.limit_in_bytes` | `memory.max` |
|
||||
|
||||
[`JvmController`](../src/main/java/com/ankurm/k8s/diag/JvmController.java) reads either.
|
||||
`java -XshowSettings:system -version` prints what the JVM detected, including the provider:
|
||||
|
||||
```
|
||||
Operating System Metrics:
|
||||
Provider: cgroupv1
|
||||
Effective CPU Count: 2
|
||||
CPU Period: 100000us
|
||||
CPU Quota: 150000us
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# 4. CPU limits and the garbage collector
|
||||
|
||||
[← 3. JVM ergonomics](03-jvm-ergonomics.md) · [Index](../README.md) · Next: [5. Graceful shutdown →](05-graceful-shutdown.md)
|
||||
|
||||
A CPU limit in Kubernetes is a CFS bandwidth quota: `limits.cpu: 1` means 100 ms of CPU time per
|
||||
100 ms period, *shared by every thread in the container*. When the container has used its quota,
|
||||
all its threads stop until the next period. Not slow down - stop. That includes the garbage
|
||||
collector in the middle of a stop-the-world pause.
|
||||
|
||||
## The experiment
|
||||
|
||||
[`demo-gc-throttling.sh`](../scripts/demo-gc-throttling.sh) runs one pod
|
||||
([`k8s/gc-lab.yaml`](../k8s/gc-lab.yaml)) under five resource shapes and drives it for 60 s with 4
|
||||
clients calling `/alloc?mb=16` - allocation-heavy work with a slowly churning old generation. It
|
||||
reads GC pauses from `-Xlog:gc,gc+cpu`, CFS throttling from `cpu.stat` before and after, and
|
||||
latency from the load generator ([`gc-throttling.txt`](output/gc-throttling.txt)):
|
||||
|
||||
```
|
||||
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
|
||||
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
|
||||
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
|
||||
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
|
||||
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
|
||||
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
|
||||
```
|
||||
|
||||
And how much CPU the collector actually got while its pauses were running
|
||||
([`gc-cpu-ratio.txt`](output/gc-cpu-ratio.txt)):
|
||||
|
||||
```
|
||||
collections user+sys real cpu/real
|
||||
A 2189 5.32s 10.71s 0.50
|
||||
B 4279 11.12s 11.12s 1.00
|
||||
C 1783 9.24s 10.11s 0.91
|
||||
D 1960 12.03s 8.69s 1.38
|
||||
E 1878 12.58s 8.59s 1.46
|
||||
```
|
||||
|
||||
`-Xlog:gc+cpu` reports in 10 ms steps, so a single collection's ratio means little; summed over
|
||||
two thousand collections, it is a fair measure of how much of each pause the collector spent
|
||||
waiting for CPU.
|
||||
|
||||
## What it says
|
||||
|
||||
- **At 500m the container spent 37 of 60 seconds throttled**, in 97 % of CFS periods. The single
|
||||
Serial GC thread got CPU for only half the wall time of its own pauses (`cpu/real` 0.50): every
|
||||
pause was stretched to double by the quota, and the longest reached 188 ms. p99 latency: 200 ms.
|
||||
- **"Fixing" a 1-CPU pod by forcing G1 made it worse.** `-XX:ActiveProcessorCount=2 -XX:+UseG1GC`
|
||||
is a common recommendation for getting G1 on a small pod. On a 1-CPU quota, its two parallel GC
|
||||
threads plus concurrent marking burn the quota twice as fast: throttled in **92 %** of periods
|
||||
(against 33 % for Serial on the same quota), **32 % fewer requests** (10,463 vs 15,279), p99 107 ms
|
||||
vs 79 ms. Two GC threads received 0.91 CPU-seconds per pause-second - they were taking turns.
|
||||
- **1.5 CPUs is the interesting row.** The JVM rounds up to 2 processors, so G1 gets 2 threads on 1.5
|
||||
CPUs of quota - throttled in 36 % of periods, yet the best throughput here. Throttling is not
|
||||
automatically a disaster; it is a cost that grows as the quota shrinks relative to the threads
|
||||
that want to run at once.
|
||||
- **D and E are within noise of each other.** The node has exactly 2 CPUs, shared with k3s and
|
||||
the load generator, so a 2-CPU limit mostly meant "no limit". On a bigger node, E would pull ahead.
|
||||
|
||||
## What to do about it
|
||||
|
||||
1. **Know what you are running.** A 1-CPU, 1 GiB pod runs SerialGC ([chapter 3](03-jvm-ergonomics.md)).
|
||||
That can be the right collector for it - B beat C - but it should be a decision.
|
||||
2. **Do not raise `ActiveProcessorCount` above the quota** to get a "better" collector. You buy
|
||||
more threads competing for the same 100 ms.
|
||||
3. **Consider no CPU limit at all**, with a CPU request sized for steady state. Requests guarantee a
|
||||
share under contention and never throttle; limits throttle even on an idle node. Keep the memory
|
||||
limit - memory is not compressible.
|
||||
4. **Watch throttling, not just CPU usage.** `container_cpu_cfs_throttled_periods_total` from
|
||||
cAdvisor (or `nr_throttled` in `cpu.stat`) is the signal; a pod can show 40 % average CPU while
|
||||
being throttled in most periods.
|
||||
|
||||
These are single 60 s runs on a small node: the directions are robust, the exact numbers are not.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 5. Graceful shutdown and the rolling-update race
|
||||
|
||||
[← 4. CPU limits and GC](04-cpu-limits-and-gc.md) · [Index](../README.md) · Next: [6. HPA on a custom metric →](06-hpa-custom-metrics.md)
|
||||
|
||||
When Kubernetes deletes a pod, two things start **at the same time**: the kubelet runs the
|
||||
`preStop` hook and then sends SIGTERM, and the endpoints controller removes the pod from the
|
||||
Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short
|
||||
window a pod that is shutting down still receives new connections.
|
||||
|
||||
Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - `server.shutdown=graceful`) handles
|
||||
the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish,
|
||||
for up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s).
|
||||
|
||||
## The experiment
|
||||
|
||||
[`demo-shutdown.sh`](../scripts/demo-shutdown.sh): 20 closed-loop clients send `POST /work?ms=300`
|
||||
through the Service for 45 s; at t≈8 s, `kubectl rollout restart` replaces both replicas. Each
|
||||
variant three times ([`shutdown-post-summary.txt`](output/shutdown-post-summary.txt)):
|
||||
|
||||
```
|
||||
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
|
||||
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
|
||||
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
|
||||
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
|
||||
```
|
||||
|
||||
Reading it by failure type:
|
||||
|
||||
- **`IOException`** is a request cut off mid-flight, or sent on a keep-alive connection the server
|
||||
closed. `server.shutdown=immediate` produces 24-26 per restart; graceful shutdown cuts that to a
|
||||
handful.
|
||||
- **`ConnectException`** is a new connection to a pod that has already stopped listening but is
|
||||
still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6).
|
||||
**A 5 s `preStop: sleep` removed it in all three runs**: the pod keeps serving while the endpoint
|
||||
removal propagates, and only then gets SIGTERM.
|
||||
- **Variant 4 is variant 2 with extra steps.** An `exec` hook running `sh -c "sleep 5"` fails on
|
||||
the distroless image because there is no `sh`; Kubernetes records `FailedPreStopHook`, sends
|
||||
SIGTERM immediately, and the connect errors come back (4, 5, 7).
|
||||
|
||||
```
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
|
||||
```
|
||||
|
||||
## Why the experiment uses POST
|
||||
|
||||
The first version used GET and reported far fewer failures - 7 for `immediate`
|
||||
([`shutdown-get-1-immediate-no-prestop.txt`](output/shutdown-get-1-immediate-no-prestop.txt)), all
|
||||
`ConnectException`, against 26-32 with POST. The JDK `HttpClient` quietly retries an idempotent
|
||||
request whose connection was closed under it, so a GET load test hides exactly the failures this
|
||||
experiment is looking for. Your clients may or may not retry; a POST shows what the server did.
|
||||
|
||||
## The residue
|
||||
|
||||
Variant 3 still lost 0-2 requests out of ~2,930 per run, all `IOException`: a client reusing an idle
|
||||
keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it
|
||||
is why non-idempotent calls between services need retries with idempotency keys, whatever the
|
||||
deployment does.
|
||||
|
||||
## Sizing the grace period
|
||||
|
||||
`terminationGracePeriodSeconds` (default 30 s) covers the `preStop` hook **and** the shutdown
|
||||
after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s
|
||||
total. Keep `preStop` + `timeout-per-shutdown-phase` below the grace period, or raise it.
|
||||
@@ -0,0 +1,74 @@
|
||||
# 6. HPA on a custom Micrometer metric
|
||||
|
||||
[← 5. Graceful shutdown](05-graceful-shutdown.md) · [Index](../README.md)
|
||||
|
||||
CPU is a poor scaling signal for a JVM service: startup and JIT compilation burn CPU while serving
|
||||
nothing, GC competes with requests for the same quota ([chapter 4](04-cpu-limits-and-gc.md)), and a
|
||||
service waiting on I/O can be saturated at 20 % CPU. What you usually want to scale on is work in
|
||||
progress. The demo scales on in-flight requests.
|
||||
|
||||
## The pipeline
|
||||
|
||||
1. **Micrometer.** [`WorkController`](../src/main/java/com/ankurm/k8s/web/WorkController.java)
|
||||
registers a gauge `app.inflight.requests`. The Prometheus registry renames it: dots become
|
||||
underscores, a gauge gets no suffix (a counter would get `_total`).
|
||||
2. **Prometheus** scrapes `/actuator/prometheus` every 5 s ([`k8s/hpa/prometheus.yaml`](../k8s/hpa/prometheus.yaml))
|
||||
and attaches `namespace` and `pod` labels from Kubernetes service discovery.
|
||||
3. **prometheus-adapter** turns the series into the `custom.metrics.k8s.io` API with one rule
|
||||
([`k8s/hpa/prometheus-adapter.yaml`](../k8s/hpa/prometheus-adapter.yaml)). The
|
||||
`resources.overrides` block is what maps the `pod` label to a Kubernetes Pod - without it the
|
||||
metric exists in Prometheus and never appears in the API.
|
||||
4. **The HPA** ([`k8s/hpa/hpa.yaml`](../k8s/hpa/hpa.yaml)) targets an average of 5 per pod.
|
||||
|
||||
What each end sees ([`hpa-custom-metrics-api.txt`](output/hpa-custom-metrics-api.txt)):
|
||||
|
||||
```
|
||||
app_inflight_requests{application="orders"} 0.0
|
||||
```
|
||||
|
||||
```
|
||||
"metricName": "app_inflight_requests",
|
||||
"timestamp": "2026-09-11T16:33:28Z",
|
||||
"value": "0",
|
||||
```
|
||||
|
||||
## The run
|
||||
|
||||
2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms
|
||||
([`hpa-custom-metric.txt`](output/hpa-custom-metric.txt)):
|
||||
|
||||
```
|
||||
t clients metric desired ready pods
|
||||
0s 2 0 1 1
|
||||
22s 30 333m 1 1
|
||||
32s 30 6333m 2 2
|
||||
43s 30 10333m 4 2
|
||||
54s 30 10333m 4 4
|
||||
64s 30 7475m 4 4
|
||||
148s 0 6416m 4 4
|
||||
169s 0 2499m 4 4
|
||||
179s 0 0 2 2
|
||||
200s 0 0 1 1
|
||||
```
|
||||
|
||||
- **Reaction: ~12 s** from load to the first scale-up - a 5 s scrape, the adapter's query, and the
|
||||
HPA controller's 15 s sync period.
|
||||
- **It overshoots on purpose.** At t=43 the metric was 10.3 against a target of 5 on 2 pods, so the
|
||||
HPA asked for 5 and was capped at `maxReplicas: 4`. 30 clients over 4 pods settles at ~7.5 - still
|
||||
above target, which is what a ceiling looks like.
|
||||
- **Scale-down took 40-60 s after the load stopped**, because the adapter rule averages over 30 s
|
||||
and the HPA's scale-down stabilization window (shortened to 30 s here; **the default is 300 s**)
|
||||
holds the highest recommendation it saw.
|
||||
- **7,209 requests, 0 failures** through two scale-downs - the `preStop` sleep from
|
||||
[chapter 5](05-graceful-shutdown.md) doing its job.
|
||||
|
||||
## Choices in the rule worth knowing about
|
||||
|
||||
- `metricsQuery: avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s])` smooths a gauge that jumps
|
||||
with every request. Without it the HPA chases noise; with too long a window it reacts late.
|
||||
- For a **counter** (say `http_server_requests_seconds_count`) the query must be a `rate(...)`, and
|
||||
the adapter's documented examples rename `..._total` series to `..._per_second` - so the name the
|
||||
HPA asks for is not the name Micrometer exported.
|
||||
- prometheus-adapter 0.12.0 (May 2024) is still its latest release. KEDA's Prometheus scaler is the
|
||||
common alternative: it ships its own metrics API server, reads PromQL directly, and can scale to
|
||||
zero.
|
||||
@@ -0,0 +1,8 @@
|
||||
# CPU the collector received during its pauses, from -Xlog:gc+cpu (sum over every collection)
|
||||
|
||||
collections user+sys real cpu/real
|
||||
A 2189 5.32s 10.71s 0.50
|
||||
B 4279 11.12s 11.12s 1.00
|
||||
C 1783 9.24s 10.11s 0.91
|
||||
D 1960 12.03s 8.69s 1.38
|
||||
E 1878 12.58s 8.59s 1.46
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=28
|
||||
t= 1s conc=4 ok=44
|
||||
t= 2s conc=4 ok=45
|
||||
t= 3s conc=4 ok=51
|
||||
t= 4s conc=4 ok=69
|
||||
t= 5s conc=4 ok=60
|
||||
t= 6s conc=4 ok=62
|
||||
t= 7s conc=4 ok=76
|
||||
t= 8s conc=4 ok=76
|
||||
t= 9s conc=4 ok=90
|
||||
t= 10s conc=4 ok=29
|
||||
t= 11s conc=4 ok=96
|
||||
t= 12s conc=4 ok=116
|
||||
t= 13s conc=4 ok=105
|
||||
t= 14s conc=4 ok=107
|
||||
t= 15s conc=4 ok=109
|
||||
t= 16s conc=4 ok=97
|
||||
t= 17s conc=4 ok=96
|
||||
t= 18s conc=4 ok=92
|
||||
t= 19s conc=4 ok=83
|
||||
t= 20s conc=4 ok=127
|
||||
t= 21s conc=4 ok=110
|
||||
t= 22s conc=4 ok=93
|
||||
t= 23s conc=4 ok=94
|
||||
t= 24s conc=4 ok=118
|
||||
t= 25s conc=4 ok=126
|
||||
t= 26s conc=4 ok=133
|
||||
t= 27s conc=4 ok=133
|
||||
t= 28s conc=4 ok=111
|
||||
t= 29s conc=4 ok=120
|
||||
t= 30s conc=4 ok=117
|
||||
t= 31s conc=4 ok=138
|
||||
t= 32s conc=4 ok=116
|
||||
t= 33s conc=4 ok=92
|
||||
t= 34s conc=4 ok=124
|
||||
t= 35s conc=4 ok=146
|
||||
t= 36s conc=4 ok=107
|
||||
t= 37s conc=4 ok=159
|
||||
t= 38s conc=4 ok=133
|
||||
t= 39s conc=4 ok=151
|
||||
t= 40s conc=4 ok=162
|
||||
t= 41s conc=4 ok=169
|
||||
t= 42s conc=4 ok=129
|
||||
t= 43s conc=4 ok=169
|
||||
t= 44s conc=4 ok=171
|
||||
t= 45s conc=4 ok=159
|
||||
t= 46s conc=4 ok=140
|
||||
t= 47s conc=4 ok=112
|
||||
t= 48s conc=4 ok=108
|
||||
t= 49s conc=4 ok=96
|
||||
t= 50s conc=4 ok=127
|
||||
t= 51s conc=4 ok=100
|
||||
t= 52s conc=4 ok=59
|
||||
t= 53s conc=4 ok=140
|
||||
t= 54s conc=4 ok=85
|
||||
t= 55s conc=4 ok=81
|
||||
t= 56s conc=4 ok=153
|
||||
t= 57s conc=4 ok=115
|
||||
t= 58s conc=4 ok=114
|
||||
t= 59s conc=4 ok=87
|
||||
t= 60s conc=4 ok=14
|
||||
METHOD GET
|
||||
TOTAL {ok=6469}
|
||||
LATENCY ms p50=15 p90=84 p99=200 max=805 (n=6469)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=78
|
||||
t= 1s conc=4 ok=110
|
||||
t= 2s conc=4 ok=131
|
||||
t= 3s conc=4 ok=144
|
||||
t= 4s conc=4 ok=123
|
||||
t= 5s conc=4 ok=78
|
||||
t= 6s conc=4 ok=241
|
||||
t= 7s conc=4 ok=218
|
||||
t= 8s conc=4 ok=195
|
||||
t= 9s conc=4 ok=183
|
||||
t= 10s conc=4 ok=272
|
||||
t= 11s conc=4 ok=224
|
||||
t= 12s conc=4 ok=243
|
||||
t= 13s conc=4 ok=317
|
||||
t= 14s conc=4 ok=235
|
||||
t= 15s conc=4 ok=280
|
||||
t= 16s conc=4 ok=249
|
||||
t= 17s conc=4 ok=248
|
||||
t= 18s conc=4 ok=311
|
||||
t= 19s conc=4 ok=335
|
||||
t= 20s conc=4 ok=286
|
||||
t= 21s conc=4 ok=311
|
||||
t= 22s conc=4 ok=273
|
||||
t= 23s conc=4 ok=232
|
||||
t= 24s conc=4 ok=204
|
||||
t= 25s conc=4 ok=269
|
||||
t= 26s conc=4 ok=312
|
||||
t= 27s conc=4 ok=254
|
||||
t= 28s conc=4 ok=255
|
||||
t= 29s conc=4 ok=249
|
||||
t= 30s conc=4 ok=262
|
||||
t= 31s conc=4 ok=302
|
||||
t= 32s conc=4 ok=243
|
||||
t= 33s conc=4 ok=312
|
||||
t= 34s conc=4 ok=335
|
||||
t= 35s conc=4 ok=323
|
||||
t= 36s conc=4 ok=253
|
||||
t= 37s conc=4 ok=320
|
||||
t= 38s conc=4 ok=226
|
||||
t= 39s conc=4 ok=292
|
||||
t= 40s conc=4 ok=202
|
||||
t= 41s conc=4 ok=180
|
||||
t= 42s conc=4 ok=311
|
||||
t= 43s conc=4 ok=299
|
||||
t= 44s conc=4 ok=231
|
||||
t= 45s conc=4 ok=314
|
||||
t= 46s conc=4 ok=304
|
||||
t= 47s conc=4 ok=265
|
||||
t= 48s conc=4 ok=295
|
||||
t= 49s conc=4 ok=319
|
||||
t= 50s conc=4 ok=278
|
||||
t= 51s conc=4 ok=211
|
||||
t= 52s conc=4 ok=271
|
||||
t= 53s conc=4 ok=319
|
||||
t= 54s conc=4 ok=285
|
||||
t= 55s conc=4 ok=243
|
||||
t= 56s conc=4 ok=306
|
||||
t= 57s conc=4 ok=285
|
||||
t= 58s conc=4 ok=313
|
||||
t= 59s conc=4 ok=288
|
||||
t= 60s conc=4 ok=32
|
||||
METHOD GET
|
||||
TOTAL {ok=15279}
|
||||
LATENCY ms p50=12 p90=25 p99=79 max=447 (n=15279)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=54
|
||||
t= 1s conc=4 ok=85
|
||||
t= 2s conc=4 ok=106
|
||||
t= 3s conc=4 ok=106
|
||||
t= 4s conc=4 ok=119
|
||||
t= 5s conc=4 ok=116
|
||||
t= 6s conc=4 ok=117
|
||||
t= 7s conc=4 ok=127
|
||||
t= 8s conc=4 ok=117
|
||||
t= 9s conc=4 ok=116
|
||||
t= 10s conc=4 ok=131
|
||||
t= 11s conc=4 ok=135
|
||||
t= 12s conc=4 ok=106
|
||||
t= 13s conc=4 ok=112
|
||||
t= 14s conc=4 ok=131
|
||||
t= 15s conc=4 ok=139
|
||||
t= 16s conc=4 ok=140
|
||||
t= 17s conc=4 ok=153
|
||||
t= 18s conc=4 ok=164
|
||||
t= 19s conc=4 ok=166
|
||||
t= 20s conc=4 ok=159
|
||||
t= 21s conc=4 ok=172
|
||||
t= 22s conc=4 ok=194
|
||||
t= 23s conc=4 ok=158
|
||||
t= 24s conc=4 ok=154
|
||||
t= 25s conc=4 ok=188
|
||||
t= 26s conc=4 ok=183
|
||||
t= 27s conc=4 ok=198
|
||||
t= 28s conc=4 ok=208
|
||||
t= 29s conc=4 ok=201
|
||||
t= 30s conc=4 ok=163
|
||||
t= 31s conc=4 ok=194
|
||||
t= 32s conc=4 ok=214
|
||||
t= 33s conc=4 ok=220
|
||||
t= 34s conc=4 ok=176
|
||||
t= 35s conc=4 ok=196
|
||||
t= 36s conc=4 ok=167
|
||||
t= 37s conc=4 ok=158
|
||||
t= 38s conc=4 ok=202
|
||||
t= 39s conc=4 ok=168
|
||||
t= 40s conc=4 ok=222
|
||||
t= 41s conc=4 ok=168
|
||||
t= 42s conc=4 ok=217
|
||||
t= 43s conc=4 ok=222
|
||||
t= 44s conc=4 ok=215
|
||||
t= 45s conc=4 ok=193
|
||||
t= 46s conc=4 ok=189
|
||||
t= 47s conc=4 ok=189
|
||||
t= 48s conc=4 ok=224
|
||||
t= 49s conc=4 ok=201
|
||||
t= 50s conc=4 ok=226
|
||||
t= 51s conc=4 ok=236
|
||||
t= 52s conc=4 ok=226
|
||||
t= 53s conc=4 ok=226
|
||||
t= 54s conc=4 ok=253
|
||||
t= 55s conc=4 ok=250
|
||||
t= 56s conc=4 ok=198
|
||||
t= 57s conc=4 ok=189
|
||||
t= 58s conc=4 ok=239
|
||||
t= 59s conc=4 ok=238
|
||||
t= 60s conc=4 ok=29
|
||||
METHOD GET
|
||||
TOTAL {ok=10463}
|
||||
LATENCY ms p50=13 p90=56 p99=107 max=588 (n=10463)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=125
|
||||
t= 1s conc=4 ok=239
|
||||
t= 2s conc=4 ok=246
|
||||
t= 3s conc=4 ok=257
|
||||
t= 4s conc=4 ok=266
|
||||
t= 5s conc=4 ok=269
|
||||
t= 6s conc=4 ok=357
|
||||
t= 7s conc=4 ok=346
|
||||
t= 8s conc=4 ok=205
|
||||
t= 9s conc=4 ok=228
|
||||
t= 10s conc=4 ok=217
|
||||
t= 11s conc=4 ok=216
|
||||
t= 12s conc=4 ok=244
|
||||
t= 13s conc=4 ok=228
|
||||
t= 14s conc=4 ok=269
|
||||
t= 15s conc=4 ok=295
|
||||
t= 16s conc=4 ok=297
|
||||
t= 17s conc=4 ok=441
|
||||
t= 18s conc=4 ok=417
|
||||
t= 19s conc=4 ok=325
|
||||
t= 20s conc=4 ok=366
|
||||
t= 21s conc=4 ok=446
|
||||
t= 22s conc=4 ok=371
|
||||
t= 23s conc=4 ok=468
|
||||
t= 24s conc=4 ok=375
|
||||
t= 25s conc=4 ok=306
|
||||
t= 26s conc=4 ok=434
|
||||
t= 27s conc=4 ok=405
|
||||
t= 28s conc=4 ok=369
|
||||
t= 29s conc=4 ok=338
|
||||
t= 30s conc=4 ok=374
|
||||
t= 31s conc=4 ok=441
|
||||
t= 32s conc=4 ok=485
|
||||
t= 33s conc=4 ok=286
|
||||
t= 34s conc=4 ok=539
|
||||
t= 35s conc=4 ok=413
|
||||
t= 36s conc=4 ok=380
|
||||
t= 37s conc=4 ok=437
|
||||
t= 38s conc=4 ok=283
|
||||
t= 39s conc=4 ok=378
|
||||
t= 40s conc=4 ok=485
|
||||
t= 41s conc=4 ok=463
|
||||
t= 42s conc=4 ok=501
|
||||
t= 43s conc=4 ok=527
|
||||
t= 44s conc=4 ok=502
|
||||
t= 45s conc=4 ok=442
|
||||
t= 46s conc=4 ok=539
|
||||
t= 47s conc=4 ok=549
|
||||
t= 48s conc=4 ok=504
|
||||
t= 49s conc=4 ok=529
|
||||
t= 50s conc=4 ok=503
|
||||
t= 51s conc=4 ok=512
|
||||
t= 52s conc=4 ok=551
|
||||
t= 53s conc=4 ok=384
|
||||
t= 54s conc=4 ok=411
|
||||
t= 55s conc=4 ok=562
|
||||
t= 56s conc=4 ok=555
|
||||
t= 57s conc=4 ok=566
|
||||
t= 58s conc=4 ok=479
|
||||
t= 59s conc=4 ok=524
|
||||
t= 60s conc=4 ok=48
|
||||
METHOD GET
|
||||
TOTAL {ok=23517}
|
||||
LATENCY ms p50=7 p90=18 p99=51 max=221 (n=23517)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=109
|
||||
t= 1s conc=4 ok=199
|
||||
t= 2s conc=4 ok=229
|
||||
t= 3s conc=4 ok=257
|
||||
t= 4s conc=4 ok=304
|
||||
t= 5s conc=4 ok=290
|
||||
t= 6s conc=4 ok=337
|
||||
t= 7s conc=4 ok=320
|
||||
t= 8s conc=4 ok=314
|
||||
t= 9s conc=4 ok=204
|
||||
t= 10s conc=4 ok=191
|
||||
t= 11s conc=4 ok=228
|
||||
t= 12s conc=4 ok=231
|
||||
t= 13s conc=4 ok=256
|
||||
t= 14s conc=4 ok=247
|
||||
t= 15s conc=4 ok=273
|
||||
t= 16s conc=4 ok=245
|
||||
t= 17s conc=4 ok=237
|
||||
t= 18s conc=4 ok=316
|
||||
t= 19s conc=4 ok=371
|
||||
t= 20s conc=4 ok=329
|
||||
t= 21s conc=4 ok=331
|
||||
t= 22s conc=4 ok=408
|
||||
t= 23s conc=4 ok=349
|
||||
t= 24s conc=4 ok=240
|
||||
t= 25s conc=4 ok=297
|
||||
t= 26s conc=4 ok=323
|
||||
t= 27s conc=4 ok=270
|
||||
t= 28s conc=4 ok=438
|
||||
t= 29s conc=4 ok=414
|
||||
t= 30s conc=4 ok=461
|
||||
t= 31s conc=4 ok=400
|
||||
t= 32s conc=4 ok=371
|
||||
t= 33s conc=4 ok=476
|
||||
t= 34s conc=4 ok=429
|
||||
t= 35s conc=4 ok=460
|
||||
t= 36s conc=4 ok=380
|
||||
t= 37s conc=4 ok=309
|
||||
t= 38s conc=4 ok=425
|
||||
t= 39s conc=4 ok=478
|
||||
t= 40s conc=4 ok=404
|
||||
t= 41s conc=4 ok=482
|
||||
t= 42s conc=4 ok=438
|
||||
t= 43s conc=4 ok=404
|
||||
t= 44s conc=4 ok=523
|
||||
t= 45s conc=4 ok=447
|
||||
t= 46s conc=4 ok=471
|
||||
t= 47s conc=4 ok=405
|
||||
t= 48s conc=4 ok=516
|
||||
t= 49s conc=4 ok=444
|
||||
t= 50s conc=4 ok=388
|
||||
t= 51s conc=4 ok=515
|
||||
t= 52s conc=4 ok=459
|
||||
t= 53s conc=4 ok=489
|
||||
t= 54s conc=4 ok=505
|
||||
t= 55s conc=4 ok=538
|
||||
t= 56s conc=4 ok=486
|
||||
t= 57s conc=4 ok=477
|
||||
t= 58s conc=4 ok=492
|
||||
t= 59s conc=4 ok=489
|
||||
t= 60s conc=4 ok=63
|
||||
METHOD GET
|
||||
TOTAL {ok=22181}
|
||||
LATENCY ms p50=7 p90=20 p99=50 max=184 (n=22181)
|
||||
@@ -0,0 +1,10 @@
|
||||
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
|
||||
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
|
||||
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
|
||||
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
|
||||
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
|
||||
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
|
||||
|
||||
# thr = ParallelGCThreads. pauses/pause sum/max from the GC log during the 60 s run.
|
||||
# throttled = CFS periods in which the container hit its quota / periods elapsed, from cpu.stat.
|
||||
# requests = successful /alloc?mb=16 calls by 4 closed-loop clients in 60 s.
|
||||
@@ -0,0 +1,37 @@
|
||||
# HPA orders: target app_inflight_requests averageValue 5, min 1, max 4; scaleDown stabilization 30 s
|
||||
# load: t=0 2 clients, t=20 30 clients, t=140 0 clients; GET /work?ms=500
|
||||
|
||||
t clients metric desired ready pods
|
||||
0s 2 0 1 1
|
||||
11s 2 333m 1 1
|
||||
22s 30 333m 1 1
|
||||
32s 30 6333m 2 2
|
||||
43s 30 10333m 4 2
|
||||
54s 30 10333m 4 4
|
||||
64s 30 7475m 4 4
|
||||
75s 30 7416m 4 4
|
||||
85s 30 7124m 4 4
|
||||
96s 30 7124m 4 4
|
||||
106s 30 7083m 4 4
|
||||
117s 30 7333m 4 4
|
||||
127s 30 7333m 4 4
|
||||
138s 30 7583m 4 4
|
||||
148s 0 6416m 4 4
|
||||
159s 0 6416m 4 4
|
||||
169s 0 2499m 4 4
|
||||
179s 0 0 2 2
|
||||
190s 0 0 2 2
|
||||
200s 0 0 1 1
|
||||
211s 0 0 1 1
|
||||
221s 0 0 1 1
|
||||
231s 0 0 1 1
|
||||
|
||||
# HPA events:
|
||||
SuccessfulRescale New size: 2; reason: pods metric app_inflight_requests above target
|
||||
SuccessfulRescale New size: 4; reason: pods metric app_inflight_requests above target
|
||||
SuccessfulRescale New size: 2; reason: All metrics below target
|
||||
SuccessfulRescale New size: 1; reason: All metrics below target
|
||||
|
||||
# Load generator summary:
|
||||
TOTAL {ok=7209}
|
||||
LATENCY ms p50=503 p90=510 p99=567 max=698 (n=7209)
|
||||
@@ -0,0 +1,27 @@
|
||||
# What Micrometer exports (one pod):
|
||||
$ curl <pod>:8080/actuator/prometheus | grep app_inflight
|
||||
# HELP app_inflight_requests Requests currently being processed by /work
|
||||
# TYPE app_inflight_requests gauge
|
||||
app_inflight_requests{application="orders"} 0.0
|
||||
|
||||
# What the HPA controller sees through the custom metrics API:
|
||||
$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/demo/pods/%2A/app_inflight_requests
|
||||
{
|
||||
"kind": "MetricValueList",
|
||||
"apiVersion": "custom.metrics.k8s.io/v1beta1",
|
||||
"metadata": {},
|
||||
"items": [
|
||||
{
|
||||
"describedObject": {
|
||||
"kind": "Pod",
|
||||
"namespace": "demo",
|
||||
"name": "orders-7774f94779-gvc7k",
|
||||
"apiVersion": "/v1"
|
||||
},
|
||||
"metricName": "app_inflight_requests",
|
||||
"timestamp": "2026-09-11T16:33:28Z",
|
||||
"value": "0",
|
||||
"selector": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
t= 0s conc=2 ok=2
|
||||
t= 1s conc=2 ok=4
|
||||
t= 2s conc=2 ok=4
|
||||
t= 3s conc=2 ok=4
|
||||
t= 4s conc=2 ok=4
|
||||
t= 5s conc=2 ok=4
|
||||
t= 6s conc=2 ok=4
|
||||
t= 7s conc=2 ok=4
|
||||
t= 8s conc=2 ok=4
|
||||
t= 9s conc=2 ok=4
|
||||
t= 10s conc=2 ok=4
|
||||
t= 11s conc=2 ok=4
|
||||
t= 12s conc=2 ok=4
|
||||
t= 13s conc=2 ok=4
|
||||
t= 14s conc=2 ok=4
|
||||
t= 15s conc=2 ok=4
|
||||
t= 16s conc=2 ok=3
|
||||
t= 17s conc=2 ok=3
|
||||
t= 18s conc=2 ok=4
|
||||
t= 19s conc=2 ok=4
|
||||
t= 20s conc=30 ok=32
|
||||
t= 21s conc=30 ok=60
|
||||
t= 22s conc=30 ok=60
|
||||
t= 23s conc=30 ok=60
|
||||
t= 24s conc=30 ok=60
|
||||
t= 25s conc=30 ok=60
|
||||
t= 26s conc=30 ok=60
|
||||
t= 27s conc=30 ok=60
|
||||
t= 28s conc=30 ok=60
|
||||
t= 29s conc=30 ok=60
|
||||
t= 30s conc=30 ok=60
|
||||
t= 31s conc=30 ok=60
|
||||
t= 32s conc=30 ok=58
|
||||
t= 33s conc=30 ok=60
|
||||
t= 34s conc=30 ok=60
|
||||
t= 35s conc=30 ok=60
|
||||
t= 36s conc=30 ok=60
|
||||
t= 37s conc=30 ok=60
|
||||
t= 38s conc=30 ok=54
|
||||
t= 39s conc=30 ok=57
|
||||
t= 40s conc=30 ok=53
|
||||
t= 41s conc=30 ok=60
|
||||
t= 42s conc=30 ok=60
|
||||
t= 43s conc=30 ok=60
|
||||
t= 44s conc=30 ok=60
|
||||
t= 45s conc=30 ok=53
|
||||
t= 46s conc=30 ok=54
|
||||
t= 47s conc=30 ok=58
|
||||
t= 48s conc=30 ok=57
|
||||
t= 49s conc=30 ok=54
|
||||
t= 50s conc=30 ok=60
|
||||
t= 51s conc=30 ok=60
|
||||
t= 52s conc=30 ok=60
|
||||
t= 53s conc=30 ok=60
|
||||
t= 54s conc=30 ok=60
|
||||
t= 55s conc=30 ok=59
|
||||
t= 56s conc=30 ok=59
|
||||
t= 57s conc=30 ok=60
|
||||
t= 58s conc=30 ok=60
|
||||
t= 59s conc=30 ok=60
|
||||
t= 60s conc=30 ok=60
|
||||
t= 61s conc=30 ok=60
|
||||
t= 62s conc=30 ok=60
|
||||
t= 63s conc=30 ok=60
|
||||
t= 64s conc=30 ok=60
|
||||
t= 65s conc=30 ok=60
|
||||
t= 66s conc=30 ok=60
|
||||
t= 67s conc=30 ok=60
|
||||
t= 68s conc=30 ok=60
|
||||
t= 69s conc=30 ok=60
|
||||
t= 70s conc=30 ok=60
|
||||
t= 71s conc=30 ok=60
|
||||
t= 72s conc=30 ok=60
|
||||
t= 73s conc=30 ok=60
|
||||
t= 74s conc=30 ok=60
|
||||
t= 75s conc=30 ok=59
|
||||
t= 76s conc=30 ok=60
|
||||
t= 77s conc=30 ok=60
|
||||
t= 78s conc=30 ok=57
|
||||
t= 79s conc=30 ok=60
|
||||
t= 80s conc=30 ok=59
|
||||
t= 81s conc=30 ok=60
|
||||
t= 82s conc=30 ok=60
|
||||
t= 83s conc=30 ok=60
|
||||
t= 84s conc=30 ok=60
|
||||
t= 85s conc=30 ok=60
|
||||
t= 86s conc=30 ok=60
|
||||
t= 87s conc=30 ok=56
|
||||
t= 88s conc=30 ok=60
|
||||
t= 89s conc=30 ok=59
|
||||
t= 90s conc=30 ok=60
|
||||
t= 91s conc=30 ok=59
|
||||
t= 92s conc=30 ok=58
|
||||
t= 93s conc=30 ok=60
|
||||
t= 94s conc=30 ok=58
|
||||
t= 95s conc=30 ok=58
|
||||
t= 96s conc=30 ok=60
|
||||
t= 97s conc=30 ok=60
|
||||
t= 98s conc=30 ok=59
|
||||
t= 99s conc=30 ok=58
|
||||
t=100s conc=30 ok=60
|
||||
t=101s conc=30 ok=60
|
||||
t=102s conc=30 ok=59
|
||||
t=103s conc=30 ok=58
|
||||
t=104s conc=30 ok=60
|
||||
t=105s conc=30 ok=60
|
||||
t=106s conc=30 ok=60
|
||||
t=107s conc=30 ok=60
|
||||
t=108s conc=30 ok=60
|
||||
t=109s conc=30 ok=58
|
||||
t=110s conc=30 ok=60
|
||||
t=111s conc=30 ok=60
|
||||
t=112s conc=30 ok=60
|
||||
t=113s conc=30 ok=60
|
||||
t=114s conc=30 ok=60
|
||||
t=115s conc=30 ok=60
|
||||
t=116s conc=30 ok=60
|
||||
t=117s conc=30 ok=60
|
||||
t=118s conc=30 ok=60
|
||||
t=119s conc=30 ok=60
|
||||
t=120s conc=30 ok=60
|
||||
t=121s conc=30 ok=60
|
||||
t=122s conc=30 ok=60
|
||||
t=123s conc=30 ok=60
|
||||
t=124s conc=30 ok=59
|
||||
t=125s conc=30 ok=57
|
||||
t=126s conc=30 ok=60
|
||||
t=127s conc=30 ok=60
|
||||
t=128s conc=30 ok=60
|
||||
t=129s conc=30 ok=60
|
||||
t=130s conc=30 ok=60
|
||||
t=131s conc=30 ok=60
|
||||
t=132s conc=30 ok=60
|
||||
t=133s conc=30 ok=59
|
||||
t=134s conc=30 ok=60
|
||||
t=135s conc=30 ok=59
|
||||
t=136s conc=30 ok=60
|
||||
t=137s conc=30 ok=60
|
||||
t=138s conc=30 ok=60
|
||||
t=139s conc=30 ok=60
|
||||
t=140s conc=0 ok=36
|
||||
t=141s conc=0
|
||||
t=142s conc=0
|
||||
t=143s conc=0
|
||||
t=144s conc=0
|
||||
t=145s conc=0
|
||||
t=146s conc=0
|
||||
t=147s conc=0
|
||||
t=148s conc=0
|
||||
t=149s conc=0
|
||||
t=150s conc=0
|
||||
t=151s conc=0
|
||||
t=152s conc=0
|
||||
t=153s conc=0
|
||||
t=154s conc=0
|
||||
t=155s conc=0
|
||||
t=156s conc=0
|
||||
t=157s conc=0
|
||||
t=158s conc=0
|
||||
t=159s conc=0
|
||||
t=160s conc=0
|
||||
t=161s conc=0
|
||||
t=162s conc=0
|
||||
t=163s conc=0
|
||||
t=164s conc=0
|
||||
t=165s conc=0
|
||||
t=166s conc=0
|
||||
t=167s conc=0
|
||||
t=168s conc=0
|
||||
t=169s conc=0
|
||||
t=170s conc=0
|
||||
t=171s conc=0
|
||||
t=172s conc=0
|
||||
t=173s conc=0
|
||||
t=174s conc=0
|
||||
t=175s conc=0
|
||||
t=176s conc=0
|
||||
t=177s conc=0
|
||||
t=178s conc=0
|
||||
t=179s conc=0
|
||||
t=180s conc=0
|
||||
t=181s conc=0
|
||||
t=182s conc=0
|
||||
t=183s conc=0
|
||||
t=184s conc=0
|
||||
t=185s conc=0
|
||||
t=186s conc=0
|
||||
t=187s conc=0
|
||||
t=188s conc=0
|
||||
t=189s conc=0
|
||||
t=190s conc=0
|
||||
t=191s conc=0
|
||||
t=192s conc=0
|
||||
t=193s conc=0
|
||||
t=194s conc=0
|
||||
t=195s conc=0
|
||||
t=196s conc=0
|
||||
t=197s conc=0
|
||||
t=198s conc=0
|
||||
t=199s conc=0
|
||||
t=200s conc=0
|
||||
t=201s conc=0
|
||||
t=202s conc=0
|
||||
t=203s conc=0
|
||||
t=204s conc=0
|
||||
t=205s conc=0
|
||||
t=206s conc=0
|
||||
t=207s conc=0
|
||||
t=208s conc=0
|
||||
t=209s conc=0
|
||||
t=210s conc=0
|
||||
t=211s conc=0
|
||||
t=212s conc=0
|
||||
t=213s conc=0
|
||||
t=214s conc=0
|
||||
t=215s conc=0
|
||||
t=216s conc=0
|
||||
t=217s conc=0
|
||||
t=218s conc=0
|
||||
t=219s conc=0
|
||||
t=220s conc=0
|
||||
t=221s conc=0
|
||||
t=222s conc=0
|
||||
t=223s conc=0
|
||||
t=224s conc=0
|
||||
t=225s conc=0
|
||||
t=226s conc=0
|
||||
t=227s conc=0
|
||||
t=228s conc=0
|
||||
t=229s conc=0
|
||||
t=230s conc=0
|
||||
METHOD GET
|
||||
TOTAL {ok=7209}
|
||||
LATENCY ms p50=503 p90=510 p99=567 max=698 (n=7209)
|
||||
@@ -0,0 +1,34 @@
|
||||
# JVM ergonomics per pod resources. Image: sbd/k8s-demo:1, no JVM options. Node: 2 CPUs.
|
||||
# cgroup v1 node; see docs/03-jvm-ergonomics.md for the v2 equivalents.
|
||||
|
||||
cpu.req cpu.lim mem.req mem.lim | CPUs GC MaxHeap PGCThr CGCThr JITThr
|
||||
100m none 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 500m 256Mi 512Mi | 1 SerialGC 128M 0 0 2
|
||||
100m 1 256Mi 1Gi | 1 SerialGC 256M 0 0 2
|
||||
100m 1 256Mi 2Gi | 1 SerialGC 512M 0 0 2
|
||||
100m 1500m 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 2 256Mi 1Gi | 2 SerialGC 256M 0 0 2
|
||||
100m 2 256Mi 1700Mi | 2 SerialGC 426M 0 0 2
|
||||
100m 2 256Mi 1800Mi | 2 G1GC 450M 2 1 2
|
||||
100m 2 256Mi 4Gi | 2 G1GC 1024M 2 1 2
|
||||
|
||||
# java -XshowSettings:system for the 1500m case:
|
||||
Operating System Metrics:
|
||||
Provider: cgroupv1
|
||||
Effective CPU Count: 2
|
||||
CPU Period: 100000us
|
||||
CPU Quota: 150000us
|
||||
CPU Shares: 102us
|
||||
List of Processors, 2 total:
|
||||
0 1
|
||||
List of Effective Processors, 2 total:
|
||||
0 1
|
||||
List of Memory Nodes, 1 total:
|
||||
0
|
||||
List of Available Memory Nodes, 1 total:
|
||||
0
|
||||
Memory Limit: 2.00G
|
||||
Memory Soft Limit: Unlimited
|
||||
Memory & Swap Limit: 2.00G
|
||||
Maximum Processes Limit: Unlimited
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# downstream health indicator in NEITHER group (Spring Boot's default probe groups)
|
||||
# health groups: liveness= readiness=
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s btcl4 ready=false restarts=0 terminated | 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 11s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 18s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 29s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 34s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 40s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 46s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 51s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 57s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
-------- downstream restored --------
|
||||
t= 62s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 69s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 75s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 86s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 97s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=103s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=109s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=114s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=120s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.207:8080/actuator/health/liveness": dial tcp 10.42.0.207:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.206:8080/actuator/health/liveness": dial tcp 10.42.0.206:8080: connect: connection refused
|
||||
@@ -0,0 +1,37 @@
|
||||
# downstream health indicator in the LIVENESS group
|
||||
# health groups: liveness=livenessState,downstream readiness=
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | gvc7k ready=false restarts=1 terminated | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 12s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 32s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 39s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 46s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 54s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 62s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 74s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 85s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 98s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=103s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=109s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=114s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=120s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=125s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=131s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Warning Unhealthy Startup probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Warning Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Normal Killing Stopping container app
|
||||
2 Normal Killing Container app failed liveness probe, will be restarted
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.201:8080/actuator/health/liveness": dial tcp 10.42.0.201:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.200:8080/actuator/health/liveness": dial tcp 10.42.0.200:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.199:8080/actuator/health/liveness": dial tcp 10.42.0.199:8080: connect: connection refused
|
||||
1 Warning Unhealthy Readiness probe failed: Get "http://10.42.0.201:8080/actuator/health/readiness": dial tcp 10.42.0.201:8080: connect: connection refused
|
||||
@@ -0,0 +1,33 @@
|
||||
# downstream health indicator in the READINESS group
|
||||
# health groups: liveness= readiness=readinessState,downstream
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | x8bjd ready=false restarts=1 terminated | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 14s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 21s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 28s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 35s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 42s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 49s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 55s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 62s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: ok
|
||||
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 75s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 86s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 97s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=102s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=108s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=113s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=119s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=125s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=130s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.204:8080/actuator/health/liveness": dial tcp 10.42.0.204:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.203:8080/actuator/health/liveness": dial tcp 10.42.0.203:8080: connect: connection refused
|
||||
@@ -0,0 +1,55 @@
|
||||
# server.shutdown=immediate, no preStop hook
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=66
|
||||
t= 4s conc=20 ok=67
|
||||
t= 5s conc=20 ok=67
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=66
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=64
|
||||
t= 10s conc=20 ok=59
|
||||
t= 11s conc=20 ok=67
|
||||
t= 12s conc=20 ok=59
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ConnectException=2 ok=55
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=68
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=5 ok=57
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=66
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=62
|
||||
t= 26s conc=20 ok=66
|
||||
t= 27s conc=20 ok=66
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=63
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=71
|
||||
t= 32s conc=20 ok=61
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=66
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=71
|
||||
t= 39s conc=20 ok=61
|
||||
t= 40s conc=20 ok=62
|
||||
t= 41s conc=20 ok=72
|
||||
t= 42s conc=20 ok=65
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=66
|
||||
t= 45s conc=20 ok=34
|
||||
TOTAL {ConnectException=7, ok=2905}
|
||||
LATENCY ms p50=305 p90=315 p99=503 max=747 (n=2905)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-64544dd744-fb4t9 Stopping container app
|
||||
1 Normal Killing pod/orders-64544dd744-jtqk5 Stopping container app
|
||||
@@ -0,0 +1,55 @@
|
||||
# graceful shutdown (default), no preStop hook
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=62
|
||||
t= 4s conc=20 ok=71
|
||||
t= 5s conc=20 ok=67
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=63
|
||||
t= 8s conc=20 ok=70
|
||||
t= 9s conc=20 ok=67
|
||||
t= 10s conc=20 ok=60
|
||||
t= 11s conc=20 ok=62
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ok=60
|
||||
t= 15s conc=20 ok=56
|
||||
t= 16s conc=20 ok=62
|
||||
t= 17s conc=20 ok=70
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=62
|
||||
t= 20s conc=20 ok=61
|
||||
t= 21s conc=20 ConnectException=5 ok=60
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=67
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=66
|
||||
t= 26s conc=20 ok=61
|
||||
t= 27s conc=20 ok=69
|
||||
t= 28s conc=20 ok=67
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=63
|
||||
t= 31s conc=20 ok=67
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=67
|
||||
t= 36s conc=20 ok=64
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=69
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=66
|
||||
t= 45s conc=20 ok=24
|
||||
TOTAL {ConnectException=5, ok=2908}
|
||||
LATENCY ms p50=305 p90=317 p99=404 max=566 (n=2908)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-765b4cb65c-7jwsj Stopping container app
|
||||
1 Normal Killing pod/orders-765b4cb65c-g2xzr Stopping container app
|
||||
@@ -0,0 +1,55 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5}
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=63
|
||||
t= 3s conc=20 ok=65
|
||||
t= 4s conc=20 ok=72
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=64
|
||||
t= 7s conc=20 ok=67
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=61
|
||||
t= 10s conc=20 ok=64
|
||||
t= 11s conc=20 ok=66
|
||||
t= 12s conc=20 ok=63
|
||||
t= 13s conc=20 ok=64
|
||||
t= 14s conc=20 ok=64
|
||||
t= 15s conc=20 ok=62
|
||||
t= 16s conc=20 ok=68
|
||||
t= 17s conc=20 ok=61
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=62
|
||||
t= 21s conc=20 ok=69
|
||||
t= 22s conc=20 ok=64
|
||||
t= 23s conc=20 ok=67
|
||||
t= 24s conc=20 ok=60
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=63
|
||||
t= 28s conc=20 ok=60
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=70
|
||||
t= 31s conc=20 ok=66
|
||||
t= 32s conc=20 ok=60
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=72
|
||||
t= 38s conc=20 ok=63
|
||||
t= 39s conc=20 ok=63
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=70
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=60
|
||||
t= 44s conc=20 ok=71
|
||||
t= 45s conc=20 ok=23
|
||||
TOTAL {ok=2918}
|
||||
LATENCY ms p50=305 p90=315 p99=376 max=512 (n=2918)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-7c869db794-bbr2d Stopping container app
|
||||
1 Normal Killing pod/orders-7c869db794-gdv2r Stopping container app
|
||||
@@ -0,0 +1,57 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=61
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=69
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=61
|
||||
t= 7s conc=20 ok=71
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=62
|
||||
t= 12s conc=20 ok=58
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ConnectException=1 ok=69
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=65
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=60
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=67
|
||||
t= 21s conc=20 ConnectException=4 ok=62
|
||||
t= 22s conc=20 ok=67
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=65
|
||||
t= 28s conc=20 ok=61
|
||||
t= 29s conc=20 ok=67
|
||||
t= 30s conc=20 ok=67
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=66
|
||||
t= 35s conc=20 ok=63
|
||||
t= 36s conc=20 ok=65
|
||||
t= 37s conc=20 ok=67
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=68
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=63
|
||||
t= 45s conc=20 ok=31
|
||||
TOTAL {ConnectException=5, ok=2922}
|
||||
LATENCY ms p50=305 p90=316 p99=394 max=531 (n=2922)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-5bfb45b45f-k5kxw Stopping container app
|
||||
1 Normal Killing pod/orders-5bfb45b45f-lccf9 Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-5bfb45b45f-k5kxw PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-5bfb45b45f-lccf9 PreStopHook failed
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=66
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=64
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=64
|
||||
t= 7s conc=20 ok=70
|
||||
t= 8s conc=20 ok=64
|
||||
t= 9s conc=20 ok=62
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=70
|
||||
t= 12s conc=20 ok=65
|
||||
t= 13s conc=20 ok=62
|
||||
t= 14s conc=20 IOException=14 ok=51
|
||||
t= 15s conc=20 ok=59
|
||||
t= 16s conc=20 ok=64
|
||||
t= 17s conc=20 ok=71
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 ok=60
|
||||
t= 20s conc=20 ok=65
|
||||
t= 21s conc=20 IOException=12 ok=54
|
||||
t= 22s conc=20 ok=68
|
||||
t= 23s conc=20 ok=69
|
||||
t= 24s conc=20 ok=60
|
||||
t= 25s conc=20 ok=67
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=66
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=65
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=64
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=69
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=64
|
||||
t= 45s conc=20 ok=33
|
||||
METHOD POST
|
||||
TOTAL {IOException=26, ok=2909}
|
||||
LATENCY ms p50=305 p90=316 p99=419 max=515 (n=2909)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-54c6f9fcdf-s8t24 Stopping container app
|
||||
1 Normal Killing pod/orders-54c6f9fcdf-t44vr Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=80
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=69
|
||||
t= 6s conc=20 ok=71
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=61
|
||||
t= 10s conc=20 ok=72
|
||||
t= 11s conc=20 ok=67
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=62
|
||||
t= 14s conc=20 ConnectException=3 IOException=12 ok=62
|
||||
t= 15s conc=20 ok=64
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=60
|
||||
t= 18s conc=20 ok=65
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=69
|
||||
t= 21s conc=20 ConnectException=3 IOException=14 ok=58
|
||||
t= 22s conc=20 ok=62
|
||||
t= 23s conc=20 ok=61
|
||||
t= 24s conc=20 ok=64
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=66
|
||||
t= 27s conc=20 ok=60
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=73
|
||||
t= 30s conc=20 ok=61
|
||||
t= 31s conc=20 ok=64
|
||||
t= 32s conc=20 ok=69
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=60
|
||||
t= 35s conc=20 ok=65
|
||||
t= 36s conc=20 ok=70
|
||||
t= 37s conc=20 ok=65
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=71
|
||||
t= 40s conc=20 ok=61
|
||||
t= 41s conc=20 ok=64
|
||||
t= 42s conc=20 ok=65
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=20
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=6, IOException=26, ok=2909}
|
||||
LATENCY ms p50=304 p90=315 p99=375 max=466 (n=2909)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-69d489d468-ncnf5 Stopping container app
|
||||
1 Normal Killing pod/orders-69d489d468-v6dg2 Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=68
|
||||
t= 3s conc=20 ok=72
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=72
|
||||
t= 7s conc=20 ok=66
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=68
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=64
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=68
|
||||
t= 14s conc=20 ConnectException=2 IOException=8 ok=61
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=60
|
||||
t= 18s conc=20 ok=63
|
||||
t= 19s conc=20 ok=69
|
||||
t= 20s conc=20 ok=68
|
||||
t= 21s conc=20 ConnectException=3 IOException=16 ok=46
|
||||
t= 22s conc=20 ok=60
|
||||
t= 23s conc=20 ok=62
|
||||
t= 24s conc=20 ok=72
|
||||
t= 25s conc=20 ok=66
|
||||
t= 26s conc=20 ok=62
|
||||
t= 27s conc=20 ok=65
|
||||
t= 28s conc=20 ok=70
|
||||
t= 29s conc=20 ok=63
|
||||
t= 30s conc=20 ok=62
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=68
|
||||
t= 33s conc=20 ok=60
|
||||
t= 34s conc=20 ok=63
|
||||
t= 35s conc=20 ok=74
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=62
|
||||
t= 38s conc=20 ok=70
|
||||
t= 39s conc=20 ok=68
|
||||
t= 40s conc=20 ok=62
|
||||
t= 41s conc=20 ok=60
|
||||
t= 42s conc=20 ok=75
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=38
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=5, IOException=24, ok=2915}
|
||||
LATENCY ms p50=304 p90=315 p99=422 max=491 (n=2915)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-d59686bf-fg2x7 Stopping container app
|
||||
1 Normal Killing pod/orders-d59686bf-vkrrs Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=64
|
||||
t= 4s conc=20 ok=61
|
||||
t= 5s conc=20 ok=65
|
||||
t= 6s conc=20 ok=69
|
||||
t= 7s conc=20 ok=65
|
||||
t= 8s conc=20 ok=61
|
||||
t= 9s conc=20 ok=65
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=68
|
||||
t= 12s conc=20 ok=64
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=62
|
||||
t= 15s conc=20 ok=66
|
||||
t= 16s conc=20 ok=63
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=61
|
||||
t= 20s conc=20 ok=71
|
||||
t= 21s conc=20 ok=61
|
||||
t= 22s conc=20 IOException=3 ok=65
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=62
|
||||
t= 28s conc=20 ok=68
|
||||
t= 29s conc=20 ok=65
|
||||
t= 30s conc=20 ok=65
|
||||
t= 31s conc=20 ok=63
|
||||
t= 32s conc=20 ok=69
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=64
|
||||
t= 35s conc=20 ok=69
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=66
|
||||
t= 41s conc=20 ok=64
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=33
|
||||
METHOD POST
|
||||
TOTAL {IOException=3, ok=2924}
|
||||
LATENCY ms p50=305 p90=316 p99=365 max=572 (n=2924)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-98b44f5f4-vwrm2 Stopping container app
|
||||
1 Normal Killing pod/orders-98b44f5f4-xshdx Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=61
|
||||
t= 2s conc=20 ok=79
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=68
|
||||
t= 6s conc=20 ok=72
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=63
|
||||
t= 10s conc=20 ok=64
|
||||
t= 11s conc=20 ok=73
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=65
|
||||
t= 14s conc=20 ConnectException=2 ok=61
|
||||
t= 15s conc=20 ok=67
|
||||
t= 16s conc=20 ok=65
|
||||
t= 17s conc=20 ok=62
|
||||
t= 18s conc=20 ok=61
|
||||
t= 19s conc=20 ok=72
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=6 IOException=11 ok=62
|
||||
t= 22s conc=20 ok=63
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=67
|
||||
t= 28s conc=20 ok=64
|
||||
t= 29s conc=20 ok=69
|
||||
t= 30s conc=20 ok=62
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=69
|
||||
t= 37s conc=20 ok=62
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=71
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=61
|
||||
t= 43s conc=20 ok=72
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=29
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=8, IOException=11, ok=2932}
|
||||
LATENCY ms p50=305 p90=315 p99=357 max=462 (n=2932)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-548cdf7b4c-gh4p6 Stopping container app
|
||||
1 Normal Killing pod/orders-548cdf7b4c-r55sk Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=80
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=71
|
||||
t= 6s conc=20 ok=69
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=72
|
||||
t= 10s conc=20 ok=68
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=78
|
||||
t= 14s conc=20 ConnectException=4 IOException=1 ok=62
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=61
|
||||
t= 18s conc=20 ok=68
|
||||
t= 19s conc=20 ok=64
|
||||
t= 20s conc=20 ok=67
|
||||
t= 21s conc=20 ConnectException=2 IOException=4 ok=61
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=65
|
||||
t= 24s conc=20 ok=73
|
||||
t= 25s conc=20 ok=61
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=68
|
||||
t= 28s conc=20 ok=69
|
||||
t= 29s conc=20 ok=60
|
||||
t= 30s conc=20 ok=63
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=65
|
||||
t= 35s conc=20 ok=72
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=63
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=69
|
||||
t= 43s conc=20 ok=65
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=21
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=6, IOException=5, ok=2929}
|
||||
LATENCY ms p50=304 p90=314 p99=369 max=475 (n=2929)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-64f8fbf668-2znqt Stopping container app
|
||||
1 Normal Killing pod/orders-64f8fbf668-sbmgw Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=64
|
||||
t= 3s conc=20 ok=63
|
||||
t= 4s conc=20 ok=68
|
||||
t= 5s conc=20 ok=65
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=68
|
||||
t= 8s conc=20 ok=67
|
||||
t= 9s conc=20 ok=65
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=72
|
||||
t= 14s conc=20 ok=61
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=67
|
||||
t= 17s conc=20 ok=64
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 IOException=1 ok=65
|
||||
t= 20s conc=20 ok=59
|
||||
t= 21s conc=20 ok=70
|
||||
t= 22s conc=20 ok=67
|
||||
t= 23s conc=20 ok=63
|
||||
t= 24s conc=20 ok=64
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 IOException=1 ok=68
|
||||
t= 27s conc=20 ok=61
|
||||
t= 28s conc=20 ok=63
|
||||
t= 29s conc=20 ok=70
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=61
|
||||
t= 32s conc=20 ok=64
|
||||
t= 33s conc=20 ok=71
|
||||
t= 34s conc=20 ok=65
|
||||
t= 35s conc=20 ok=60
|
||||
t= 36s conc=20 ok=66
|
||||
t= 37s conc=20 ok=71
|
||||
t= 38s conc=20 ok=63
|
||||
t= 39s conc=20 ok=64
|
||||
t= 40s conc=20 ok=66
|
||||
t= 41s conc=20 ok=70
|
||||
t= 42s conc=20 ok=60
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=71
|
||||
t= 45s conc=20 ok=29
|
||||
METHOD POST
|
||||
TOTAL {IOException=2, ok=2927}
|
||||
LATENCY ms p50=305 p90=316 p99=370 max=522 (n=2927)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-c7b9b585f-9nrcr Stopping container app
|
||||
1 Normal Killing pod/orders-c7b9b585f-zvcxr Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=70
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=61
|
||||
t= 6s conc=20 ok=75
|
||||
t= 7s conc=20 ok=64
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=80
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=65
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=72
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=62
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=69
|
||||
t= 21s conc=20 ok=62
|
||||
t= 22s conc=20 ok=66
|
||||
t= 23s conc=20 ok=72
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=65
|
||||
t= 26s conc=20 ok=65
|
||||
t= 27s conc=20 ok=68
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=62
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=61
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=67
|
||||
t= 36s conc=20 ok=61
|
||||
t= 37s conc=20 ok=65
|
||||
t= 38s conc=20 ok=72
|
||||
t= 39s conc=20 ok=63
|
||||
t= 40s conc=20 ok=60
|
||||
t= 41s conc=20 ok=72
|
||||
t= 42s conc=20 ok=67
|
||||
t= 43s conc=20 ok=61
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=31
|
||||
METHOD POST
|
||||
TOTAL {ok=2942}
|
||||
LATENCY ms p50=304 p90=313 p99=360 max=493 (n=2942)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-766f65d94b-4hwnl Stopping container app
|
||||
1 Normal Killing pod/orders-766f65d94b-l5nmd Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=70
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=74
|
||||
t= 7s conc=20 ok=64
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=65
|
||||
t= 11s conc=20 ok=74
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=63
|
||||
t= 15s conc=20 ok=74
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=61
|
||||
t= 19s conc=20 IOException=1 ok=69
|
||||
t= 20s conc=20 ok=65
|
||||
t= 21s conc=20 ok=60
|
||||
t= 22s conc=20 ok=65
|
||||
t= 23s conc=20 ok=69
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=61
|
||||
t= 26s conc=20 ok=69
|
||||
t= 27s conc=20 ok=64
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=66
|
||||
t= 32s conc=20 ok=64
|
||||
t= 33s conc=20 ok=65
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=64
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=66
|
||||
t= 40s conc=20 ok=69
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=68
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=69
|
||||
t= 45s conc=20 ok=28
|
||||
METHOD POST
|
||||
TOTAL {IOException=1, ok=2939}
|
||||
LATENCY ms p50=304 p90=313 p99=360 max=476 (n=2939)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-5cfc646684-4stbt Stopping container app
|
||||
1 Normal Killing pod/orders-5cfc646684-zst8t Stopping container app
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=63
|
||||
t= 4s conc=20 ok=65
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=70
|
||||
t= 7s conc=20 ok=63
|
||||
t= 8s conc=20 ok=65
|
||||
t= 9s conc=20 ok=62
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=68
|
||||
t= 12s conc=20 ok=61
|
||||
t= 13s conc=20 ok=68
|
||||
t= 14s conc=20 ConnectException=2 ok=63
|
||||
t= 15s conc=20 ok=61
|
||||
t= 16s conc=20 ok=67
|
||||
t= 17s conc=20 ok=64
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=61
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=2 IOException=3 ok=69
|
||||
t= 22s conc=20 ok=68
|
||||
t= 23s conc=20 ok=60
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=70
|
||||
t= 27s conc=20 ok=60
|
||||
t= 28s conc=20 ok=65
|
||||
t= 29s conc=20 ok=67
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=62
|
||||
t= 32s conc=20 ok=66
|
||||
t= 33s conc=20 ok=72
|
||||
t= 34s conc=20 ok=60
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=68
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=66
|
||||
t= 40s conc=20 ok=72
|
||||
t= 41s conc=20 ok=60
|
||||
t= 42s conc=20 ok=68
|
||||
t= 43s conc=20 ok=64
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=28
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=4, IOException=3, ok=2921}
|
||||
LATENCY ms p50=305 p90=315 p99=370 max=545 (n=2921)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-68fd794d97-sjq65 Stopping container app
|
||||
1 Normal Killing pod/orders-68fd794d97-zsznq Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=71
|
||||
t= 3s conc=20 ok=69
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=76
|
||||
t= 7s conc=20 ok=62
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=69
|
||||
t= 10s conc=20 ok=66
|
||||
t= 11s conc=20 ok=65
|
||||
t= 12s conc=20 ok=57
|
||||
t= 13s conc=20 ok=60
|
||||
t= 14s conc=20 ConnectException=2 ok=69
|
||||
t= 15s conc=20 ok=57
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=71
|
||||
t= 18s conc=20 ok=66
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=3 IOException=2 ok=66
|
||||
t= 22s conc=20 ok=66
|
||||
t= 23s conc=20 ok=65
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=64
|
||||
t= 27s conc=20 ok=64
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=70
|
||||
t= 30s conc=20 ok=64
|
||||
t= 31s conc=20 ok=64
|
||||
t= 32s conc=20 ok=70
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=64
|
||||
t= 35s conc=20 ok=62
|
||||
t= 36s conc=20 ok=70
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=62
|
||||
t= 40s conc=20 ok=74
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=63
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=25
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=5, IOException=2, ok=2926}
|
||||
LATENCY ms p50=305 p90=314 p99=376 max=606 (n=2926)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-54f46cb799-5bjgb Stopping container app
|
||||
1 Normal Killing pod/orders-54f46cb799-mz47p Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-54f46cb799-5bjgb PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-54f46cb799-mz47p PreStopHook failed
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=74
|
||||
t= 3s conc=20 ok=66
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=68
|
||||
t= 7s conc=20 ok=72
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=66
|
||||
t= 11s conc=20 ok=73
|
||||
t= 12s conc=20 ok=61
|
||||
t= 13s conc=20 ok=64
|
||||
t= 14s conc=20 ConnectException=3 IOException=1 ok=64
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=66
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=62
|
||||
t= 19s conc=20 ok=72
|
||||
t= 20s conc=20 ok=60
|
||||
t= 21s conc=20 ConnectException=4 ok=66
|
||||
t= 22s conc=20 ok=62
|
||||
t= 23s conc=20 ok=66
|
||||
t= 24s conc=20 ok=67
|
||||
t= 25s conc=20 ok=65
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=70
|
||||
t= 28s conc=20 ok=67
|
||||
t= 29s conc=20 ok=62
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=62
|
||||
t= 32s conc=20 ok=70
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=70
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=65
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=68
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=22
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=7, IOException=1, ok=2930}
|
||||
LATENCY ms p50=305 p90=314 p99=354 max=468 (n=2930)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-579c8f9fd9-st6sf Stopping container app
|
||||
1 Normal Killing pod/orders-579c8f9fd9-x2zwq Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-579c8f9fd9-st6sf PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-579c8f9fd9-x2zwq PreStopHook failed
|
||||
@@ -0,0 +1,6 @@
|
||||
# POST /work?ms=300, 20 clients, rolling restart of 2 replicas. Failed requests per run (successful in brackets).
|
||||
|
||||
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
|
||||
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
|
||||
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
|
||||
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
|
||||
@@ -0,0 +1,56 @@
|
||||
# DEMO_STARTUP_DELAY=40s: the context takes ~45 s to refresh, and Tomcat only listens after that.
|
||||
# 1 replica, rolling update (maxSurge 1, maxUnavailable 0). Liveness: period 5 s, failureThreshold 3.
|
||||
# Pods being deleted are not listed. The old pod keeps serving while the new one is not ready.
|
||||
|
||||
## no startupProbe
|
||||
t= 0s [7ltj7 ready=true restarts=0]
|
||||
t= 7s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 13s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 19s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 25s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 32s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 38s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 44s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 50s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 57s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 63s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 69s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 75s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 81s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 88s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t= 94s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=100s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=106s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=112s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=118s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
|
||||
# Events so far:
|
||||
1 Normal Killing Container app failed liveness probe, will be restarted
|
||||
1 Warning Unhealthy Liveness probe failed: Get "http://10.42.0.221:8080/actuator/health/liveness": dial tcp 10.42.0.221:8080: connect: connection refused
|
||||
1 Warning Unhealthy Readiness probe failed: Get "http://10.42.0.221:8080/actuator/health/readiness": dial tcp 10.42.0.221:8080: connect: connection refused
|
||||
|
||||
## startupProbe: /actuator/health/liveness every 2 s, failureThreshold 60 (a 120 s budget)
|
||||
t= 0s [7ltj7 ready=true restarts=0]
|
||||
t= 7s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 13s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 20s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 26s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 32s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 39s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 45s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 51s [jthjg ready=true restarts=0]
|
||||
t= 58s [jthjg ready=true restarts=0]
|
||||
t= 64s [jthjg ready=true restarts=0]
|
||||
t= 70s [jthjg ready=true restarts=0]
|
||||
t= 76s [jthjg ready=true restarts=0]
|
||||
t= 82s [jthjg ready=true restarts=0]
|
||||
t= 88s [jthjg ready=true restarts=0]
|
||||
t= 95s [jthjg ready=true restarts=0]
|
||||
t=101s [jthjg ready=true restarts=0]
|
||||
t=108s [jthjg ready=true restarts=0]
|
||||
t=114s [jthjg ready=true restarts=0]
|
||||
t=120s [jthjg ready=true restarts=0]
|
||||
|
||||
# Events during the startupProbe run:
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.222:8080/actuator/health/liveness": dial tcp 10.42.0.222:8080: connect: connection refused
|
||||
Reference in New Issue
Block a user