Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,64 @@
|
||||
# Spring Boot 4 on Kubernetes: probes, graceful shutdown, limits and JVM ergonomics
|
||||
|
||||
Companion project for [**Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics**](https://ankurm.com/spring-boot-4-kubernetes-probes-graceful-shutdown-cpu-limits-hpa/)
|
||||
on ankurm.com.
|
||||
|
||||
A small Spring Boot 4 service, the manifests that deploy it, and one script per experiment - a
|
||||
dependency outage under three probe configurations, a rolling restart under load four ways, the JVM's
|
||||
ergonomic choices for nine pod shapes, the same GC-heavy load under five CPU limits, and a
|
||||
HorizontalPodAutoscaler driven by a Micrometer gauge. Every figure in the article is in
|
||||
[`docs/output/`](docs/output).
|
||||
|
||||
## Versions
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Spring Boot | 4.1.1 |
|
||||
| JDK | Temurin 25.0.4.1 (in `gcr.io/distroless/java25-debian13:nonroot`) |
|
||||
| Kubernetes | k3s v1.36.4+k3s1, single node, 2 vCPU / 8 GB, cgroup v1 ([docs/01-the-lab.md](docs/01-the-lab.md)) |
|
||||
| Prometheus | 3.14.0 |
|
||||
| prometheus-adapter | 0.12.0 |
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
export JAVA_HOME=/path/to/jdk-25
|
||||
mvn -DskipTests package
|
||||
docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .
|
||||
# load the image into your cluster (k3s ctr images import / kind load docker-image / minikube image load)
|
||||
kubectl apply -f k8s/00-namespace.yaml -f k8s/downstream.yaml -f k8s/orders.yaml -f k8s/client.yaml
|
||||
./scripts/demo-probes.sh # ...and the other demo-*.sh; see the table below
|
||||
```
|
||||
|
||||
## What is in it
|
||||
|
||||
| Path | |
|
||||
|---|---|
|
||||
| [`k8s/orders.yaml`](k8s/orders.yaml) | the baseline Deployment: startup/liveness/readiness probes, `preStop: sleep`, `MaxRAMPercentage=75` |
|
||||
| [`k8s/downstream.yaml`](k8s/downstream.yaml) | a dependency to take away |
|
||||
| [`k8s/gc-lab.yaml`](k8s/gc-lab.yaml) | the single pod the CPU-limit experiment reshapes |
|
||||
| [`k8s/hpa/`](k8s/hpa) | Prometheus, prometheus-adapter with one rule, the HPA |
|
||||
| [`WorkController`](src/main/java/com/ankurm/k8s/web/WorkController.java) | `/work` (timed requests, the `app.inflight.requests` gauge), `/alloc` (GC load) |
|
||||
| [`DownstreamHealthIndicator`](src/main/java/com/ankurm/k8s/health/DownstreamHealthIndicator.java) | a health indicator whose probe group is the experiment |
|
||||
| [`JvmController`](src/main/java/com/ankurm/k8s/diag/JvmController.java) | `/diag/jvm`, `/diag/gc` - ergonomics and cgroup throttling counters. Diagnostic; delete before shipping |
|
||||
| [`LoadGen`](src/main/java/com/ankurm/k8s/loadgen/LoadGen.java) | the in-cluster load generator, run from the same image |
|
||||
|
||||
## Documentation
|
||||
|
||||
1. [The lab](docs/01-the-lab.md)
|
||||
2. [Probes: which checks belong in which probe](docs/02-probes.md)
|
||||
3. [JVM ergonomics: what the JVM decides from your pod spec](docs/03-jvm-ergonomics.md)
|
||||
4. [CPU limits and the garbage collector](docs/04-cpu-limits-and-gc.md)
|
||||
5. [Graceful shutdown and the rolling-update race](docs/05-graceful-shutdown.md)
|
||||
6. [HPA on a custom Micrometer metric](docs/06-hpa-custom-metrics.md)
|
||||
|
||||
## Captured output
|
||||
|
||||
| File | Produced by |
|
||||
|---|---|
|
||||
| `probes-*.txt` | `scripts/demo-probes.sh` |
|
||||
| [`startup-probe.txt`](docs/output/startup-probe.txt) | `scripts/demo-startup.sh` |
|
||||
| [`jvm-ergonomics.txt`](docs/output/jvm-ergonomics.txt) | `scripts/demo-ergonomics.sh` |
|
||||
| [`gc-throttling.txt`](docs/output/gc-throttling.txt) + raw GC logs | `scripts/demo-gc-throttling.sh` |
|
||||
| [`shutdown-post-summary.txt`](docs/output/shutdown-post-summary.txt), `shutdown-*.txt` | `scripts/demo-shutdown.sh` (`METHOD=GET` for the `-get-` files) |
|
||||
| [`hpa-custom-metric.txt`](docs/output/hpa-custom-metric.txt), [`hpa-custom-metrics-api.txt`](docs/output/hpa-custom-metrics-api.txt) | `scripts/demo-hpa.sh` |
|
||||
@@ -0,0 +1,14 @@
|
||||
# The image every Kubernetes experiment runs: the layered, distroless variant recommended in the
|
||||
# Docker article (spring-boot-demo/docker-images). No shell - which matters for preStop hooks.
|
||||
FROM eclipse-temurin:25-jre AS builder
|
||||
WORKDIR /builder
|
||||
COPY target/app.jar application.jar
|
||||
RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted
|
||||
|
||||
FROM gcr.io/distroless/java25-debian13:nonroot
|
||||
WORKDIR /application
|
||||
COPY --from=builder /builder/extracted/dependencies/ ./
|
||||
COPY --from=builder /builder/extracted/spring-boot-loader/ ./
|
||||
COPY --from=builder /builder/extracted/snapshot-dependencies/ ./
|
||||
COPY --from=builder /builder/extracted/application/ ./
|
||||
ENTRYPOINT ["java", "-jar", "application.jar"]
|
||||
@@ -0,0 +1,47 @@
|
||||
# 1. The lab
|
||||
|
||||
[Index](../README.md) · Next: [2. Probes →](02-probes.md)
|
||||
|
||||
Every transcript in [`output/`](output) came from a single-node **k3s v1.36.4+k3s1** cluster on a
|
||||
2-vCPU, 8 GB Ubuntu 24.04 VM. Small on purpose: CPU contention is the subject of two chapters, and
|
||||
on a 64-core node it is much harder to see.
|
||||
|
||||
## Reproducing it
|
||||
|
||||
Any cluster works - kind, minikube, k3d, a real one. What the scripts assume:
|
||||
|
||||
- namespace `demo` ([`k8s/00-namespace.yaml`](../k8s/00-namespace.yaml))
|
||||
- the image `sbd/k8s-demo:1` loaded into the cluster's container runtime, pulled with
|
||||
`imagePullPolicy: Never`. Build it with `mvn -DskipTests package && docker build -f docker/Dockerfile -t sbd/k8s-demo:1 .`,
|
||||
then `k3s ctr -n k8s.io images import`, `kind load docker-image`, or `minikube image load`
|
||||
- `busybox:1.37` available the same way (the `downstream` service and the `client` pod)
|
||||
- for [chapter 6](06-hpa-custom-metrics.md): `quay.io/prometheus/prometheus:v3.14.0` and
|
||||
`registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0`
|
||||
|
||||
The image is the layered, distroless one recommended in the
|
||||
[Docker article](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/):
|
||||
no shell, which matters in [chapter 5](05-graceful-shutdown.md).
|
||||
|
||||
## Two things specific to this lab
|
||||
|
||||
**The node runs cgroup v1.** Kubernetes 1.35 changed the kubelet's `failCgroupV1` default to `true`
|
||||
(KEP-5573), so on a v1 host the kubelet refuses to start unless told otherwise. The lab passes
|
||||
`--kubelet-arg=fail-cgroupv1=false`. Production nodes should be cgroup v2; [chapter 3](03-jvm-ergonomics.md)
|
||||
lists the file names that differ. The CFS bandwidth controller that throttles CPU is the same in both.
|
||||
|
||||
**The VM's process had no `CAP_SYS_RESOURCE`**, so runc could not lower `oom_score_adj` for the pod
|
||||
sandboxes and every pod failed with `failed to update /proc/self/oom_score_adj: Permission denied`.
|
||||
containerd's `restrict_oom_score_adj = true` (a drop-in under
|
||||
`/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/`) fixes it - the same setting rootless
|
||||
setups use.
|
||||
|
||||
**The kubelet garbage-collected images it was about to need.** The VM's disk is shared with its
|
||||
host and reported 94 % used, above the kubelet's default `image-gc-high-threshold` of 85 %. The
|
||||
kubelet deleted every image no running container used - the Prometheus and adapter images among
|
||||
them - and their pods failed with `ErrImageNeverPull`. With `imagePullPolicy: Never` there is no
|
||||
registry to fall back on. The lab now passes `image-gc-high-threshold=100`,
|
||||
`image-gc-low-threshold=99` and a 1 GiB `eviction-hard` threshold; on a real node, fix the disk.
|
||||
|
||||
And one mistake worth not repeating: starting k3s from a shell with `HTTPS_PROXY` set makes the API
|
||||
server route its own connections to the kubelet (`kubectl logs`, `exec`) through that proxy, where
|
||||
they fail. Start it with the proxy variables unset.
|
||||
@@ -0,0 +1,104 @@
|
||||
# 2. Probes: which checks belong in which probe
|
||||
|
||||
[← 1. The lab](01-the-lab.md) · [Index](../README.md) · Next: [3. JVM ergonomics →](03-jvm-ergonomics.md)
|
||||
|
||||
Spring Boot exposes two health groups for Kubernetes, `/actuator/health/liveness` and
|
||||
`/actuator/health/readiness`. They are added automatically when Boot detects Kubernetes (the
|
||||
`*_SERVICE_HOST` / `*_SERVICE_PORT` variables), and explicitly here with
|
||||
`management.endpoint.health.probes.enabled=true` so they also exist on a laptop. By default each
|
||||
contains exactly one thing - the application's `LivenessState` and `ReadinessState` - and **nothing
|
||||
else**, not the database, not `diskSpace`, not your own indicators.
|
||||
|
||||
The three probes answer different questions, and the kubelet does something different on failure:
|
||||
|
||||
| Probe | Question | On failure |
|
||||
|---|---|---|
|
||||
| startup | has it finished starting? | kill after `failureThreshold × periodSeconds`; liveness/readiness wait until it passes |
|
||||
| liveness | is this process broken beyond repair? | **kill and restart the container** |
|
||||
| readiness | should it get traffic right now? | remove from the Service; no restart |
|
||||
|
||||
## One dependency outage, three configurations
|
||||
|
||||
[`DownstreamHealthIndicator`](../src/main/java/com/ankurm/k8s/health/DownstreamHealthIndicator.java)
|
||||
calls a `downstream` service. [`demo-probes.sh`](../scripts/demo-probes.sh) scales the downstream to
|
||||
zero for 60 s and watches the two `orders` pods, the ready endpoints, and whether `GET /work` - an
|
||||
endpoint that **does not use the downstream at all** - still works through the Service.
|
||||
|
||||
**In the liveness group** ([`probes-liveness-includes-downstream.txt`](output/probes-liveness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
Both replicas were killed and restarted together - three failed liveness probes, 15 s - and the
|
||||
restarted containers could not pass their startup probe (it checks the same group) until the
|
||||
downstream came back. A dependency outage became a full outage of a service that did not need the
|
||||
dependency, and the JVMs lost their warm caches and JIT state on top. Had the outage outlasted the
|
||||
startup probe's 120 s budget, the pods would have been restarted again, into `CrashLoopBackOff`.
|
||||
|
||||
**In the readiness group** ([`probes-readiness-includes-downstream.txt`](output/probes-readiness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
No restarts, and recovery within seconds of the downstream returning. **But still zero serving
|
||||
endpoints within 6 s** - every replica checks the same shared dependency, so every replica leaves
|
||||
the Service at once. Clients get connection refused instead of a meaningful error, including for
|
||||
the requests that never needed the downstream.
|
||||
|
||||
**In neither group** - Spring Boot's default ([`probes-default-groups.txt`](output/probes-default-groups.txt)):
|
||||
|
||||
```
|
||||
t= 23s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
The pods stay in service. The requests that need the downstream fail - with whatever error your
|
||||
code returns, which can be a precise 503 - and the rest keep working.
|
||||
|
||||
## The rule this gives
|
||||
|
||||
- **Liveness**: only things a restart would fix. Boot's default `livenessState` is usually exactly
|
||||
right. A shared dependency in liveness restarts every replica at once.
|
||||
- **Readiness**: only things specific to *this* instance - warming a local cache, a full local
|
||||
queue. A shared dependency in readiness takes every replica out at once.
|
||||
- **Shared dependencies** belong in the request path: timeouts, a circuit breaker, a fast 503.
|
||||
Keep them in `/actuator/health` for dashboards - just not in a probe group.
|
||||
|
||||
## Slow starts: the startup probe
|
||||
|
||||
[`demo-startup.sh`](../scripts/demo-startup.sh) makes the context take ~45 s to refresh
|
||||
(`DEMO_STARTUP_DELAY=40s`, a sleep in a `@PostConstruct`) and rolls it out to one replica, first
|
||||
without a startup probe ([`startup-probe.txt`](output/startup-probe.txt)):
|
||||
|
||||
```
|
||||
t= 44s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 50s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 88s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=118s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
```
|
||||
|
||||
Liveness (every 5 s, three failures) starts failing the moment the container starts, because
|
||||
Tomcat is not listening yet (`connect: connection refused` in the events). The new pod is killed
|
||||
before it can ever finish starting, over and over. The first kill only *shows* as a restart at
|
||||
t=50 s because the kill runs the 5 s `preStop` sleep and then waits for the JVM to exit.
|
||||
|
||||
The old pod `7ltj7` keeps serving throughout: with `maxUnavailable: 0` the rollout simply stalls
|
||||
until `progressDeadlineSeconds` (600 s) marks it failed. With more replicas and a
|
||||
`maxUnavailable` above zero, the same change takes capacity away with every attempt.
|
||||
|
||||
The same rollout with a startup probe:
|
||||
|
||||
```
|
||||
t= 45s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 51s [jthjg ready=true restarts=0]
|
||||
```
|
||||
|
||||
Ready at ~51 s, no restarts, and liveness only begins once the startup probe has passed. Size the
|
||||
startup budget (`failureThreshold × periodSeconds`, 120 s here) for your slowest start - a cold
|
||||
node, a slow config server - and keep liveness tight.
|
||||
@@ -0,0 +1,60 @@
|
||||
# 3. JVM ergonomics: what the JVM decides from your pod spec
|
||||
|
||||
[← 2. Probes](02-probes.md) · [Index](../README.md) · Next: [4. CPU limits and GC →](04-cpu-limits-and-gc.md)
|
||||
|
||||
With no JVM options at all, the JVM reads the container's cgroup limits and picks a processor
|
||||
count, a garbage collector, a heap size and thread counts. [`demo-ergonomics.sh`](../scripts/demo-ergonomics.sh)
|
||||
runs `java -XX:+PrintFlagsFinal -version` in a pod per resource shape
|
||||
([`jvm-ergonomics.txt`](output/jvm-ergonomics.txt)):
|
||||
|
||||
```
|
||||
cpu.req cpu.lim mem.req mem.lim | CPUs GC MaxHeap PGCThr CGCThr JITThr
|
||||
100m none 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 500m 256Mi 512Mi | 1 SerialGC 128M 0 0 2
|
||||
100m 1 256Mi 1Gi | 1 SerialGC 256M 0 0 2
|
||||
100m 1 256Mi 2Gi | 1 SerialGC 512M 0 0 2
|
||||
100m 1500m 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 2 256Mi 1Gi | 2 SerialGC 256M 0 0 2
|
||||
100m 2 256Mi 1700Mi | 2 SerialGC 426M 0 0 2
|
||||
100m 2 256Mi 1800Mi | 2 G1GC 450M 2 1 2
|
||||
100m 2 256Mi 4Gi | 2 G1GC 1024M 2 1 2
|
||||
```
|
||||
|
||||
## The rules the table follows
|
||||
|
||||
- **CPUs = the limit, rounded up.** 500m → 1, 1500m → 2. With no limit, the JVM sees every CPU on
|
||||
the node - here 2, on a production node perhaps 64.
|
||||
- **Requests are ignored.** Every row has a 100m request. JDK 19 removed CPU shares from the
|
||||
calculation (JDK-8281181), so on a modern JDK `requests.cpu` affects scheduling and nothing the
|
||||
JVM decides.
|
||||
- **G1 needs 2 CPUs *and* about 1792 MB.** Below either, the JVM is not a "server-class machine"
|
||||
and picks SerialGC: a single-threaded, stop-the-world collector. 2 CPUs with 1700Mi → Serial;
|
||||
1800Mi → G1. A typical Spring Boot pod sized 512Mi-1Gi therefore **runs SerialGC
|
||||
without anyone having chosen it**.
|
||||
- **Heap = 25 % of the memory limit** (`MaxRAMPercentage=25`). A 512Mi pod gets a 128 MB heap and
|
||||
384 MB for metaspace, code cache, thread stacks and native memory - usually far more than they use. `-XX:MaxRAMPercentage=75` is the usual
|
||||
correction; the manifests here set it through `JAVA_TOOL_OPTIONS`.
|
||||
- **1500m gives G1 two parallel GC threads on one and a half CPUs of quota.** That combination is
|
||||
what [chapter 4](04-cpu-limits-and-gc.md) measures.
|
||||
|
||||
## cgroup v1 here, v2 in production
|
||||
|
||||
The lab node is cgroup v1 (a limitation of the VM it ran in; Kubernetes 1.35+ refuses v1 unless
|
||||
`failCgroupV1=false`). JVM container detection works the same on both. The files differ:
|
||||
|
||||
| | cgroup v1 | cgroup v2 |
|
||||
|---|---|---|
|
||||
| CPU limit | `cpu/cpu.cfs_quota_us` / `cpu.cfs_period_us` | `cpu.max` (`150000 100000`) |
|
||||
| throttling | `cpu/cpu.stat`: `nr_throttled`, `throttled_time` (ns) | `cpu.stat`: `nr_throttled`, `throttled_usec` |
|
||||
| memory limit | `memory/memory.limit_in_bytes` | `memory.max` |
|
||||
|
||||
[`JvmController`](../src/main/java/com/ankurm/k8s/diag/JvmController.java) reads either.
|
||||
`java -XshowSettings:system -version` prints what the JVM detected, including the provider:
|
||||
|
||||
```
|
||||
Operating System Metrics:
|
||||
Provider: cgroupv1
|
||||
Effective CPU Count: 2
|
||||
CPU Period: 100000us
|
||||
CPU Quota: 150000us
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# 4. CPU limits and the garbage collector
|
||||
|
||||
[← 3. JVM ergonomics](03-jvm-ergonomics.md) · [Index](../README.md) · Next: [5. Graceful shutdown →](05-graceful-shutdown.md)
|
||||
|
||||
A CPU limit in Kubernetes is a CFS bandwidth quota: `limits.cpu: 1` means 100 ms of CPU time per
|
||||
100 ms period, *shared by every thread in the container*. When the container has used its quota,
|
||||
all its threads stop until the next period. Not slow down - stop. That includes the garbage
|
||||
collector in the middle of a stop-the-world pause.
|
||||
|
||||
## The experiment
|
||||
|
||||
[`demo-gc-throttling.sh`](../scripts/demo-gc-throttling.sh) runs one pod
|
||||
([`k8s/gc-lab.yaml`](../k8s/gc-lab.yaml)) under five resource shapes and drives it for 60 s with 4
|
||||
clients calling `/alloc?mb=16` - allocation-heavy work with a slowly churning old generation. It
|
||||
reads GC pauses from `-Xlog:gc,gc+cpu`, CFS throttling from `cpu.stat` before and after, and
|
||||
latency from the load generator ([`gc-throttling.txt`](output/gc-throttling.txt)):
|
||||
|
||||
```
|
||||
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
|
||||
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
|
||||
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
|
||||
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
|
||||
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
|
||||
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
|
||||
```
|
||||
|
||||
And how much CPU the collector actually got while its pauses were running
|
||||
([`gc-cpu-ratio.txt`](output/gc-cpu-ratio.txt)):
|
||||
|
||||
```
|
||||
collections user+sys real cpu/real
|
||||
A 2189 5.32s 10.71s 0.50
|
||||
B 4279 11.12s 11.12s 1.00
|
||||
C 1783 9.24s 10.11s 0.91
|
||||
D 1960 12.03s 8.69s 1.38
|
||||
E 1878 12.58s 8.59s 1.46
|
||||
```
|
||||
|
||||
`-Xlog:gc+cpu` reports in 10 ms steps, so a single collection's ratio means little; summed over
|
||||
two thousand collections, it is a fair measure of how much of each pause the collector spent
|
||||
waiting for CPU.
|
||||
|
||||
## What it says
|
||||
|
||||
- **At 500m the container spent 37 of 60 seconds throttled**, in 97 % of CFS periods. The single
|
||||
Serial GC thread got CPU for only half the wall time of its own pauses (`cpu/real` 0.50): every
|
||||
pause was stretched to double by the quota, and the longest reached 188 ms. p99 latency: 200 ms.
|
||||
- **"Fixing" a 1-CPU pod by forcing G1 made it worse.** `-XX:ActiveProcessorCount=2 -XX:+UseG1GC`
|
||||
is a common recommendation for getting G1 on a small pod. On a 1-CPU quota, its two parallel GC
|
||||
threads plus concurrent marking burn the quota twice as fast: throttled in **92 %** of periods
|
||||
(against 33 % for Serial on the same quota), **32 % fewer requests** (10,463 vs 15,279), p99 107 ms
|
||||
vs 79 ms. Two GC threads received 0.91 CPU-seconds per pause-second - they were taking turns.
|
||||
- **1.5 CPUs is the interesting row.** The JVM rounds up to 2 processors, so G1 gets 2 threads on 1.5
|
||||
CPUs of quota - throttled in 36 % of periods, yet the best throughput here. Throttling is not
|
||||
automatically a disaster; it is a cost that grows as the quota shrinks relative to the threads
|
||||
that want to run at once.
|
||||
- **D and E are within noise of each other.** The node has exactly 2 CPUs, shared with k3s and
|
||||
the load generator, so a 2-CPU limit mostly meant "no limit". On a bigger node, E would pull ahead.
|
||||
|
||||
## What to do about it
|
||||
|
||||
1. **Know what you are running.** A 1-CPU, 1 GiB pod runs SerialGC ([chapter 3](03-jvm-ergonomics.md)).
|
||||
That can be the right collector for it - B beat C - but it should be a decision.
|
||||
2. **Do not raise `ActiveProcessorCount` above the quota** to get a "better" collector. You buy
|
||||
more threads competing for the same 100 ms.
|
||||
3. **Consider no CPU limit at all**, with a CPU request sized for steady state. Requests guarantee a
|
||||
share under contention and never throttle; limits throttle even on an idle node. Keep the memory
|
||||
limit - memory is not compressible.
|
||||
4. **Watch throttling, not just CPU usage.** `container_cpu_cfs_throttled_periods_total` from
|
||||
cAdvisor (or `nr_throttled` in `cpu.stat`) is the signal; a pod can show 40 % average CPU while
|
||||
being throttled in most periods.
|
||||
|
||||
These are single 60 s runs on a small node: the directions are robust, the exact numbers are not.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 5. Graceful shutdown and the rolling-update race
|
||||
|
||||
[← 4. CPU limits and GC](04-cpu-limits-and-gc.md) · [Index](../README.md) · Next: [6. HPA on a custom metric →](06-hpa-custom-metrics.md)
|
||||
|
||||
When Kubernetes deletes a pod, two things start **at the same time**: the kubelet runs the
|
||||
`preStop` hook and then sends SIGTERM, and the endpoints controller removes the pod from the
|
||||
Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short
|
||||
window a pod that is shutting down still receives new connections.
|
||||
|
||||
Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - `server.shutdown=graceful`) handles
|
||||
the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish,
|
||||
for up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s).
|
||||
|
||||
## The experiment
|
||||
|
||||
[`demo-shutdown.sh`](../scripts/demo-shutdown.sh): 20 closed-loop clients send `POST /work?ms=300`
|
||||
through the Service for 45 s; at t≈8 s, `kubectl rollout restart` replaces both replicas. Each
|
||||
variant three times ([`shutdown-post-summary.txt`](output/shutdown-post-summary.txt)):
|
||||
|
||||
```
|
||||
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
|
||||
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
|
||||
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
|
||||
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
|
||||
```
|
||||
|
||||
Reading it by failure type:
|
||||
|
||||
- **`IOException`** is a request cut off mid-flight, or sent on a keep-alive connection the server
|
||||
closed. `server.shutdown=immediate` produces 24-26 per restart; graceful shutdown cuts that to a
|
||||
handful.
|
||||
- **`ConnectException`** is a new connection to a pod that has already stopped listening but is
|
||||
still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6).
|
||||
**A 5 s `preStop: sleep` removed it in all three runs**: the pod keeps serving while the endpoint
|
||||
removal propagates, and only then gets SIGTERM.
|
||||
- **Variant 4 is variant 2 with extra steps.** An `exec` hook running `sh -c "sleep 5"` fails on
|
||||
the distroless image because there is no `sh`; Kubernetes records `FailedPreStopHook`, sends
|
||||
SIGTERM immediately, and the connect errors come back (4, 5, 7).
|
||||
|
||||
```
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
|
||||
```
|
||||
|
||||
## Why the experiment uses POST
|
||||
|
||||
The first version used GET and reported far fewer failures - 7 for `immediate`
|
||||
([`shutdown-get-1-immediate-no-prestop.txt`](output/shutdown-get-1-immediate-no-prestop.txt)), all
|
||||
`ConnectException`, against 26-32 with POST. The JDK `HttpClient` quietly retries an idempotent
|
||||
request whose connection was closed under it, so a GET load test hides exactly the failures this
|
||||
experiment is looking for. Your clients may or may not retry; a POST shows what the server did.
|
||||
|
||||
## The residue
|
||||
|
||||
Variant 3 still lost 0-2 requests out of ~2,930 per run, all `IOException`: a client reusing an idle
|
||||
keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it
|
||||
is why non-idempotent calls between services need retries with idempotency keys, whatever the
|
||||
deployment does.
|
||||
|
||||
## Sizing the grace period
|
||||
|
||||
`terminationGracePeriodSeconds` (default 30 s) covers the `preStop` hook **and** the shutdown
|
||||
after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s
|
||||
total. Keep `preStop` + `timeout-per-shutdown-phase` below the grace period, or raise it.
|
||||
@@ -0,0 +1,74 @@
|
||||
# 6. HPA on a custom Micrometer metric
|
||||
|
||||
[← 5. Graceful shutdown](05-graceful-shutdown.md) · [Index](../README.md)
|
||||
|
||||
CPU is a poor scaling signal for a JVM service: startup and JIT compilation burn CPU while serving
|
||||
nothing, GC competes with requests for the same quota ([chapter 4](04-cpu-limits-and-gc.md)), and a
|
||||
service waiting on I/O can be saturated at 20 % CPU. What you usually want to scale on is work in
|
||||
progress. The demo scales on in-flight requests.
|
||||
|
||||
## The pipeline
|
||||
|
||||
1. **Micrometer.** [`WorkController`](../src/main/java/com/ankurm/k8s/web/WorkController.java)
|
||||
registers a gauge `app.inflight.requests`. The Prometheus registry renames it: dots become
|
||||
underscores, a gauge gets no suffix (a counter would get `_total`).
|
||||
2. **Prometheus** scrapes `/actuator/prometheus` every 5 s ([`k8s/hpa/prometheus.yaml`](../k8s/hpa/prometheus.yaml))
|
||||
and attaches `namespace` and `pod` labels from Kubernetes service discovery.
|
||||
3. **prometheus-adapter** turns the series into the `custom.metrics.k8s.io` API with one rule
|
||||
([`k8s/hpa/prometheus-adapter.yaml`](../k8s/hpa/prometheus-adapter.yaml)). The
|
||||
`resources.overrides` block is what maps the `pod` label to a Kubernetes Pod - without it the
|
||||
metric exists in Prometheus and never appears in the API.
|
||||
4. **The HPA** ([`k8s/hpa/hpa.yaml`](../k8s/hpa/hpa.yaml)) targets an average of 5 per pod.
|
||||
|
||||
What each end sees ([`hpa-custom-metrics-api.txt`](output/hpa-custom-metrics-api.txt)):
|
||||
|
||||
```
|
||||
app_inflight_requests{application="orders"} 0.0
|
||||
```
|
||||
|
||||
```
|
||||
"metricName": "app_inflight_requests",
|
||||
"timestamp": "2026-09-11T16:33:28Z",
|
||||
"value": "0",
|
||||
```
|
||||
|
||||
## The run
|
||||
|
||||
2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms
|
||||
([`hpa-custom-metric.txt`](output/hpa-custom-metric.txt)):
|
||||
|
||||
```
|
||||
t clients metric desired ready pods
|
||||
0s 2 0 1 1
|
||||
22s 30 333m 1 1
|
||||
32s 30 6333m 2 2
|
||||
43s 30 10333m 4 2
|
||||
54s 30 10333m 4 4
|
||||
64s 30 7475m 4 4
|
||||
148s 0 6416m 4 4
|
||||
169s 0 2499m 4 4
|
||||
179s 0 0 2 2
|
||||
200s 0 0 1 1
|
||||
```
|
||||
|
||||
- **Reaction: ~12 s** from load to the first scale-up - a 5 s scrape, the adapter's query, and the
|
||||
HPA controller's 15 s sync period.
|
||||
- **It overshoots on purpose.** At t=43 the metric was 10.3 against a target of 5 on 2 pods, so the
|
||||
HPA asked for 5 and was capped at `maxReplicas: 4`. 30 clients over 4 pods settles at ~7.5 - still
|
||||
above target, which is what a ceiling looks like.
|
||||
- **Scale-down took 40-60 s after the load stopped**, because the adapter rule averages over 30 s
|
||||
and the HPA's scale-down stabilization window (shortened to 30 s here; **the default is 300 s**)
|
||||
holds the highest recommendation it saw.
|
||||
- **7,209 requests, 0 failures** through two scale-downs - the `preStop` sleep from
|
||||
[chapter 5](05-graceful-shutdown.md) doing its job.
|
||||
|
||||
## Choices in the rule worth knowing about
|
||||
|
||||
- `metricsQuery: avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s])` smooths a gauge that jumps
|
||||
with every request. Without it the HPA chases noise; with too long a window it reacts late.
|
||||
- For a **counter** (say `http_server_requests_seconds_count`) the query must be a `rate(...)`, and
|
||||
the adapter's documented examples rename `..._total` series to `..._per_second` - so the name the
|
||||
HPA asks for is not the name Micrometer exported.
|
||||
- prometheus-adapter 0.12.0 (May 2024) is still its latest release. KEDA's Prometheus scaler is the
|
||||
common alternative: it ships its own metrics API server, reads PromQL directly, and can scale to
|
||||
zero.
|
||||
@@ -0,0 +1,8 @@
|
||||
# CPU the collector received during its pauses, from -Xlog:gc+cpu (sum over every collection)
|
||||
|
||||
collections user+sys real cpu/real
|
||||
A 2189 5.32s 10.71s 0.50
|
||||
B 4279 11.12s 11.12s 1.00
|
||||
C 1783 9.24s 10.11s 0.91
|
||||
D 1960 12.03s 8.69s 1.38
|
||||
E 1878 12.58s 8.59s 1.46
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=28
|
||||
t= 1s conc=4 ok=44
|
||||
t= 2s conc=4 ok=45
|
||||
t= 3s conc=4 ok=51
|
||||
t= 4s conc=4 ok=69
|
||||
t= 5s conc=4 ok=60
|
||||
t= 6s conc=4 ok=62
|
||||
t= 7s conc=4 ok=76
|
||||
t= 8s conc=4 ok=76
|
||||
t= 9s conc=4 ok=90
|
||||
t= 10s conc=4 ok=29
|
||||
t= 11s conc=4 ok=96
|
||||
t= 12s conc=4 ok=116
|
||||
t= 13s conc=4 ok=105
|
||||
t= 14s conc=4 ok=107
|
||||
t= 15s conc=4 ok=109
|
||||
t= 16s conc=4 ok=97
|
||||
t= 17s conc=4 ok=96
|
||||
t= 18s conc=4 ok=92
|
||||
t= 19s conc=4 ok=83
|
||||
t= 20s conc=4 ok=127
|
||||
t= 21s conc=4 ok=110
|
||||
t= 22s conc=4 ok=93
|
||||
t= 23s conc=4 ok=94
|
||||
t= 24s conc=4 ok=118
|
||||
t= 25s conc=4 ok=126
|
||||
t= 26s conc=4 ok=133
|
||||
t= 27s conc=4 ok=133
|
||||
t= 28s conc=4 ok=111
|
||||
t= 29s conc=4 ok=120
|
||||
t= 30s conc=4 ok=117
|
||||
t= 31s conc=4 ok=138
|
||||
t= 32s conc=4 ok=116
|
||||
t= 33s conc=4 ok=92
|
||||
t= 34s conc=4 ok=124
|
||||
t= 35s conc=4 ok=146
|
||||
t= 36s conc=4 ok=107
|
||||
t= 37s conc=4 ok=159
|
||||
t= 38s conc=4 ok=133
|
||||
t= 39s conc=4 ok=151
|
||||
t= 40s conc=4 ok=162
|
||||
t= 41s conc=4 ok=169
|
||||
t= 42s conc=4 ok=129
|
||||
t= 43s conc=4 ok=169
|
||||
t= 44s conc=4 ok=171
|
||||
t= 45s conc=4 ok=159
|
||||
t= 46s conc=4 ok=140
|
||||
t= 47s conc=4 ok=112
|
||||
t= 48s conc=4 ok=108
|
||||
t= 49s conc=4 ok=96
|
||||
t= 50s conc=4 ok=127
|
||||
t= 51s conc=4 ok=100
|
||||
t= 52s conc=4 ok=59
|
||||
t= 53s conc=4 ok=140
|
||||
t= 54s conc=4 ok=85
|
||||
t= 55s conc=4 ok=81
|
||||
t= 56s conc=4 ok=153
|
||||
t= 57s conc=4 ok=115
|
||||
t= 58s conc=4 ok=114
|
||||
t= 59s conc=4 ok=87
|
||||
t= 60s conc=4 ok=14
|
||||
METHOD GET
|
||||
TOTAL {ok=6469}
|
||||
LATENCY ms p50=15 p90=84 p99=200 max=805 (n=6469)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=78
|
||||
t= 1s conc=4 ok=110
|
||||
t= 2s conc=4 ok=131
|
||||
t= 3s conc=4 ok=144
|
||||
t= 4s conc=4 ok=123
|
||||
t= 5s conc=4 ok=78
|
||||
t= 6s conc=4 ok=241
|
||||
t= 7s conc=4 ok=218
|
||||
t= 8s conc=4 ok=195
|
||||
t= 9s conc=4 ok=183
|
||||
t= 10s conc=4 ok=272
|
||||
t= 11s conc=4 ok=224
|
||||
t= 12s conc=4 ok=243
|
||||
t= 13s conc=4 ok=317
|
||||
t= 14s conc=4 ok=235
|
||||
t= 15s conc=4 ok=280
|
||||
t= 16s conc=4 ok=249
|
||||
t= 17s conc=4 ok=248
|
||||
t= 18s conc=4 ok=311
|
||||
t= 19s conc=4 ok=335
|
||||
t= 20s conc=4 ok=286
|
||||
t= 21s conc=4 ok=311
|
||||
t= 22s conc=4 ok=273
|
||||
t= 23s conc=4 ok=232
|
||||
t= 24s conc=4 ok=204
|
||||
t= 25s conc=4 ok=269
|
||||
t= 26s conc=4 ok=312
|
||||
t= 27s conc=4 ok=254
|
||||
t= 28s conc=4 ok=255
|
||||
t= 29s conc=4 ok=249
|
||||
t= 30s conc=4 ok=262
|
||||
t= 31s conc=4 ok=302
|
||||
t= 32s conc=4 ok=243
|
||||
t= 33s conc=4 ok=312
|
||||
t= 34s conc=4 ok=335
|
||||
t= 35s conc=4 ok=323
|
||||
t= 36s conc=4 ok=253
|
||||
t= 37s conc=4 ok=320
|
||||
t= 38s conc=4 ok=226
|
||||
t= 39s conc=4 ok=292
|
||||
t= 40s conc=4 ok=202
|
||||
t= 41s conc=4 ok=180
|
||||
t= 42s conc=4 ok=311
|
||||
t= 43s conc=4 ok=299
|
||||
t= 44s conc=4 ok=231
|
||||
t= 45s conc=4 ok=314
|
||||
t= 46s conc=4 ok=304
|
||||
t= 47s conc=4 ok=265
|
||||
t= 48s conc=4 ok=295
|
||||
t= 49s conc=4 ok=319
|
||||
t= 50s conc=4 ok=278
|
||||
t= 51s conc=4 ok=211
|
||||
t= 52s conc=4 ok=271
|
||||
t= 53s conc=4 ok=319
|
||||
t= 54s conc=4 ok=285
|
||||
t= 55s conc=4 ok=243
|
||||
t= 56s conc=4 ok=306
|
||||
t= 57s conc=4 ok=285
|
||||
t= 58s conc=4 ok=313
|
||||
t= 59s conc=4 ok=288
|
||||
t= 60s conc=4 ok=32
|
||||
METHOD GET
|
||||
TOTAL {ok=15279}
|
||||
LATENCY ms p50=12 p90=25 p99=79 max=447 (n=15279)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=54
|
||||
t= 1s conc=4 ok=85
|
||||
t= 2s conc=4 ok=106
|
||||
t= 3s conc=4 ok=106
|
||||
t= 4s conc=4 ok=119
|
||||
t= 5s conc=4 ok=116
|
||||
t= 6s conc=4 ok=117
|
||||
t= 7s conc=4 ok=127
|
||||
t= 8s conc=4 ok=117
|
||||
t= 9s conc=4 ok=116
|
||||
t= 10s conc=4 ok=131
|
||||
t= 11s conc=4 ok=135
|
||||
t= 12s conc=4 ok=106
|
||||
t= 13s conc=4 ok=112
|
||||
t= 14s conc=4 ok=131
|
||||
t= 15s conc=4 ok=139
|
||||
t= 16s conc=4 ok=140
|
||||
t= 17s conc=4 ok=153
|
||||
t= 18s conc=4 ok=164
|
||||
t= 19s conc=4 ok=166
|
||||
t= 20s conc=4 ok=159
|
||||
t= 21s conc=4 ok=172
|
||||
t= 22s conc=4 ok=194
|
||||
t= 23s conc=4 ok=158
|
||||
t= 24s conc=4 ok=154
|
||||
t= 25s conc=4 ok=188
|
||||
t= 26s conc=4 ok=183
|
||||
t= 27s conc=4 ok=198
|
||||
t= 28s conc=4 ok=208
|
||||
t= 29s conc=4 ok=201
|
||||
t= 30s conc=4 ok=163
|
||||
t= 31s conc=4 ok=194
|
||||
t= 32s conc=4 ok=214
|
||||
t= 33s conc=4 ok=220
|
||||
t= 34s conc=4 ok=176
|
||||
t= 35s conc=4 ok=196
|
||||
t= 36s conc=4 ok=167
|
||||
t= 37s conc=4 ok=158
|
||||
t= 38s conc=4 ok=202
|
||||
t= 39s conc=4 ok=168
|
||||
t= 40s conc=4 ok=222
|
||||
t= 41s conc=4 ok=168
|
||||
t= 42s conc=4 ok=217
|
||||
t= 43s conc=4 ok=222
|
||||
t= 44s conc=4 ok=215
|
||||
t= 45s conc=4 ok=193
|
||||
t= 46s conc=4 ok=189
|
||||
t= 47s conc=4 ok=189
|
||||
t= 48s conc=4 ok=224
|
||||
t= 49s conc=4 ok=201
|
||||
t= 50s conc=4 ok=226
|
||||
t= 51s conc=4 ok=236
|
||||
t= 52s conc=4 ok=226
|
||||
t= 53s conc=4 ok=226
|
||||
t= 54s conc=4 ok=253
|
||||
t= 55s conc=4 ok=250
|
||||
t= 56s conc=4 ok=198
|
||||
t= 57s conc=4 ok=189
|
||||
t= 58s conc=4 ok=239
|
||||
t= 59s conc=4 ok=238
|
||||
t= 60s conc=4 ok=29
|
||||
METHOD GET
|
||||
TOTAL {ok=10463}
|
||||
LATENCY ms p50=13 p90=56 p99=107 max=588 (n=10463)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=125
|
||||
t= 1s conc=4 ok=239
|
||||
t= 2s conc=4 ok=246
|
||||
t= 3s conc=4 ok=257
|
||||
t= 4s conc=4 ok=266
|
||||
t= 5s conc=4 ok=269
|
||||
t= 6s conc=4 ok=357
|
||||
t= 7s conc=4 ok=346
|
||||
t= 8s conc=4 ok=205
|
||||
t= 9s conc=4 ok=228
|
||||
t= 10s conc=4 ok=217
|
||||
t= 11s conc=4 ok=216
|
||||
t= 12s conc=4 ok=244
|
||||
t= 13s conc=4 ok=228
|
||||
t= 14s conc=4 ok=269
|
||||
t= 15s conc=4 ok=295
|
||||
t= 16s conc=4 ok=297
|
||||
t= 17s conc=4 ok=441
|
||||
t= 18s conc=4 ok=417
|
||||
t= 19s conc=4 ok=325
|
||||
t= 20s conc=4 ok=366
|
||||
t= 21s conc=4 ok=446
|
||||
t= 22s conc=4 ok=371
|
||||
t= 23s conc=4 ok=468
|
||||
t= 24s conc=4 ok=375
|
||||
t= 25s conc=4 ok=306
|
||||
t= 26s conc=4 ok=434
|
||||
t= 27s conc=4 ok=405
|
||||
t= 28s conc=4 ok=369
|
||||
t= 29s conc=4 ok=338
|
||||
t= 30s conc=4 ok=374
|
||||
t= 31s conc=4 ok=441
|
||||
t= 32s conc=4 ok=485
|
||||
t= 33s conc=4 ok=286
|
||||
t= 34s conc=4 ok=539
|
||||
t= 35s conc=4 ok=413
|
||||
t= 36s conc=4 ok=380
|
||||
t= 37s conc=4 ok=437
|
||||
t= 38s conc=4 ok=283
|
||||
t= 39s conc=4 ok=378
|
||||
t= 40s conc=4 ok=485
|
||||
t= 41s conc=4 ok=463
|
||||
t= 42s conc=4 ok=501
|
||||
t= 43s conc=4 ok=527
|
||||
t= 44s conc=4 ok=502
|
||||
t= 45s conc=4 ok=442
|
||||
t= 46s conc=4 ok=539
|
||||
t= 47s conc=4 ok=549
|
||||
t= 48s conc=4 ok=504
|
||||
t= 49s conc=4 ok=529
|
||||
t= 50s conc=4 ok=503
|
||||
t= 51s conc=4 ok=512
|
||||
t= 52s conc=4 ok=551
|
||||
t= 53s conc=4 ok=384
|
||||
t= 54s conc=4 ok=411
|
||||
t= 55s conc=4 ok=562
|
||||
t= 56s conc=4 ok=555
|
||||
t= 57s conc=4 ok=566
|
||||
t= 58s conc=4 ok=479
|
||||
t= 59s conc=4 ok=524
|
||||
t= 60s conc=4 ok=48
|
||||
METHOD GET
|
||||
TOTAL {ok=23517}
|
||||
LATENCY ms p50=7 p90=18 p99=51 max=221 (n=23517)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
t= 0s conc=4 ok=109
|
||||
t= 1s conc=4 ok=199
|
||||
t= 2s conc=4 ok=229
|
||||
t= 3s conc=4 ok=257
|
||||
t= 4s conc=4 ok=304
|
||||
t= 5s conc=4 ok=290
|
||||
t= 6s conc=4 ok=337
|
||||
t= 7s conc=4 ok=320
|
||||
t= 8s conc=4 ok=314
|
||||
t= 9s conc=4 ok=204
|
||||
t= 10s conc=4 ok=191
|
||||
t= 11s conc=4 ok=228
|
||||
t= 12s conc=4 ok=231
|
||||
t= 13s conc=4 ok=256
|
||||
t= 14s conc=4 ok=247
|
||||
t= 15s conc=4 ok=273
|
||||
t= 16s conc=4 ok=245
|
||||
t= 17s conc=4 ok=237
|
||||
t= 18s conc=4 ok=316
|
||||
t= 19s conc=4 ok=371
|
||||
t= 20s conc=4 ok=329
|
||||
t= 21s conc=4 ok=331
|
||||
t= 22s conc=4 ok=408
|
||||
t= 23s conc=4 ok=349
|
||||
t= 24s conc=4 ok=240
|
||||
t= 25s conc=4 ok=297
|
||||
t= 26s conc=4 ok=323
|
||||
t= 27s conc=4 ok=270
|
||||
t= 28s conc=4 ok=438
|
||||
t= 29s conc=4 ok=414
|
||||
t= 30s conc=4 ok=461
|
||||
t= 31s conc=4 ok=400
|
||||
t= 32s conc=4 ok=371
|
||||
t= 33s conc=4 ok=476
|
||||
t= 34s conc=4 ok=429
|
||||
t= 35s conc=4 ok=460
|
||||
t= 36s conc=4 ok=380
|
||||
t= 37s conc=4 ok=309
|
||||
t= 38s conc=4 ok=425
|
||||
t= 39s conc=4 ok=478
|
||||
t= 40s conc=4 ok=404
|
||||
t= 41s conc=4 ok=482
|
||||
t= 42s conc=4 ok=438
|
||||
t= 43s conc=4 ok=404
|
||||
t= 44s conc=4 ok=523
|
||||
t= 45s conc=4 ok=447
|
||||
t= 46s conc=4 ok=471
|
||||
t= 47s conc=4 ok=405
|
||||
t= 48s conc=4 ok=516
|
||||
t= 49s conc=4 ok=444
|
||||
t= 50s conc=4 ok=388
|
||||
t= 51s conc=4 ok=515
|
||||
t= 52s conc=4 ok=459
|
||||
t= 53s conc=4 ok=489
|
||||
t= 54s conc=4 ok=505
|
||||
t= 55s conc=4 ok=538
|
||||
t= 56s conc=4 ok=486
|
||||
t= 57s conc=4 ok=477
|
||||
t= 58s conc=4 ok=492
|
||||
t= 59s conc=4 ok=489
|
||||
t= 60s conc=4 ok=63
|
||||
METHOD GET
|
||||
TOTAL {ok=22181}
|
||||
LATENCY ms p50=7 p90=20 p99=50 max=184 (n=22181)
|
||||
@@ -0,0 +1,10 @@
|
||||
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
|
||||
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
|
||||
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
|
||||
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
|
||||
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
|
||||
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
|
||||
|
||||
# thr = ParallelGCThreads. pauses/pause sum/max from the GC log during the 60 s run.
|
||||
# throttled = CFS periods in which the container hit its quota / periods elapsed, from cpu.stat.
|
||||
# requests = successful /alloc?mb=16 calls by 4 closed-loop clients in 60 s.
|
||||
@@ -0,0 +1,37 @@
|
||||
# HPA orders: target app_inflight_requests averageValue 5, min 1, max 4; scaleDown stabilization 30 s
|
||||
# load: t=0 2 clients, t=20 30 clients, t=140 0 clients; GET /work?ms=500
|
||||
|
||||
t clients metric desired ready pods
|
||||
0s 2 0 1 1
|
||||
11s 2 333m 1 1
|
||||
22s 30 333m 1 1
|
||||
32s 30 6333m 2 2
|
||||
43s 30 10333m 4 2
|
||||
54s 30 10333m 4 4
|
||||
64s 30 7475m 4 4
|
||||
75s 30 7416m 4 4
|
||||
85s 30 7124m 4 4
|
||||
96s 30 7124m 4 4
|
||||
106s 30 7083m 4 4
|
||||
117s 30 7333m 4 4
|
||||
127s 30 7333m 4 4
|
||||
138s 30 7583m 4 4
|
||||
148s 0 6416m 4 4
|
||||
159s 0 6416m 4 4
|
||||
169s 0 2499m 4 4
|
||||
179s 0 0 2 2
|
||||
190s 0 0 2 2
|
||||
200s 0 0 1 1
|
||||
211s 0 0 1 1
|
||||
221s 0 0 1 1
|
||||
231s 0 0 1 1
|
||||
|
||||
# HPA events:
|
||||
SuccessfulRescale New size: 2; reason: pods metric app_inflight_requests above target
|
||||
SuccessfulRescale New size: 4; reason: pods metric app_inflight_requests above target
|
||||
SuccessfulRescale New size: 2; reason: All metrics below target
|
||||
SuccessfulRescale New size: 1; reason: All metrics below target
|
||||
|
||||
# Load generator summary:
|
||||
TOTAL {ok=7209}
|
||||
LATENCY ms p50=503 p90=510 p99=567 max=698 (n=7209)
|
||||
@@ -0,0 +1,27 @@
|
||||
# What Micrometer exports (one pod):
|
||||
$ curl <pod>:8080/actuator/prometheus | grep app_inflight
|
||||
# HELP app_inflight_requests Requests currently being processed by /work
|
||||
# TYPE app_inflight_requests gauge
|
||||
app_inflight_requests{application="orders"} 0.0
|
||||
|
||||
# What the HPA controller sees through the custom metrics API:
|
||||
$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/demo/pods/%2A/app_inflight_requests
|
||||
{
|
||||
"kind": "MetricValueList",
|
||||
"apiVersion": "custom.metrics.k8s.io/v1beta1",
|
||||
"metadata": {},
|
||||
"items": [
|
||||
{
|
||||
"describedObject": {
|
||||
"kind": "Pod",
|
||||
"namespace": "demo",
|
||||
"name": "orders-7774f94779-gvc7k",
|
||||
"apiVersion": "/v1"
|
||||
},
|
||||
"metricName": "app_inflight_requests",
|
||||
"timestamp": "2026-09-11T16:33:28Z",
|
||||
"value": "0",
|
||||
"selector": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
t= 0s conc=2 ok=2
|
||||
t= 1s conc=2 ok=4
|
||||
t= 2s conc=2 ok=4
|
||||
t= 3s conc=2 ok=4
|
||||
t= 4s conc=2 ok=4
|
||||
t= 5s conc=2 ok=4
|
||||
t= 6s conc=2 ok=4
|
||||
t= 7s conc=2 ok=4
|
||||
t= 8s conc=2 ok=4
|
||||
t= 9s conc=2 ok=4
|
||||
t= 10s conc=2 ok=4
|
||||
t= 11s conc=2 ok=4
|
||||
t= 12s conc=2 ok=4
|
||||
t= 13s conc=2 ok=4
|
||||
t= 14s conc=2 ok=4
|
||||
t= 15s conc=2 ok=4
|
||||
t= 16s conc=2 ok=3
|
||||
t= 17s conc=2 ok=3
|
||||
t= 18s conc=2 ok=4
|
||||
t= 19s conc=2 ok=4
|
||||
t= 20s conc=30 ok=32
|
||||
t= 21s conc=30 ok=60
|
||||
t= 22s conc=30 ok=60
|
||||
t= 23s conc=30 ok=60
|
||||
t= 24s conc=30 ok=60
|
||||
t= 25s conc=30 ok=60
|
||||
t= 26s conc=30 ok=60
|
||||
t= 27s conc=30 ok=60
|
||||
t= 28s conc=30 ok=60
|
||||
t= 29s conc=30 ok=60
|
||||
t= 30s conc=30 ok=60
|
||||
t= 31s conc=30 ok=60
|
||||
t= 32s conc=30 ok=58
|
||||
t= 33s conc=30 ok=60
|
||||
t= 34s conc=30 ok=60
|
||||
t= 35s conc=30 ok=60
|
||||
t= 36s conc=30 ok=60
|
||||
t= 37s conc=30 ok=60
|
||||
t= 38s conc=30 ok=54
|
||||
t= 39s conc=30 ok=57
|
||||
t= 40s conc=30 ok=53
|
||||
t= 41s conc=30 ok=60
|
||||
t= 42s conc=30 ok=60
|
||||
t= 43s conc=30 ok=60
|
||||
t= 44s conc=30 ok=60
|
||||
t= 45s conc=30 ok=53
|
||||
t= 46s conc=30 ok=54
|
||||
t= 47s conc=30 ok=58
|
||||
t= 48s conc=30 ok=57
|
||||
t= 49s conc=30 ok=54
|
||||
t= 50s conc=30 ok=60
|
||||
t= 51s conc=30 ok=60
|
||||
t= 52s conc=30 ok=60
|
||||
t= 53s conc=30 ok=60
|
||||
t= 54s conc=30 ok=60
|
||||
t= 55s conc=30 ok=59
|
||||
t= 56s conc=30 ok=59
|
||||
t= 57s conc=30 ok=60
|
||||
t= 58s conc=30 ok=60
|
||||
t= 59s conc=30 ok=60
|
||||
t= 60s conc=30 ok=60
|
||||
t= 61s conc=30 ok=60
|
||||
t= 62s conc=30 ok=60
|
||||
t= 63s conc=30 ok=60
|
||||
t= 64s conc=30 ok=60
|
||||
t= 65s conc=30 ok=60
|
||||
t= 66s conc=30 ok=60
|
||||
t= 67s conc=30 ok=60
|
||||
t= 68s conc=30 ok=60
|
||||
t= 69s conc=30 ok=60
|
||||
t= 70s conc=30 ok=60
|
||||
t= 71s conc=30 ok=60
|
||||
t= 72s conc=30 ok=60
|
||||
t= 73s conc=30 ok=60
|
||||
t= 74s conc=30 ok=60
|
||||
t= 75s conc=30 ok=59
|
||||
t= 76s conc=30 ok=60
|
||||
t= 77s conc=30 ok=60
|
||||
t= 78s conc=30 ok=57
|
||||
t= 79s conc=30 ok=60
|
||||
t= 80s conc=30 ok=59
|
||||
t= 81s conc=30 ok=60
|
||||
t= 82s conc=30 ok=60
|
||||
t= 83s conc=30 ok=60
|
||||
t= 84s conc=30 ok=60
|
||||
t= 85s conc=30 ok=60
|
||||
t= 86s conc=30 ok=60
|
||||
t= 87s conc=30 ok=56
|
||||
t= 88s conc=30 ok=60
|
||||
t= 89s conc=30 ok=59
|
||||
t= 90s conc=30 ok=60
|
||||
t= 91s conc=30 ok=59
|
||||
t= 92s conc=30 ok=58
|
||||
t= 93s conc=30 ok=60
|
||||
t= 94s conc=30 ok=58
|
||||
t= 95s conc=30 ok=58
|
||||
t= 96s conc=30 ok=60
|
||||
t= 97s conc=30 ok=60
|
||||
t= 98s conc=30 ok=59
|
||||
t= 99s conc=30 ok=58
|
||||
t=100s conc=30 ok=60
|
||||
t=101s conc=30 ok=60
|
||||
t=102s conc=30 ok=59
|
||||
t=103s conc=30 ok=58
|
||||
t=104s conc=30 ok=60
|
||||
t=105s conc=30 ok=60
|
||||
t=106s conc=30 ok=60
|
||||
t=107s conc=30 ok=60
|
||||
t=108s conc=30 ok=60
|
||||
t=109s conc=30 ok=58
|
||||
t=110s conc=30 ok=60
|
||||
t=111s conc=30 ok=60
|
||||
t=112s conc=30 ok=60
|
||||
t=113s conc=30 ok=60
|
||||
t=114s conc=30 ok=60
|
||||
t=115s conc=30 ok=60
|
||||
t=116s conc=30 ok=60
|
||||
t=117s conc=30 ok=60
|
||||
t=118s conc=30 ok=60
|
||||
t=119s conc=30 ok=60
|
||||
t=120s conc=30 ok=60
|
||||
t=121s conc=30 ok=60
|
||||
t=122s conc=30 ok=60
|
||||
t=123s conc=30 ok=60
|
||||
t=124s conc=30 ok=59
|
||||
t=125s conc=30 ok=57
|
||||
t=126s conc=30 ok=60
|
||||
t=127s conc=30 ok=60
|
||||
t=128s conc=30 ok=60
|
||||
t=129s conc=30 ok=60
|
||||
t=130s conc=30 ok=60
|
||||
t=131s conc=30 ok=60
|
||||
t=132s conc=30 ok=60
|
||||
t=133s conc=30 ok=59
|
||||
t=134s conc=30 ok=60
|
||||
t=135s conc=30 ok=59
|
||||
t=136s conc=30 ok=60
|
||||
t=137s conc=30 ok=60
|
||||
t=138s conc=30 ok=60
|
||||
t=139s conc=30 ok=60
|
||||
t=140s conc=0 ok=36
|
||||
t=141s conc=0
|
||||
t=142s conc=0
|
||||
t=143s conc=0
|
||||
t=144s conc=0
|
||||
t=145s conc=0
|
||||
t=146s conc=0
|
||||
t=147s conc=0
|
||||
t=148s conc=0
|
||||
t=149s conc=0
|
||||
t=150s conc=0
|
||||
t=151s conc=0
|
||||
t=152s conc=0
|
||||
t=153s conc=0
|
||||
t=154s conc=0
|
||||
t=155s conc=0
|
||||
t=156s conc=0
|
||||
t=157s conc=0
|
||||
t=158s conc=0
|
||||
t=159s conc=0
|
||||
t=160s conc=0
|
||||
t=161s conc=0
|
||||
t=162s conc=0
|
||||
t=163s conc=0
|
||||
t=164s conc=0
|
||||
t=165s conc=0
|
||||
t=166s conc=0
|
||||
t=167s conc=0
|
||||
t=168s conc=0
|
||||
t=169s conc=0
|
||||
t=170s conc=0
|
||||
t=171s conc=0
|
||||
t=172s conc=0
|
||||
t=173s conc=0
|
||||
t=174s conc=0
|
||||
t=175s conc=0
|
||||
t=176s conc=0
|
||||
t=177s conc=0
|
||||
t=178s conc=0
|
||||
t=179s conc=0
|
||||
t=180s conc=0
|
||||
t=181s conc=0
|
||||
t=182s conc=0
|
||||
t=183s conc=0
|
||||
t=184s conc=0
|
||||
t=185s conc=0
|
||||
t=186s conc=0
|
||||
t=187s conc=0
|
||||
t=188s conc=0
|
||||
t=189s conc=0
|
||||
t=190s conc=0
|
||||
t=191s conc=0
|
||||
t=192s conc=0
|
||||
t=193s conc=0
|
||||
t=194s conc=0
|
||||
t=195s conc=0
|
||||
t=196s conc=0
|
||||
t=197s conc=0
|
||||
t=198s conc=0
|
||||
t=199s conc=0
|
||||
t=200s conc=0
|
||||
t=201s conc=0
|
||||
t=202s conc=0
|
||||
t=203s conc=0
|
||||
t=204s conc=0
|
||||
t=205s conc=0
|
||||
t=206s conc=0
|
||||
t=207s conc=0
|
||||
t=208s conc=0
|
||||
t=209s conc=0
|
||||
t=210s conc=0
|
||||
t=211s conc=0
|
||||
t=212s conc=0
|
||||
t=213s conc=0
|
||||
t=214s conc=0
|
||||
t=215s conc=0
|
||||
t=216s conc=0
|
||||
t=217s conc=0
|
||||
t=218s conc=0
|
||||
t=219s conc=0
|
||||
t=220s conc=0
|
||||
t=221s conc=0
|
||||
t=222s conc=0
|
||||
t=223s conc=0
|
||||
t=224s conc=0
|
||||
t=225s conc=0
|
||||
t=226s conc=0
|
||||
t=227s conc=0
|
||||
t=228s conc=0
|
||||
t=229s conc=0
|
||||
t=230s conc=0
|
||||
METHOD GET
|
||||
TOTAL {ok=7209}
|
||||
LATENCY ms p50=503 p90=510 p99=567 max=698 (n=7209)
|
||||
@@ -0,0 +1,34 @@
|
||||
# JVM ergonomics per pod resources. Image: sbd/k8s-demo:1, no JVM options. Node: 2 CPUs.
|
||||
# cgroup v1 node; see docs/03-jvm-ergonomics.md for the v2 equivalents.
|
||||
|
||||
cpu.req cpu.lim mem.req mem.lim | CPUs GC MaxHeap PGCThr CGCThr JITThr
|
||||
100m none 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 500m 256Mi 512Mi | 1 SerialGC 128M 0 0 2
|
||||
100m 1 256Mi 1Gi | 1 SerialGC 256M 0 0 2
|
||||
100m 1 256Mi 2Gi | 1 SerialGC 512M 0 0 2
|
||||
100m 1500m 256Mi 2Gi | 2 G1GC 512M 2 1 2
|
||||
100m 2 256Mi 1Gi | 2 SerialGC 256M 0 0 2
|
||||
100m 2 256Mi 1700Mi | 2 SerialGC 426M 0 0 2
|
||||
100m 2 256Mi 1800Mi | 2 G1GC 450M 2 1 2
|
||||
100m 2 256Mi 4Gi | 2 G1GC 1024M 2 1 2
|
||||
|
||||
# java -XshowSettings:system for the 1500m case:
|
||||
Operating System Metrics:
|
||||
Provider: cgroupv1
|
||||
Effective CPU Count: 2
|
||||
CPU Period: 100000us
|
||||
CPU Quota: 150000us
|
||||
CPU Shares: 102us
|
||||
List of Processors, 2 total:
|
||||
0 1
|
||||
List of Effective Processors, 2 total:
|
||||
0 1
|
||||
List of Memory Nodes, 1 total:
|
||||
0
|
||||
List of Available Memory Nodes, 1 total:
|
||||
0
|
||||
Memory Limit: 2.00G
|
||||
Memory Soft Limit: Unlimited
|
||||
Memory & Swap Limit: 2.00G
|
||||
Maximum Processes Limit: Unlimited
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# downstream health indicator in NEITHER group (Spring Boot's default probe groups)
|
||||
# health groups: liveness= readiness=
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s btcl4 ready=false restarts=0 terminated | 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 11s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 18s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 29s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 34s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 40s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 46s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 51s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 57s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
-------- downstream restored --------
|
||||
t= 62s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 69s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 75s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 86s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 97s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=103s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=109s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=114s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=120s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.207:8080/actuator/health/liveness": dial tcp 10.42.0.207:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.206:8080/actuator/health/liveness": dial tcp 10.42.0.206:8080: connect: connection refused
|
||||
@@ -0,0 +1,37 @@
|
||||
# downstream health indicator in the LIVENESS group
|
||||
# health groups: liveness=livenessState,downstream readiness=
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | gvc7k ready=false restarts=1 terminated | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 12s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 32s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 39s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 46s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 54s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 62s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 74s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 85s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 98s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=103s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=109s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=114s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=120s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=125s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=131s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Warning Unhealthy Startup probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Warning Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Normal Killing Stopping container app
|
||||
2 Normal Killing Container app failed liveness probe, will be restarted
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.201:8080/actuator/health/liveness": dial tcp 10.42.0.201:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.200:8080/actuator/health/liveness": dial tcp 10.42.0.200:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.199:8080/actuator/health/liveness": dial tcp 10.42.0.199:8080: connect: connection refused
|
||||
1 Warning Unhealthy Readiness probe failed: Get "http://10.42.0.201:8080/actuator/health/readiness": dial tcp 10.42.0.201:8080: connect: connection refused
|
||||
@@ -0,0 +1,33 @@
|
||||
# downstream health indicator in the READINESS group
|
||||
# health groups: liveness= readiness=readinessState,downstream
|
||||
# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1
|
||||
|
||||
t= 0s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | x8bjd ready=false restarts=1 terminated | serving endpoints=2 GET /work via Service: ok
|
||||
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 14s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 21s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 28s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 35s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 42s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 49s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
t= 55s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 62s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: ok
|
||||
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 75s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 80s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 86s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 91s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 97s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=102s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=108s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=113s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=119s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=125s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t=130s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
|
||||
# Events (probe failures and kills) for orders pods:
|
||||
2 Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.204:8080/actuator/health/liveness": dial tcp 10.42.0.204:8080: connect: connection refused
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.203:8080/actuator/health/liveness": dial tcp 10.42.0.203:8080: connect: connection refused
|
||||
@@ -0,0 +1,55 @@
|
||||
# server.shutdown=immediate, no preStop hook
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=66
|
||||
t= 4s conc=20 ok=67
|
||||
t= 5s conc=20 ok=67
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=66
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=64
|
||||
t= 10s conc=20 ok=59
|
||||
t= 11s conc=20 ok=67
|
||||
t= 12s conc=20 ok=59
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ConnectException=2 ok=55
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=68
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=5 ok=57
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=66
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=62
|
||||
t= 26s conc=20 ok=66
|
||||
t= 27s conc=20 ok=66
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=63
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=71
|
||||
t= 32s conc=20 ok=61
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=66
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=71
|
||||
t= 39s conc=20 ok=61
|
||||
t= 40s conc=20 ok=62
|
||||
t= 41s conc=20 ok=72
|
||||
t= 42s conc=20 ok=65
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=66
|
||||
t= 45s conc=20 ok=34
|
||||
TOTAL {ConnectException=7, ok=2905}
|
||||
LATENCY ms p50=305 p90=315 p99=503 max=747 (n=2905)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-64544dd744-fb4t9 Stopping container app
|
||||
1 Normal Killing pod/orders-64544dd744-jtqk5 Stopping container app
|
||||
@@ -0,0 +1,55 @@
|
||||
# graceful shutdown (default), no preStop hook
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=62
|
||||
t= 4s conc=20 ok=71
|
||||
t= 5s conc=20 ok=67
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=63
|
||||
t= 8s conc=20 ok=70
|
||||
t= 9s conc=20 ok=67
|
||||
t= 10s conc=20 ok=60
|
||||
t= 11s conc=20 ok=62
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ok=60
|
||||
t= 15s conc=20 ok=56
|
||||
t= 16s conc=20 ok=62
|
||||
t= 17s conc=20 ok=70
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=62
|
||||
t= 20s conc=20 ok=61
|
||||
t= 21s conc=20 ConnectException=5 ok=60
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=67
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=66
|
||||
t= 26s conc=20 ok=61
|
||||
t= 27s conc=20 ok=69
|
||||
t= 28s conc=20 ok=67
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=63
|
||||
t= 31s conc=20 ok=67
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=67
|
||||
t= 36s conc=20 ok=64
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=69
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=66
|
||||
t= 45s conc=20 ok=24
|
||||
TOTAL {ConnectException=5, ok=2908}
|
||||
LATENCY ms p50=305 p90=317 p99=404 max=566 (n=2908)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-765b4cb65c-7jwsj Stopping container app
|
||||
1 Normal Killing pod/orders-765b4cb65c-g2xzr Stopping container app
|
||||
@@ -0,0 +1,55 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5}
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=63
|
||||
t= 3s conc=20 ok=65
|
||||
t= 4s conc=20 ok=72
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=64
|
||||
t= 7s conc=20 ok=67
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=61
|
||||
t= 10s conc=20 ok=64
|
||||
t= 11s conc=20 ok=66
|
||||
t= 12s conc=20 ok=63
|
||||
t= 13s conc=20 ok=64
|
||||
t= 14s conc=20 ok=64
|
||||
t= 15s conc=20 ok=62
|
||||
t= 16s conc=20 ok=68
|
||||
t= 17s conc=20 ok=61
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=62
|
||||
t= 21s conc=20 ok=69
|
||||
t= 22s conc=20 ok=64
|
||||
t= 23s conc=20 ok=67
|
||||
t= 24s conc=20 ok=60
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=63
|
||||
t= 28s conc=20 ok=60
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=70
|
||||
t= 31s conc=20 ok=66
|
||||
t= 32s conc=20 ok=60
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=72
|
||||
t= 38s conc=20 ok=63
|
||||
t= 39s conc=20 ok=63
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=70
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=60
|
||||
t= 44s conc=20 ok=71
|
||||
t= 45s conc=20 ok=23
|
||||
TOTAL {ok=2918}
|
||||
LATENCY ms p50=305 p90=315 p99=376 max=512 (n=2918)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-7c869db794-bbr2d Stopping container app
|
||||
1 Normal Killing pod/orders-7c869db794-gdv2r Stopping container app
|
||||
@@ -0,0 +1,57 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image
|
||||
# 20 clients, GET /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=61
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=69
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=61
|
||||
t= 7s conc=20 ok=71
|
||||
t= 8s conc=20 ok=68
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=62
|
||||
t= 12s conc=20 ok=58
|
||||
t= 13s conc=20 ok=71
|
||||
t= 14s conc=20 ConnectException=1 ok=69
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=65
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=60
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=67
|
||||
t= 21s conc=20 ConnectException=4 ok=62
|
||||
t= 22s conc=20 ok=67
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=65
|
||||
t= 28s conc=20 ok=61
|
||||
t= 29s conc=20 ok=67
|
||||
t= 30s conc=20 ok=67
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=66
|
||||
t= 35s conc=20 ok=63
|
||||
t= 36s conc=20 ok=65
|
||||
t= 37s conc=20 ok=67
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=68
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=63
|
||||
t= 45s conc=20 ok=31
|
||||
TOTAL {ConnectException=5, ok=2922}
|
||||
LATENCY ms p50=305 p90=316 p99=394 max=531 (n=2922)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-5bfb45b45f-k5kxw Stopping container app
|
||||
1 Normal Killing pod/orders-5bfb45b45f-lccf9 Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-5bfb45b45f-k5kxw PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-5bfb45b45f-lccf9 PreStopHook failed
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=66
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=64
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=64
|
||||
t= 7s conc=20 ok=70
|
||||
t= 8s conc=20 ok=64
|
||||
t= 9s conc=20 ok=62
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=70
|
||||
t= 12s conc=20 ok=65
|
||||
t= 13s conc=20 ok=62
|
||||
t= 14s conc=20 IOException=14 ok=51
|
||||
t= 15s conc=20 ok=59
|
||||
t= 16s conc=20 ok=64
|
||||
t= 17s conc=20 ok=71
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 ok=60
|
||||
t= 20s conc=20 ok=65
|
||||
t= 21s conc=20 IOException=12 ok=54
|
||||
t= 22s conc=20 ok=68
|
||||
t= 23s conc=20 ok=69
|
||||
t= 24s conc=20 ok=60
|
||||
t= 25s conc=20 ok=67
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=66
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=65
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=64
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=67
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=69
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=64
|
||||
t= 45s conc=20 ok=33
|
||||
METHOD POST
|
||||
TOTAL {IOException=26, ok=2909}
|
||||
LATENCY ms p50=305 p90=316 p99=419 max=515 (n=2909)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-54c6f9fcdf-s8t24 Stopping container app
|
||||
1 Normal Killing pod/orders-54c6f9fcdf-t44vr Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=80
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=69
|
||||
t= 6s conc=20 ok=71
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=61
|
||||
t= 10s conc=20 ok=72
|
||||
t= 11s conc=20 ok=67
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=62
|
||||
t= 14s conc=20 ConnectException=3 IOException=12 ok=62
|
||||
t= 15s conc=20 ok=64
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=60
|
||||
t= 18s conc=20 ok=65
|
||||
t= 19s conc=20 ok=66
|
||||
t= 20s conc=20 ok=69
|
||||
t= 21s conc=20 ConnectException=3 IOException=14 ok=58
|
||||
t= 22s conc=20 ok=62
|
||||
t= 23s conc=20 ok=61
|
||||
t= 24s conc=20 ok=64
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=66
|
||||
t= 27s conc=20 ok=60
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=73
|
||||
t= 30s conc=20 ok=61
|
||||
t= 31s conc=20 ok=64
|
||||
t= 32s conc=20 ok=69
|
||||
t= 33s conc=20 ok=67
|
||||
t= 34s conc=20 ok=60
|
||||
t= 35s conc=20 ok=65
|
||||
t= 36s conc=20 ok=70
|
||||
t= 37s conc=20 ok=65
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=71
|
||||
t= 40s conc=20 ok=61
|
||||
t= 41s conc=20 ok=64
|
||||
t= 42s conc=20 ok=65
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=20
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=6, IOException=26, ok=2909}
|
||||
LATENCY ms p50=304 p90=315 p99=375 max=466 (n=2909)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-69d489d468-ncnf5 Stopping container app
|
||||
1 Normal Killing pod/orders-69d489d468-v6dg2 Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# server.shutdown=immediate, no preStop hook (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=68
|
||||
t= 3s conc=20 ok=72
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=72
|
||||
t= 7s conc=20 ok=66
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=68
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=64
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=68
|
||||
t= 14s conc=20 ConnectException=2 IOException=8 ok=61
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=60
|
||||
t= 18s conc=20 ok=63
|
||||
t= 19s conc=20 ok=69
|
||||
t= 20s conc=20 ok=68
|
||||
t= 21s conc=20 ConnectException=3 IOException=16 ok=46
|
||||
t= 22s conc=20 ok=60
|
||||
t= 23s conc=20 ok=62
|
||||
t= 24s conc=20 ok=72
|
||||
t= 25s conc=20 ok=66
|
||||
t= 26s conc=20 ok=62
|
||||
t= 27s conc=20 ok=65
|
||||
t= 28s conc=20 ok=70
|
||||
t= 29s conc=20 ok=63
|
||||
t= 30s conc=20 ok=62
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=68
|
||||
t= 33s conc=20 ok=60
|
||||
t= 34s conc=20 ok=63
|
||||
t= 35s conc=20 ok=74
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=62
|
||||
t= 38s conc=20 ok=70
|
||||
t= 39s conc=20 ok=68
|
||||
t= 40s conc=20 ok=62
|
||||
t= 41s conc=20 ok=60
|
||||
t= 42s conc=20 ok=75
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=38
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=5, IOException=24, ok=2915}
|
||||
LATENCY ms p50=304 p90=315 p99=422 max=491 (n=2915)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-d59686bf-fg2x7 Stopping container app
|
||||
1 Normal Killing pod/orders-d59686bf-vkrrs Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=64
|
||||
t= 4s conc=20 ok=61
|
||||
t= 5s conc=20 ok=65
|
||||
t= 6s conc=20 ok=69
|
||||
t= 7s conc=20 ok=65
|
||||
t= 8s conc=20 ok=61
|
||||
t= 9s conc=20 ok=65
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=68
|
||||
t= 12s conc=20 ok=64
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=62
|
||||
t= 15s conc=20 ok=66
|
||||
t= 16s conc=20 ok=63
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=61
|
||||
t= 20s conc=20 ok=71
|
||||
t= 21s conc=20 ok=61
|
||||
t= 22s conc=20 IOException=3 ok=65
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=67
|
||||
t= 27s conc=20 ok=62
|
||||
t= 28s conc=20 ok=68
|
||||
t= 29s conc=20 ok=65
|
||||
t= 30s conc=20 ok=65
|
||||
t= 31s conc=20 ok=63
|
||||
t= 32s conc=20 ok=69
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=64
|
||||
t= 35s conc=20 ok=69
|
||||
t= 36s conc=20 ok=63
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=66
|
||||
t= 41s conc=20 ok=64
|
||||
t= 42s conc=20 ok=66
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=33
|
||||
METHOD POST
|
||||
TOTAL {IOException=3, ok=2924}
|
||||
LATENCY ms p50=305 p90=316 p99=365 max=572 (n=2924)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-98b44f5f4-vwrm2 Stopping container app
|
||||
1 Normal Killing pod/orders-98b44f5f4-xshdx Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=61
|
||||
t= 2s conc=20 ok=79
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=68
|
||||
t= 6s conc=20 ok=72
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=63
|
||||
t= 10s conc=20 ok=64
|
||||
t= 11s conc=20 ok=73
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=65
|
||||
t= 14s conc=20 ConnectException=2 ok=61
|
||||
t= 15s conc=20 ok=67
|
||||
t= 16s conc=20 ok=65
|
||||
t= 17s conc=20 ok=62
|
||||
t= 18s conc=20 ok=61
|
||||
t= 19s conc=20 ok=72
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=6 IOException=11 ok=62
|
||||
t= 22s conc=20 ok=63
|
||||
t= 23s conc=20 ok=64
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=67
|
||||
t= 28s conc=20 ok=64
|
||||
t= 29s conc=20 ok=69
|
||||
t= 30s conc=20 ok=62
|
||||
t= 31s conc=20 ok=65
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=69
|
||||
t= 37s conc=20 ok=62
|
||||
t= 38s conc=20 ok=65
|
||||
t= 39s conc=20 ok=71
|
||||
t= 40s conc=20 ok=64
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=61
|
||||
t= 43s conc=20 ok=72
|
||||
t= 44s conc=20 ok=62
|
||||
t= 45s conc=20 ok=29
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=8, IOException=11, ok=2932}
|
||||
LATENCY ms p50=305 p90=315 p99=357 max=462 (n=2932)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-548cdf7b4c-gh4p6 Stopping container app
|
||||
1 Normal Killing pod/orders-548cdf7b4c-r55sk Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown (default), no preStop hook (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=80
|
||||
t= 3s conc=20 ok=60
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=71
|
||||
t= 6s conc=20 ok=69
|
||||
t= 7s conc=20 ok=60
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=72
|
||||
t= 10s conc=20 ok=68
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=78
|
||||
t= 14s conc=20 ConnectException=4 IOException=1 ok=62
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=61
|
||||
t= 18s conc=20 ok=68
|
||||
t= 19s conc=20 ok=64
|
||||
t= 20s conc=20 ok=67
|
||||
t= 21s conc=20 ConnectException=2 IOException=4 ok=61
|
||||
t= 22s conc=20 ok=61
|
||||
t= 23s conc=20 ok=65
|
||||
t= 24s conc=20 ok=73
|
||||
t= 25s conc=20 ok=61
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=68
|
||||
t= 28s conc=20 ok=69
|
||||
t= 29s conc=20 ok=60
|
||||
t= 30s conc=20 ok=63
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=67
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=65
|
||||
t= 35s conc=20 ok=72
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=63
|
||||
t= 41s conc=20 ok=65
|
||||
t= 42s conc=20 ok=69
|
||||
t= 43s conc=20 ok=65
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=21
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=6, IOException=5, ok=2929}
|
||||
LATENCY ms p50=304 p90=314 p99=369 max=475 (n=2929)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-64f8fbf668-2znqt Stopping container app
|
||||
1 Normal Killing pod/orders-64f8fbf668-sbmgw Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=64
|
||||
t= 3s conc=20 ok=63
|
||||
t= 4s conc=20 ok=68
|
||||
t= 5s conc=20 ok=65
|
||||
t= 6s conc=20 ok=60
|
||||
t= 7s conc=20 ok=68
|
||||
t= 8s conc=20 ok=67
|
||||
t= 9s conc=20 ok=65
|
||||
t= 10s conc=20 ok=61
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=67
|
||||
t= 13s conc=20 ok=72
|
||||
t= 14s conc=20 ok=61
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=67
|
||||
t= 17s conc=20 ok=64
|
||||
t= 18s conc=20 ok=64
|
||||
t= 19s conc=20 IOException=1 ok=65
|
||||
t= 20s conc=20 ok=59
|
||||
t= 21s conc=20 ok=70
|
||||
t= 22s conc=20 ok=67
|
||||
t= 23s conc=20 ok=63
|
||||
t= 24s conc=20 ok=64
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 IOException=1 ok=68
|
||||
t= 27s conc=20 ok=61
|
||||
t= 28s conc=20 ok=63
|
||||
t= 29s conc=20 ok=70
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=61
|
||||
t= 32s conc=20 ok=64
|
||||
t= 33s conc=20 ok=71
|
||||
t= 34s conc=20 ok=65
|
||||
t= 35s conc=20 ok=60
|
||||
t= 36s conc=20 ok=66
|
||||
t= 37s conc=20 ok=71
|
||||
t= 38s conc=20 ok=63
|
||||
t= 39s conc=20 ok=64
|
||||
t= 40s conc=20 ok=66
|
||||
t= 41s conc=20 ok=70
|
||||
t= 42s conc=20 ok=60
|
||||
t= 43s conc=20 ok=66
|
||||
t= 44s conc=20 ok=71
|
||||
t= 45s conc=20 ok=29
|
||||
METHOD POST
|
||||
TOTAL {IOException=2, ok=2927}
|
||||
LATENCY ms p50=305 p90=316 p99=370 max=522 (n=2927)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-c7b9b585f-9nrcr Stopping container app
|
||||
1 Normal Killing pod/orders-c7b9b585f-zvcxr Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=70
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=61
|
||||
t= 6s conc=20 ok=75
|
||||
t= 7s conc=20 ok=64
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=80
|
||||
t= 11s conc=20 ok=60
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=65
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=72
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=62
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=69
|
||||
t= 21s conc=20 ok=62
|
||||
t= 22s conc=20 ok=66
|
||||
t= 23s conc=20 ok=72
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=65
|
||||
t= 26s conc=20 ok=65
|
||||
t= 27s conc=20 ok=68
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=62
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=70
|
||||
t= 32s conc=20 ok=61
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=67
|
||||
t= 36s conc=20 ok=61
|
||||
t= 37s conc=20 ok=65
|
||||
t= 38s conc=20 ok=72
|
||||
t= 39s conc=20 ok=63
|
||||
t= 40s conc=20 ok=60
|
||||
t= 41s conc=20 ok=72
|
||||
t= 42s conc=20 ok=67
|
||||
t= 43s conc=20 ok=61
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=31
|
||||
METHOD POST
|
||||
TOTAL {ok=2942}
|
||||
LATENCY ms p50=304 p90=313 p99=360 max=493 (n=2942)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-766f65d94b-4hwnl Stopping container app
|
||||
1 Normal Killing pod/orders-766f65d94b-l5nmd Stopping container app
|
||||
@@ -0,0 +1,56 @@
|
||||
# graceful shutdown + preStop: sleep: {seconds: 5} (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=70
|
||||
t= 3s conc=20 ok=70
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=74
|
||||
t= 7s conc=20 ok=64
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=65
|
||||
t= 11s conc=20 ok=74
|
||||
t= 12s conc=20 ok=60
|
||||
t= 13s conc=20 ok=63
|
||||
t= 14s conc=20 ok=63
|
||||
t= 15s conc=20 ok=74
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=65
|
||||
t= 18s conc=20 ok=61
|
||||
t= 19s conc=20 IOException=1 ok=69
|
||||
t= 20s conc=20 ok=65
|
||||
t= 21s conc=20 ok=60
|
||||
t= 22s conc=20 ok=65
|
||||
t= 23s conc=20 ok=69
|
||||
t= 24s conc=20 ok=66
|
||||
t= 25s conc=20 ok=61
|
||||
t= 26s conc=20 ok=69
|
||||
t= 27s conc=20 ok=64
|
||||
t= 28s conc=20 ok=66
|
||||
t= 29s conc=20 ok=64
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=66
|
||||
t= 32s conc=20 ok=64
|
||||
t= 33s conc=20 ok=65
|
||||
t= 34s conc=20 ok=69
|
||||
t= 35s conc=20 ok=66
|
||||
t= 36s conc=20 ok=64
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=66
|
||||
t= 40s conc=20 ok=69
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=68
|
||||
t= 43s conc=20 ok=63
|
||||
t= 44s conc=20 ok=69
|
||||
t= 45s conc=20 ok=28
|
||||
METHOD POST
|
||||
TOTAL {IOException=1, ok=2939}
|
||||
LATENCY ms p50=304 p90=313 p99=360 max=476 (n=2939)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-5cfc646684-4stbt Stopping container app
|
||||
1 Normal Killing pod/orders-5cfc646684-zst8t Stopping container app
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 1 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=60
|
||||
t= 3s conc=20 ok=63
|
||||
t= 4s conc=20 ok=65
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=70
|
||||
t= 7s conc=20 ok=63
|
||||
t= 8s conc=20 ok=65
|
||||
t= 9s conc=20 ok=62
|
||||
t= 10s conc=20 ok=63
|
||||
t= 11s conc=20 ok=68
|
||||
t= 12s conc=20 ok=61
|
||||
t= 13s conc=20 ok=68
|
||||
t= 14s conc=20 ConnectException=2 ok=63
|
||||
t= 15s conc=20 ok=61
|
||||
t= 16s conc=20 ok=67
|
||||
t= 17s conc=20 ok=64
|
||||
t= 18s conc=20 ok=67
|
||||
t= 19s conc=20 ok=61
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=2 IOException=3 ok=69
|
||||
t= 22s conc=20 ok=68
|
||||
t= 23s conc=20 ok=60
|
||||
t= 24s conc=20 ok=62
|
||||
t= 25s conc=20 ok=68
|
||||
t= 26s conc=20 ok=70
|
||||
t= 27s conc=20 ok=60
|
||||
t= 28s conc=20 ok=65
|
||||
t= 29s conc=20 ok=67
|
||||
t= 30s conc=20 ok=68
|
||||
t= 31s conc=20 ok=62
|
||||
t= 32s conc=20 ok=66
|
||||
t= 33s conc=20 ok=72
|
||||
t= 34s conc=20 ok=60
|
||||
t= 35s conc=20 ok=64
|
||||
t= 36s conc=20 ok=68
|
||||
t= 37s conc=20 ok=68
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=66
|
||||
t= 40s conc=20 ok=72
|
||||
t= 41s conc=20 ok=60
|
||||
t= 42s conc=20 ok=68
|
||||
t= 43s conc=20 ok=64
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=28
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=4, IOException=3, ok=2921}
|
||||
LATENCY ms p50=305 p90=315 p99=370 max=545 (n=2921)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-68fd794d97-sjq65 Stopping container app
|
||||
1 Normal Killing pod/orders-68fd794d97-zsznq Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 2 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=71
|
||||
t= 3s conc=20 ok=69
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=62
|
||||
t= 6s conc=20 ok=76
|
||||
t= 7s conc=20 ok=62
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=69
|
||||
t= 10s conc=20 ok=66
|
||||
t= 11s conc=20 ok=65
|
||||
t= 12s conc=20 ok=57
|
||||
t= 13s conc=20 ok=60
|
||||
t= 14s conc=20 ConnectException=2 ok=69
|
||||
t= 15s conc=20 ok=57
|
||||
t= 16s conc=20 ok=60
|
||||
t= 17s conc=20 ok=71
|
||||
t= 18s conc=20 ok=66
|
||||
t= 19s conc=20 ok=63
|
||||
t= 20s conc=20 ok=63
|
||||
t= 21s conc=20 ConnectException=3 IOException=2 ok=66
|
||||
t= 22s conc=20 ok=66
|
||||
t= 23s conc=20 ok=65
|
||||
t= 24s conc=20 ok=63
|
||||
t= 25s conc=20 ok=70
|
||||
t= 26s conc=20 ok=64
|
||||
t= 27s conc=20 ok=64
|
||||
t= 28s conc=20 ok=62
|
||||
t= 29s conc=20 ok=70
|
||||
t= 30s conc=20 ok=64
|
||||
t= 31s conc=20 ok=64
|
||||
t= 32s conc=20 ok=70
|
||||
t= 33s conc=20 ok=66
|
||||
t= 34s conc=20 ok=64
|
||||
t= 35s conc=20 ok=62
|
||||
t= 36s conc=20 ok=70
|
||||
t= 37s conc=20 ok=64
|
||||
t= 38s conc=20 ok=64
|
||||
t= 39s conc=20 ok=62
|
||||
t= 40s conc=20 ok=74
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=63
|
||||
t= 43s conc=20 ok=70
|
||||
t= 44s conc=20 ok=65
|
||||
t= 45s conc=20 ok=25
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=5, IOException=2, ok=2926}
|
||||
LATENCY ms p50=305 p90=314 p99=376 max=606 (n=2926)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-54f46cb799-5bjgb Stopping container app
|
||||
1 Normal Killing pod/orders-54f46cb799-mz47p Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-54f46cb799-5bjgb PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-54f46cb799-mz47p PreStopHook failed
|
||||
@@ -0,0 +1,58 @@
|
||||
# graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image (run 3 of 3)
|
||||
# 20 clients, POST /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s
|
||||
|
||||
t= 0s conc=20 ok=40
|
||||
t= 1s conc=20 ok=60
|
||||
t= 2s conc=20 ok=74
|
||||
t= 3s conc=20 ok=66
|
||||
t= 4s conc=20 ok=60
|
||||
t= 5s conc=20 ok=60
|
||||
t= 6s conc=20 ok=68
|
||||
t= 7s conc=20 ok=72
|
||||
t= 8s conc=20 ok=60
|
||||
t= 9s conc=20 ok=60
|
||||
t= 10s conc=20 ok=66
|
||||
t= 11s conc=20 ok=73
|
||||
t= 12s conc=20 ok=61
|
||||
t= 13s conc=20 ok=64
|
||||
t= 14s conc=20 ConnectException=3 IOException=1 ok=64
|
||||
t= 15s conc=20 ok=60
|
||||
t= 16s conc=20 ok=66
|
||||
t= 17s conc=20 ok=66
|
||||
t= 18s conc=20 ok=62
|
||||
t= 19s conc=20 ok=72
|
||||
t= 20s conc=20 ok=60
|
||||
t= 21s conc=20 ConnectException=4 ok=66
|
||||
t= 22s conc=20 ok=62
|
||||
t= 23s conc=20 ok=66
|
||||
t= 24s conc=20 ok=67
|
||||
t= 25s conc=20 ok=65
|
||||
t= 26s conc=20 ok=63
|
||||
t= 27s conc=20 ok=70
|
||||
t= 28s conc=20 ok=67
|
||||
t= 29s conc=20 ok=62
|
||||
t= 30s conc=20 ok=66
|
||||
t= 31s conc=20 ok=62
|
||||
t= 32s conc=20 ok=70
|
||||
t= 33s conc=20 ok=63
|
||||
t= 34s conc=20 ok=67
|
||||
t= 35s conc=20 ok=70
|
||||
t= 36s conc=20 ok=62
|
||||
t= 37s conc=20 ok=66
|
||||
t= 38s conc=20 ok=62
|
||||
t= 39s conc=20 ok=70
|
||||
t= 40s conc=20 ok=65
|
||||
t= 41s conc=20 ok=63
|
||||
t= 42s conc=20 ok=64
|
||||
t= 43s conc=20 ok=68
|
||||
t= 44s conc=20 ok=68
|
||||
t= 45s conc=20 ok=22
|
||||
METHOD POST
|
||||
TOTAL {ConnectException=7, IOException=1, ok=2930}
|
||||
LATENCY ms p50=305 p90=314 p99=354 max=468 (n=2930)
|
||||
|
||||
# Events:
|
||||
1 Normal Killing pod/orders-579c8f9fd9-st6sf Stopping container app
|
||||
1 Normal Killing pod/orders-579c8f9fd9-x2zwq Stopping container app
|
||||
1 Warning FailedPreStopHook pod/orders-579c8f9fd9-st6sf PreStopHook failed
|
||||
1 Warning FailedPreStopHook pod/orders-579c8f9fd9-x2zwq PreStopHook failed
|
||||
@@ -0,0 +1,6 @@
|
||||
# POST /work?ms=300, 20 clients, rolling restart of 2 replicas. Failed requests per run (successful in brackets).
|
||||
|
||||
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
|
||||
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
|
||||
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
|
||||
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
|
||||
@@ -0,0 +1,56 @@
|
||||
# DEMO_STARTUP_DELAY=40s: the context takes ~45 s to refresh, and Tomcat only listens after that.
|
||||
# 1 replica, rolling update (maxSurge 1, maxUnavailable 0). Liveness: period 5 s, failureThreshold 3.
|
||||
# Pods being deleted are not listed. The old pod keeps serving while the new one is not ready.
|
||||
|
||||
## no startupProbe
|
||||
t= 0s [7ltj7 ready=true restarts=0]
|
||||
t= 7s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 13s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 19s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 25s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 32s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 38s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 44s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 50s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 57s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 63s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 69s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 75s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 81s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 88s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t= 94s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=100s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=106s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=112s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=118s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
|
||||
# Events so far:
|
||||
1 Normal Killing Container app failed liveness probe, will be restarted
|
||||
1 Warning Unhealthy Liveness probe failed: Get "http://10.42.0.221:8080/actuator/health/liveness": dial tcp 10.42.0.221:8080: connect: connection refused
|
||||
1 Warning Unhealthy Readiness probe failed: Get "http://10.42.0.221:8080/actuator/health/readiness": dial tcp 10.42.0.221:8080: connect: connection refused
|
||||
|
||||
## startupProbe: /actuator/health/liveness every 2 s, failureThreshold 60 (a 120 s budget)
|
||||
t= 0s [7ltj7 ready=true restarts=0]
|
||||
t= 7s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 13s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 20s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 26s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 32s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 39s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 45s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 51s [jthjg ready=true restarts=0]
|
||||
t= 58s [jthjg ready=true restarts=0]
|
||||
t= 64s [jthjg ready=true restarts=0]
|
||||
t= 70s [jthjg ready=true restarts=0]
|
||||
t= 76s [jthjg ready=true restarts=0]
|
||||
t= 82s [jthjg ready=true restarts=0]
|
||||
t= 88s [jthjg ready=true restarts=0]
|
||||
t= 95s [jthjg ready=true restarts=0]
|
||||
t=101s [jthjg ready=true restarts=0]
|
||||
t=108s [jthjg ready=true restarts=0]
|
||||
t=114s [jthjg ready=true restarts=0]
|
||||
t=120s [jthjg ready=true restarts=0]
|
||||
|
||||
# Events during the startupProbe run:
|
||||
2 Normal Killing Stopping container app
|
||||
1 Warning Unhealthy Startup probe failed: Get "http://10.42.0.222:8080/actuator/health/liveness": dial tcp 10.42.0.222:8080: connect: connection refused
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: demo
|
||||
@@ -0,0 +1,13 @@
|
||||
# A shell inside the cluster for the scripts to call Services from (the app image has none).
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: client
|
||||
namespace: demo
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 1
|
||||
containers:
|
||||
- name: busybox
|
||||
image: busybox:1.37
|
||||
imagePullPolicy: Never
|
||||
command: ["sh", "-c", "trap 'exit 0' TERM; while true; do sleep 1; done"]
|
||||
@@ -0,0 +1,33 @@
|
||||
# A trivially small HTTP service the app's "downstream" health indicator calls. Scale it to zero to
|
||||
# simulate a dependency outage: kubectl -n demo scale deploy/downstream --replicas=0
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: downstream
|
||||
namespace: demo
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels: {app: downstream}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: downstream}
|
||||
spec:
|
||||
# busybox httpd runs as PID 1 and ignores SIGTERM, so without this a scale-to-zero leaves the
|
||||
# "outage" serving for the full 30 s grace period.
|
||||
terminationGracePeriodSeconds: 2
|
||||
containers:
|
||||
- name: httpd
|
||||
image: busybox:1.37
|
||||
imagePullPolicy: Never
|
||||
command: ["sh", "-c", "mkdir -p /www && echo ok > /www/index.html && exec httpd -f -p 8080 -h /www"]
|
||||
ports: [{containerPort: 8080}]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: downstream
|
||||
namespace: demo
|
||||
spec:
|
||||
selector: {app: downstream}
|
||||
ports: [{port: 8080, targetPort: 8080}]
|
||||
@@ -0,0 +1,38 @@
|
||||
# One replica used only by the CPU-limit / GC experiment. scripts/demo-gc-throttling.sh patches
|
||||
# resources and JAVA_TOOL_OPTIONS per variant.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: gc-lab
|
||||
namespace: demo
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels: {app: gc-lab}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: gc-lab}
|
||||
spec:
|
||||
containers:
|
||||
- name: app
|
||||
image: sbd/k8s-demo:1
|
||||
imagePullPolicy: Never
|
||||
ports: [{name: http, containerPort: 8080}]
|
||||
env:
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: ""
|
||||
resources:
|
||||
requests: {cpu: 100m, memory: 256Mi}
|
||||
limits: {cpu: "1", memory: 1Gi}
|
||||
readinessProbe:
|
||||
httpGet: {path: /actuator/health/readiness, port: http}
|
||||
periodSeconds: 2
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gc-lab
|
||||
namespace: demo
|
||||
spec:
|
||||
selector: {app: gc-lab}
|
||||
ports: [{name: http, port: 8080, targetPort: http}]
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scale orders on in-flight requests per pod, not CPU. Target: 5 in flight on average.
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata: {name: orders, namespace: demo}
|
||||
spec:
|
||||
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: orders}
|
||||
minReplicas: 1
|
||||
maxReplicas: 4
|
||||
metrics:
|
||||
- type: Pods
|
||||
pods:
|
||||
metric: {name: app_inflight_requests}
|
||||
target: {type: AverageValue, averageValue: "5"}
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 0
|
||||
scaleDown:
|
||||
# The default is 300 s. Shortened so the demo shows a scale-down inside a few minutes;
|
||||
# keep the default (or longer) in production.
|
||||
stabilizationWindowSeconds: 30
|
||||
@@ -0,0 +1,112 @@
|
||||
# prometheus-adapter v0.12.0 serving custom.metrics.k8s.io from one rule: the Micrometer gauge
|
||||
# app.inflight.requests, which Prometheus stores as app_inflight_requests.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata: {name: prometheus-adapter, namespace: monitoring}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata: {name: prometheus-adapter}
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: [namespaces, pods, services, nodes]
|
||||
verbs: [get, list, watch]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata: {name: prometheus-adapter}
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: prometheus-adapter}
|
||||
subjects: [{kind: ServiceAccount, name: prometheus-adapter, namespace: monitoring}]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata: {name: prometheus-adapter-auth-delegator}
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: system:auth-delegator}
|
||||
subjects: [{kind: ServiceAccount, name: prometheus-adapter, namespace: monitoring}]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata: {name: prometheus-adapter-auth-reader, namespace: kube-system}
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: extension-apiserver-authentication-reader}
|
||||
subjects: [{kind: ServiceAccount, name: prometheus-adapter, namespace: monitoring}]
|
||||
---
|
||||
# The HPA controller reads custom metrics as this service account.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata: {name: custom-metrics-reader}
|
||||
rules:
|
||||
- apiGroups: [custom.metrics.k8s.io]
|
||||
resources: ["*"]
|
||||
verbs: [get, list]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata: {name: hpa-custom-metrics-reader}
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: custom-metrics-reader}
|
||||
subjects: [{kind: ServiceAccount, name: horizontal-pod-autoscaler, namespace: kube-system}]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata: {name: prometheus-adapter, namespace: monitoring}
|
||||
data:
|
||||
config.yaml: |
|
||||
rules:
|
||||
# Micrometer "app.inflight.requests" -> Prometheus "app_inflight_requests" (dots become
|
||||
# underscores; a gauge gets no suffix). The series carries namespace/pod labels from the
|
||||
# scrape config, which is how the adapter attributes it to a pod.
|
||||
- seriesQuery: 'app_inflight_requests{namespace!="",pod!=""}'
|
||||
resources:
|
||||
overrides:
|
||||
namespace: {resource: namespace}
|
||||
pod: {resource: pod}
|
||||
name:
|
||||
matches: "^app_inflight_requests$"
|
||||
as: "app_inflight_requests"
|
||||
metricsQuery: 'avg_over_time(<<.Series>>{<<.LabelMatchers>>}[30s])'
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata: {name: prometheus-adapter, namespace: monitoring}
|
||||
spec:
|
||||
replicas: 1
|
||||
selector: {matchLabels: {app: prometheus-adapter}}
|
||||
template:
|
||||
metadata: {labels: {app: prometheus-adapter}}
|
||||
spec:
|
||||
serviceAccountName: prometheus-adapter
|
||||
containers:
|
||||
- name: adapter
|
||||
image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0
|
||||
imagePullPolicy: Never
|
||||
args:
|
||||
- --prometheus-url=http://prometheus.monitoring.svc:9090/
|
||||
- --metrics-relist-interval=15s
|
||||
- --config=/etc/adapter/config.yaml
|
||||
- --secure-port=6443
|
||||
- --cert-dir=/tmp/cert
|
||||
ports: [{containerPort: 6443}]
|
||||
resources: {requests: {cpu: 50m, memory: 64Mi}, limits: {memory: 256Mi}}
|
||||
volumeMounts:
|
||||
- {name: config, mountPath: /etc/adapter}
|
||||
- {name: tmp, mountPath: /tmp}
|
||||
volumes:
|
||||
- {name: config, configMap: {name: prometheus-adapter}}
|
||||
- {name: tmp, emptyDir: {}}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata: {name: prometheus-adapter, namespace: monitoring}
|
||||
spec:
|
||||
selector: {app: prometheus-adapter}
|
||||
ports: [{port: 443, targetPort: 6443}]
|
||||
---
|
||||
apiVersion: apiregistration.k8s.io/v1
|
||||
kind: APIService
|
||||
metadata: {name: v1beta1.custom.metrics.k8s.io}
|
||||
spec:
|
||||
service: {name: prometheus-adapter, namespace: monitoring}
|
||||
group: custom.metrics.k8s.io
|
||||
version: v1beta1
|
||||
insecureSkipTLSVerify: true # the adapter generated a self-signed cert; use cert-manager in real clusters
|
||||
groupPriorityMinimum: 100
|
||||
versionPriority: 100
|
||||
@@ -0,0 +1,75 @@
|
||||
# Minimal Prometheus: scrapes every pod labelled app=orders on /actuator/prometheus every 5 s and
|
||||
# keeps the namespace and pod as labels - prometheus-adapter needs both to map a series to a pod.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata: {name: monitoring}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata: {name: prometheus, namespace: monitoring}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata: {name: prometheus-sd}
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: [pods, endpoints, services]
|
||||
verbs: [get, list, watch]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata: {name: prometheus-sd}
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: prometheus-sd}
|
||||
subjects: [{kind: ServiceAccount, name: prometheus, namespace: monitoring}]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata: {name: prometheus, namespace: monitoring}
|
||||
data:
|
||||
prometheus.yml: |
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
scrape_configs:
|
||||
- job_name: orders
|
||||
metrics_path: /actuator/prometheus
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
namespaces: {names: [demo]}
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_label_app]
|
||||
regex: orders
|
||||
action: keep
|
||||
- source_labels: [__meta_kubernetes_pod_container_port_number]
|
||||
regex: "8080"
|
||||
action: keep
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata: {name: prometheus, namespace: monitoring}
|
||||
spec:
|
||||
replicas: 1
|
||||
selector: {matchLabels: {app: prometheus}}
|
||||
template:
|
||||
metadata: {labels: {app: prometheus}}
|
||||
spec:
|
||||
serviceAccountName: prometheus
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: quay.io/prometheus/prometheus:v3.14.0
|
||||
imagePullPolicy: Never
|
||||
args: ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.retention.time=2h"]
|
||||
ports: [{containerPort: 9090}]
|
||||
resources: {requests: {cpu: 50m, memory: 128Mi}, limits: {memory: 512Mi}}
|
||||
volumeMounts: [{name: config, mountPath: /etc/prometheus}]
|
||||
volumes: [{name: config, configMap: {name: prometheus}}]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata: {name: prometheus, namespace: monitoring}
|
||||
spec:
|
||||
selector: {app: prometheus}
|
||||
ports: [{port: 9090, targetPort: 9090}]
|
||||
@@ -0,0 +1,54 @@
|
||||
# The baseline Deployment every scenario starts from. Scenario scripts patch it with the specific
|
||||
# change under test (see scripts/*.sh), so each transcript differs from this file in one place.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: orders
|
||||
namespace: demo
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels: {app: orders}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: orders}
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 30
|
||||
containers:
|
||||
- name: app
|
||||
image: sbd/k8s-demo:1
|
||||
imagePullPolicy: Never
|
||||
ports: [{name: http, containerPort: 8080}]
|
||||
env:
|
||||
- name: DEMO_DOWNSTREAM_URL
|
||||
value: http://downstream:8080/
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-XX:MaxRAMPercentage=75"
|
||||
resources:
|
||||
requests: {cpu: 250m, memory: 512Mi}
|
||||
limits: {memory: 512Mi}
|
||||
startupProbe:
|
||||
httpGet: {path: /actuator/health/liveness, port: http}
|
||||
periodSeconds: 2
|
||||
failureThreshold: 60
|
||||
livenessProbe:
|
||||
httpGet: {path: /actuator/health/liveness, port: http}
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet: {path: /actuator/health/readiness, port: http}
|
||||
periodSeconds: 2
|
||||
failureThreshold: 1
|
||||
lifecycle:
|
||||
preStop:
|
||||
sleep: {seconds: 5}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: orders
|
||||
namespace: demo
|
||||
labels: {app: orders}
|
||||
spec:
|
||||
selector: {app: orders}
|
||||
ports: [{name: http, port: 8080, targetPort: http}]
|
||||
@@ -0,0 +1,54 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>kubernetes-deployment</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<name>kubernetes-deployment</name>
|
||||
<description>Spring Boot 4 on Kubernetes: probes, graceful shutdown, CPU limits and JVM ergonomics, HPA on a Micrometer metric</description>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-webmvc</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-actuator</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>io.micrometer</groupId>
|
||||
<artifactId>micrometer-registry-prometheus</artifactId>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<finalName>app</finalName>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# What the JVM decides from the pod's resources, with no JVM flags at all.
|
||||
# -> docs/output/jvm-ergonomics.txt
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
# cpu.limit | cpu.request | mem.request | mem.limit. Requests are deliberately small and constant:
|
||||
# the JVM ignores them (JDK-8281181 removed CPU shares from the calculation in JDK 19), and small
|
||||
# requests let every case schedule on a 2-CPU node.
|
||||
CASES=(
|
||||
"none|100m|256Mi|2Gi"
|
||||
"500m|100m|256Mi|512Mi"
|
||||
"1|100m|256Mi|1Gi"
|
||||
"1|100m|256Mi|2Gi"
|
||||
"1500m|100m|256Mi|2Gi"
|
||||
"2|100m|256Mi|1Gi"
|
||||
"2|100m|256Mi|1700Mi"
|
||||
"2|100m|256Mi|1800Mi"
|
||||
"2|100m|256Mi|4Gi"
|
||||
)
|
||||
flag() { grep -E "^ *[a-z_]+ +$1 " <<< "$2" | awk '{print $4}'; }
|
||||
{
|
||||
echo "# JVM ergonomics per pod resources. Image: $IMAGE, no JVM options. Node: $(nproc) CPUs."
|
||||
echo "# cgroup v$( [ -f /sys/fs/cgroup/cgroup.controllers ] && echo 2 || echo 1 ) node; see docs/03-jvm-ergonomics.md for the v2 equivalents."
|
||||
echo
|
||||
printf '%-7s %-7s %-7s %-7s | %-5s %-10s %9s %6s %6s %6s\n' "cpu.req" "cpu.lim" "mem.req" "mem.lim" "CPUs" "GC" "MaxHeap" "PGCThr" "CGCThr" "JITThr"
|
||||
i=0
|
||||
for c in "${CASES[@]}"; do
|
||||
IFS='|' read -r cl cr mr ml <<< "$c"
|
||||
if [ "$cl" = none ]; then res="{requests: {cpu: $cr, memory: $mr}, limits: {memory: $ml}}"
|
||||
else res="{requests: {cpu: $cr, memory: $mr}, limits: {cpu: $cl, memory: $ml}}"; fi
|
||||
out=$(oneoff "erg-$i" "$res" java -XX:+PrintFlagsFinal -version 2>&1)
|
||||
gc=SerialGC; [ "$(flag UseG1GC "$out")" = true ] && gc=G1GC; [ "$(flag UseParallelGC "$out")" = true ] && gc=ParallelGC
|
||||
cpus=$(oneoff "erg-$i" "$res" java -XshowSettings:system -version 2>&1 | grep -i 'Effective CPU Count' | awk -F: '{gsub(/ /,"",$2); print $2}')
|
||||
heap=$(( $(flag MaxHeapSize "$out") / 1024 / 1024 ))
|
||||
printf '%-7s %-7s %-7s %-7s | %-5s %-10s %8sM %6s %6s %6s\n' "$cr" "$cl" "$mr" "$ml" "$cpus" "$gc" "$heap" \
|
||||
"$(flag ParallelGCThreads "$out")" "$(flag ConcGCThreads "$out")" "$(flag CICompilerCount "$out")"
|
||||
i=$((i+1))
|
||||
done
|
||||
echo
|
||||
echo "# java -XshowSettings:system for the 1500m case:"
|
||||
oneoff erg-show "{requests: {cpu: 100m, memory: 256Mi}, limits: {cpu: 1500m, memory: 2Gi}}" java -XshowSettings:system -version 2>&1 | grep -vE '^(openjdk|OpenJDK)'
|
||||
} | tee "$OUT/jvm-ergonomics.txt"
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# The same allocation-heavy load against one pod under different CPU limits and GC settings.
|
||||
# Collects GC pauses from the GC log, CFS throttling from cpu.stat (via /diag/gc, before and after),
|
||||
# and request latency from the load generator. -> docs/output/gc-throttling.txt (+ raw logs)
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
kubectl apply -f "$MODULE_DIR/k8s/gc-lab.yaml" > /dev/null
|
||||
GCLOG="-Xlog:gc,gc+cpu:stdout:uptime,level,tags"
|
||||
VARIANTS=(
|
||||
"A|500m|1Gi||500m CPU, defaults"
|
||||
"B|1|1Gi||1 CPU, defaults"
|
||||
"C|1|1Gi|-XX:ActiveProcessorCount=2 -XX:+UseG1GC|1 CPU, G1 forced with ActiveProcessorCount=2"
|
||||
"D|1500m|2Gi||1.5 CPUs, defaults"
|
||||
"E|2|2Gi||2 CPUs, defaults"
|
||||
)
|
||||
diag() { curl -s "$(kubectl -n "$NS" get pod -l app=gc-lab -o jsonpath='{.items[0].status.podIP}'):8080/diag/gc"; }
|
||||
jvm() { curl -s "$(kubectl -n "$NS" get pod -l app=gc-lab -o jsonpath='{.items[0].status.podIP}'):8080/diag/jvm"; }
|
||||
|
||||
printf '%-2s %-48s | %-8s %4s | %6s %9s %8s | %10s %10s | %8s %6s %6s\n' \
|
||||
"" "variant" "GC" "thr" "pauses" "pause sum" "max" "throttled" "thr. time" "requests" "p50" "p99" > "$OUT/gc-throttling.txt"
|
||||
for v in "${VARIANTS[@]}"; do
|
||||
IFS='|' read -r id cpu mem opts desc <<< "$v"
|
||||
lid=$(echo "$id" | tr 'A-Z' 'a-z') # pod names must be lower case
|
||||
kubectl -n "$NS" set resources deploy/gc-lab --limits="cpu=$cpu,memory=$mem" > /dev/null
|
||||
kubectl -n "$NS" set env deploy/gc-lab "JAVA_TOOL_OPTIONS=$opts $GCLOG" > /dev/null
|
||||
kubectl -n "$NS" rollout status deploy/gc-lab --timeout=180s > /dev/null
|
||||
sleep 10
|
||||
pod=$(kubectl -n "$NS" get pod -l app=gc-lab -o jsonpath='{.items[0].metadata.name}')
|
||||
info=$(jvm)
|
||||
before=$(diag)
|
||||
since=$(kubectl -n "$NS" logs "$pod" | wc -l)
|
||||
"$MODULE_DIR/scripts/loadgen.sh" "lg-gc-$lid" "http://gc-lab:8080/alloc?mb=16" 4 60
|
||||
kubectl -n "$NS" wait --for=jsonpath='{.status.phase}'=Succeeded "pod/lg-gc-$lid" --timeout=150s > /dev/null
|
||||
after=$(diag)
|
||||
kubectl -n "$NS" logs "$pod" | tail -n +"$((since + 1))" | grep -E '\[gc' > "$OUT/gc-throttling-$id-gclog.txt"
|
||||
kubectl -n "$NS" logs "lg-gc-$lid" > "$OUT/gc-throttling-$id-load.txt"
|
||||
kubectl -n "$NS" delete pod "lg-gc-$lid" --wait=false > /dev/null
|
||||
python3 - "$id" "$desc" "$info" "$before" "$after" "$OUT/gc-throttling-$id-gclog.txt" "$OUT/gc-throttling-$id-load.txt" >> "$OUT/gc-throttling.txt" <<'PY'
|
||||
import json, re, sys
|
||||
id, desc, info, before, after, gclog, load = sys.argv[1:]
|
||||
info, b, a = json.loads(info), json.loads(before), json.loads(after)
|
||||
gc = "G1" if info["flags"]["UseG1GC"].startswith("true") else "Serial" if info["flags"]["UseSerialGC"].startswith("true") else "Parallel"
|
||||
threads = info["flags"]["ParallelGCThreads"].split()[0]
|
||||
pauses = [float(m.group(1)) for m in re.finditer(r"Pause .*? (\d+\.\d+)ms", open(gclog).read())]
|
||||
cs_b, cs_a = b["cpuStat"], a["cpuStat"]
|
||||
periods = cs_a["nr_periods"] - cs_b["nr_periods"]
|
||||
thr = cs_a["nr_throttled"] - cs_b["nr_throttled"]
|
||||
key = "throttled_time" if "throttled_time" in cs_a else "throttled_usec"
|
||||
tt = cs_a[key] - cs_b[key]
|
||||
tt_ms = tt / 1e6 if key == "throttled_time" else tt / 1e3
|
||||
text = open(load).read()
|
||||
total = re.search(r"TOTAL \{(.*)\}", text).group(1)
|
||||
ok = re.search(r"ok=(\d+)", total)
|
||||
lat = re.search(r"p50=(\d+) p90=(\d+) p99=(\d+) max=(\d+)", text)
|
||||
print("%-2s %-48s | %-8s %4s | %6d %8.0fms %6.1fms | %4d/%-5d %8.1fs | %8s %5sms %5sms" % (
|
||||
id, desc, gc, threads, len(pauses), sum(pauses), max(pauses) if pauses else 0,
|
||||
thr, periods, tt_ms / 1000, ok.group(1) if ok else "0", lat.group(1) if lat else "-", lat.group(3) if lat else "-"))
|
||||
PY
|
||||
tail -1 "$OUT/gc-throttling.txt"
|
||||
done
|
||||
{
|
||||
echo
|
||||
echo "# thr = ParallelGCThreads. pauses/pause sum/max from the GC log during the 60 s run."
|
||||
echo "# throttled = CFS periods in which the container hit its quota / periods elapsed, from cpu.stat."
|
||||
echo "# requests = successful /alloc?mb=16 calls by 4 closed-loop clients in 60 s."
|
||||
} >> "$OUT/gc-throttling.txt"
|
||||
kubectl -n "$NS" scale deploy/gc-lab --replicas=0 > /dev/null
|
||||
python3 "$MODULE_DIR/scripts/gc-cpu-ratio.py" > /dev/null
|
||||
cat "$OUT/gc-throttling.txt"
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# HorizontalPodAutoscaler on a Micrometer gauge through Prometheus + prometheus-adapter.
|
||||
# Load: 2 clients, then 30 from t=20 s to t=140 s, then none; each request holds for 500 ms.
|
||||
# -> docs/output/hpa-custom-metric.txt, hpa-custom-metrics-api.txt
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
H="$MODULE_DIR/k8s/hpa"
|
||||
kubectl apply -f "$H/prometheus.yaml" -f "$H/prometheus-adapter.yaml" > /dev/null
|
||||
kubectl -n monitoring rollout status deploy/prometheus --timeout=120s > /dev/null
|
||||
kubectl -n monitoring rollout status deploy/prometheus-adapter --timeout=120s > /dev/null
|
||||
kubectl -n "$NS" scale deploy/orders --replicas=1 > /dev/null
|
||||
kubectl -n "$NS" rollout status deploy/orders --timeout=120s > /dev/null
|
||||
API=/apis/custom.metrics.k8s.io/v1beta1/namespaces/demo/pods/%2A/app_inflight_requests
|
||||
for _ in $(seq 1 40); do kubectl get --raw "$API" 2>/dev/null | grep -q '"items":\[{' && break; sleep 5; done
|
||||
{
|
||||
echo "# What Micrometer exports (one pod):"
|
||||
echo "\$ curl <pod>:8080/actuator/prometheus | grep app_inflight"
|
||||
curl -s "$(kubectl -n "$NS" get pod -l app=orders -o jsonpath='{.items[0].status.podIP}'):8080/actuator/prometheus" | grep app_inflight
|
||||
echo
|
||||
echo "# What the HPA controller sees through the custom metrics API:"
|
||||
echo "\$ kubectl get --raw $API"
|
||||
kubectl get --raw "$API" | python3 -m json.tool
|
||||
} > "$OUT/hpa-custom-metrics-api.txt"
|
||||
|
||||
kubectl apply -f "$H/hpa.yaml" > /dev/null
|
||||
sleep 20
|
||||
kubectl -n "$NS" delete events --field-selector involvedObject.kind=HorizontalPodAutoscaler > /dev/null 2>&1
|
||||
"$MODULE_DIR/scripts/loadgen.sh" lg-hpa "http://orders:8080/work?ms=500" "0:2,20:30,140:0" 230
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/lg-hpa --timeout=60s > /dev/null
|
||||
start=$(date +%s)
|
||||
{
|
||||
echo "# HPA orders: target app_inflight_requests averageValue 5, min 1, max 4; scaleDown stabilization 30 s"
|
||||
echo "# load: t=0 2 clients, t=20 30 clients, t=140 0 clients; GET /work?ms=500"
|
||||
echo
|
||||
printf '%-6s %-10s %-9s %-8s %s\n' "t" "clients" "metric" "desired" "ready pods"
|
||||
while [ $(( $(date +%s) - start )) -lt 235 ]; do
|
||||
t=$(( $(date +%s) - start ))
|
||||
clients=2; [ $t -ge 20 ] && clients=30; [ $t -ge 140 ] && clients=0
|
||||
cur=$(kubectl -n "$NS" get hpa orders -o jsonpath='{.status.currentMetrics[0].pods.current.averageValue}')
|
||||
des=$(kubectl -n "$NS" get hpa orders -o jsonpath='{.status.desiredReplicas}')
|
||||
ready=$(kubectl -n "$NS" get deploy orders -o jsonpath='{.status.readyReplicas}')
|
||||
printf '%-6s %-10s %-9s %-8s %s\n' "${t}s" "$clients" "${cur:-?}" "${des:-?}" "${ready:-0}"
|
||||
sleep 10
|
||||
done
|
||||
echo
|
||||
echo "# HPA events:"
|
||||
kubectl -n "$NS" get events --field-selector involvedObject.kind=HorizontalPodAutoscaler --sort-by=.lastTimestamp \
|
||||
| awk 'NR>1 {$1=""; $2=""; $4=""; print}' | sed 's/^ *//'
|
||||
echo
|
||||
echo "# Load generator summary:"
|
||||
kubectl -n "$NS" wait --for=jsonpath='{.status.phase}'=Succeeded pod/lg-hpa --timeout=60s > /dev/null
|
||||
kubectl -n "$NS" logs lg-hpa | grep -E '^TOTAL|^LATENCY'
|
||||
} | tee "$OUT/hpa-custom-metric.txt"
|
||||
kubectl -n "$NS" logs lg-hpa > "$OUT/hpa-load.txt"
|
||||
kubectl -n "$NS" delete pod lg-hpa --wait=false > /dev/null
|
||||
kubectl -n "$NS" delete hpa orders > /dev/null
|
||||
kubectl -n "$NS" scale deploy/orders --replicas=2 > /dev/null
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# A downstream outage, with the downstream health indicator in the LIVENESS group and then in the
|
||||
# READINESS group, and in neither (the default). Outage from t=0 to t=60, then 50 s of recovery.
|
||||
# -> docs/output/probes-*.txt
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
|
||||
watch_outage() { # label, file
|
||||
local label="$1" file="$2"
|
||||
kubectl -n "$NS" rollout status deploy/orders --timeout=180s > /dev/null
|
||||
sleep 5
|
||||
{
|
||||
echo "# $label"
|
||||
echo "# health groups: liveness=$(kubectl -n "$NS" get deploy orders -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="MANAGEMENT_ENDPOINT_HEALTH_GROUP_LIVENESS_INCLUDE")].value}') readiness=$(kubectl -n "$NS" get deploy orders -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE")].value}')"
|
||||
echo "# t=0: kubectl scale deploy/downstream --replicas=0 t=60: back to 1"
|
||||
echo
|
||||
kubectl -n "$NS" scale deploy/downstream --replicas=0 > /dev/null
|
||||
local start; start=$(date +%s)
|
||||
local restored=no
|
||||
for _ in $(seq 1 22); do
|
||||
local t=$(( $(date +%s) - start ))
|
||||
if [ "$t" -ge 60 ] && [ "$restored" = no ]; then
|
||||
kubectl -n "$NS" scale deploy/downstream --replicas=1 > /dev/null; restored=yes
|
||||
echo "-------- downstream restored --------"
|
||||
fi
|
||||
local pods; pods=$(kubectl -n "$NS" get pods -l app=orders --no-headers \
|
||||
-o custom-columns=N:.metadata.name,R:.status.containerStatuses[0].ready,C:.status.containerStatuses[0].restartCount,S:.status.containerStatuses[0].state \
|
||||
| awk '{st=$4; sub(/map\[/,"",st); sub(/:.*/,"",st); printf "%s ready=%s restarts=%s %s | ", substr($1,length($1)-4), $2, $3, st}')
|
||||
local eps; eps=$(kubectl -n "$NS" get endpointslices -l kubernetes.io/service-name=orders -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{" "}{end}' | tr ' ' '\n' | grep -c true)
|
||||
local work; work=$(kubectl -n "$NS" exec client -- wget -q -T 2 -O - http://orders:8080/work?ms=1 2>/dev/null | grep -c pod || true)
|
||||
printf 't=%3ds %s serving endpoints=%s GET /work via Service: %s\n' "$t" "$pods" "$eps" "$([ "$work" = 1 ] && echo ok || echo FAIL)"
|
||||
sleep 5
|
||||
done
|
||||
echo
|
||||
echo "# Events (probe failures and kills) for orders pods:"
|
||||
kubectl -n "$NS" get events --field-selector involvedObject.kind=Pod --sort-by=.lastTimestamp \
|
||||
| grep -E '(Unhealthy|Killing|BackOff).*orders-' | awk '{ $1=""; $4=""; print }' | sed 's/^ //' | sort | uniq -c | sort -rn | head -8
|
||||
} | tee "$OUT/$file"
|
||||
}
|
||||
|
||||
kubectl apply -f "$MODULE_DIR/k8s/client.yaml" > /dev/null
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/client --timeout=60s > /dev/null
|
||||
kubectl -n "$NS" scale deploy/downstream --replicas=1 > /dev/null
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
kubectl -n "$NS" set env deploy/orders MANAGEMENT_ENDPOINT_HEALTH_GROUP_LIVENESS_INCLUDE=livenessState,downstream MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE- > /dev/null
|
||||
watch_outage "downstream health indicator in the LIVENESS group" probes-liveness-includes-downstream.txt
|
||||
kubectl -n "$NS" rollout status deploy/downstream --timeout=60s > /dev/null
|
||||
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
kubectl -n "$NS" set env deploy/orders MANAGEMENT_ENDPOINT_HEALTH_GROUP_LIVENESS_INCLUDE- MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,downstream > /dev/null
|
||||
watch_outage "downstream health indicator in the READINESS group" probes-readiness-includes-downstream.txt
|
||||
kubectl -n "$NS" rollout status deploy/downstream --timeout=60s > /dev/null
|
||||
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
kubectl -n "$NS" set env deploy/orders MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE- > /dev/null
|
||||
watch_outage "downstream health indicator in NEITHER group (Spring Boot's default probe groups)" probes-default-groups.txt
|
||||
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
# A rolling restart under load, four ways. Each run: 20 concurrent clients calling /work?ms=300
|
||||
# through the Service for 45 s, rollout restart at t=8 s. METHOD=POST (default) or GET.
|
||||
# -> docs/output/shutdown-<method>-<variant>.txt
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
export METHOD="${METHOD:-POST}"
|
||||
RUNS="${RUNS:-3}"
|
||||
M=$(echo "$METHOD" | tr 'A-Z' 'a-z')
|
||||
D="deploy/orders"
|
||||
PRESTOP_PATH=/spec/template/spec/containers/0/lifecycle
|
||||
|
||||
variant() { # name, description - runs it $RUNS times
|
||||
local base="$1" desc="$2" r
|
||||
for r in $(seq 1 "$RUNS"); do run_once "$base-run$r" "$desc (run $r of $RUNS)"; done
|
||||
}
|
||||
|
||||
run_once() { # name, description
|
||||
local name="$1" desc="$2"
|
||||
kubectl -n "$NS" rollout status "$D" --timeout=180s > /dev/null
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
sleep 3
|
||||
"$MODULE_DIR/scripts/loadgen.sh" "lg-$M-$name" "http://orders:8080/work?ms=300" 20 45
|
||||
kubectl -n "$NS" wait --for=condition=Ready "pod/lg-$M-$name" --timeout=60s > /dev/null
|
||||
sleep 8
|
||||
kubectl -n "$NS" rollout restart "$D" > /dev/null
|
||||
kubectl -n "$NS" rollout status "$D" --timeout=180s > /dev/null
|
||||
kubectl -n "$NS" wait --for=jsonpath='{.status.phase}'=Succeeded "pod/lg-$M-$name" --timeout=120s > /dev/null
|
||||
{
|
||||
echo "# $desc"
|
||||
echo "# 20 clients, $METHOD /work?ms=300 via Service orders, 45 s; kubectl rollout restart at ~t=8 s"
|
||||
echo
|
||||
kubectl -n "$NS" logs "lg-$M-$name"
|
||||
echo
|
||||
echo "# Events:"
|
||||
kubectl -n "$NS" get events --sort-by=.lastTimestamp | grep -E 'PreStop|Killing' | awk '{$1=""; print}' | sed 's/^ //' | sort | uniq -c | head -6
|
||||
} > "$OUT/shutdown-$M-$name.txt"
|
||||
grep -E '^TOTAL|^LATENCY' "$OUT/shutdown-$M-$name.txt" | sed "s/^/$M $name: /"
|
||||
kubectl -n "$NS" delete pod "lg-$M-$name" --wait=false > /dev/null
|
||||
}
|
||||
|
||||
# 1. server.shutdown=immediate, no preStop
|
||||
kubectl -n "$NS" patch "$D" --type=json -p "[{\"op\":\"remove\",\"path\":\"$PRESTOP_PATH\"}]" > /dev/null 2>&1
|
||||
kubectl -n "$NS" set env "$D" SERVER_SHUTDOWN=immediate > /dev/null
|
||||
variant 1-immediate-no-prestop "server.shutdown=immediate, no preStop hook"
|
||||
|
||||
# 2. graceful shutdown (the Boot default), no preStop
|
||||
kubectl -n "$NS" set env "$D" SERVER_SHUTDOWN- > /dev/null
|
||||
variant 2-graceful-no-prestop "graceful shutdown (default), no preStop hook"
|
||||
|
||||
# 3. graceful + native sleep preStop (Kubernetes 1.32+) - the baseline manifest
|
||||
kubectl -n "$NS" patch "$D" --type=json -p "[{\"op\":\"add\",\"path\":\"$PRESTOP_PATH\",\"value\":{\"preStop\":{\"sleep\":{\"seconds\":5}}}}]" > /dev/null
|
||||
variant 3-graceful-prestop-sleep "graceful shutdown + preStop: sleep: {seconds: 5}"
|
||||
|
||||
# 4. graceful + exec preStop that needs a shell - on a distroless image
|
||||
kubectl -n "$NS" patch "$D" --type=json -p "[{\"op\":\"replace\",\"path\":\"$PRESTOP_PATH\",\"value\":{\"preStop\":{\"exec\":{\"command\":[\"sh\",\"-c\",\"sleep 5\"]}}}}]" > /dev/null
|
||||
variant 4-graceful-prestop-exec-sh "graceful shutdown + preStop: exec: [sh, -c, sleep 5] on a distroless image"
|
||||
|
||||
# summary across runs
|
||||
{
|
||||
echo "# $METHOD /work?ms=300, 20 clients, rolling restart of 2 replicas. Failed requests per run (successful in brackets)."
|
||||
echo
|
||||
for f in "$OUT"/shutdown-$M-*-run1.txt; do
|
||||
v=$(basename "$f" -run1.txt); v=${v#shutdown-$M-}
|
||||
printf '%-32s' "$v"
|
||||
for r in $(seq 1 "$RUNS"); do
|
||||
t=$(grep '^TOTAL' "$OUT/shutdown-$M-$v-run$r.txt")
|
||||
ok=$(echo "$t" | grep -o 'ok=[0-9]*' | cut -d= -f2)
|
||||
fails=$(echo "$t" | grep -o '[A-Za-z_0-9]*=[0-9]*' | grep -v '^ok=' | tr '\n' ' ')
|
||||
printf ' | run %s: %-38s' "$r" "${fails:-0 failures} [$ok]"
|
||||
done
|
||||
echo
|
||||
done
|
||||
} | tee "$OUT/shutdown-$M-summary.txt"
|
||||
|
||||
# restore the baseline
|
||||
kubectl -n "$NS" patch "$D" --type=json -p "[{\"op\":\"replace\",\"path\":\"$PRESTOP_PATH\",\"value\":{\"preStop\":{\"sleep\":{\"seconds\":5}}}}]" > /dev/null
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# An application that needs ~45 s to start, with and without a startupProbe. 1 replica; each
|
||||
# phase is ONE patch, so there is one new ReplicaSet per phase. -> docs/output/startup-probe.txt
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
D="deploy/orders"
|
||||
observe() {
|
||||
local label="$1"
|
||||
echo "## $label"
|
||||
local start; start=$(date +%s)
|
||||
for _ in $(seq 1 20); do
|
||||
local t=$(( $(date +%s) - start ))
|
||||
printf 't=%3ds ' "$t"
|
||||
kubectl -n "$NS" get pods -l app=orders --no-headers \
|
||||
-o custom-columns=N:.metadata.name,H:.metadata.labels.pod-template-hash,R:.status.containerStatuses[0].ready,C:.status.containerStatuses[0].restartCount,W:.status.containerStatuses[0].state.waiting.reason,X:.metadata.deletionTimestamp \
|
||||
| awk '{ if ($6 != "<none>") next; printf "[%s ready=%s restarts=%s%s] ", substr($1,length($1)-4), $3, $4, ($5=="<none>"?"":" "$5)}'
|
||||
echo
|
||||
sleep 6
|
||||
done
|
||||
echo
|
||||
}
|
||||
kubectl -n "$NS" scale "$D" --replicas=1 > /dev/null
|
||||
kubectl -n "$NS" rollout status "$D" --timeout=120s > /dev/null
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
{
|
||||
echo "# DEMO_STARTUP_DELAY=40s: the context takes ~45 s to refresh, and Tomcat only listens after that."
|
||||
echo "# 1 replica, rolling update (maxSurge 1, maxUnavailable 0). Liveness: period 5 s, failureThreshold 3."
|
||||
echo "# Pods being deleted are not listed. The old pod keeps serving while the new one is not ready."
|
||||
echo
|
||||
kubectl -n "$NS" patch "$D" --type=json -p '[
|
||||
{"op":"remove","path":"/spec/template/spec/containers/0/startupProbe"},
|
||||
{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"DEMO_STARTUP_DELAY","value":"40s"}}]' > /dev/null
|
||||
observe "no startupProbe"
|
||||
echo "# Events so far:"
|
||||
kubectl -n "$NS" get events --sort-by=.lastTimestamp | grep -E 'Unhealthy|Killing|BackOff' | awk '{$1=""; $4=""; print}' | sed 's/^ *//' | sort | uniq -c | head -6
|
||||
echo
|
||||
kubectl -n "$NS" delete events --all > /dev/null 2>&1
|
||||
kubectl -n "$NS" patch "$D" --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/startupProbe","value":{"httpGet":{"path":"/actuator/health/liveness","port":"http"},"periodSeconds":2,"failureThreshold":60}}]' > /dev/null
|
||||
observe "startupProbe: /actuator/health/liveness every 2 s, failureThreshold 60 (a 120 s budget)"
|
||||
echo "# Events during the startupProbe run:"
|
||||
kubectl -n "$NS" get events --sort-by=.lastTimestamp | grep -E 'Unhealthy|Killing|BackOff' | awk '{$1=""; $4=""; print}' | sed 's/^ *//' | sort | uniq -c | head -6
|
||||
} | tee "$OUT/startup-probe.txt"
|
||||
kubectl -n "$NS" set env "$D" DEMO_STARTUP_DELAY- > /dev/null
|
||||
kubectl -n "$NS" scale "$D" --replicas=2 > /dev/null
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
MODULE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT="$MODULE_DIR/docs/output"
|
||||
NS=demo
|
||||
IMAGE="${IMAGE:-sbd/k8s-demo:1}"
|
||||
mkdir -p "$OUT"
|
||||
export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}"
|
||||
# Run a one-off pod from the app image with an arbitrary command and resources; print its logs.
|
||||
# oneoff <name> '<resources-json>' <command...>
|
||||
oneoff() {
|
||||
local name="$1" resources="$2"; shift 2
|
||||
local cmd; cmd=$(printf '"%s",' "$@"); cmd="[${cmd%,}]"
|
||||
kubectl -n "$NS" delete pod "$name" --ignore-not-found --wait=true > /dev/null
|
||||
kubectl -n "$NS" apply -f - > /dev/null <<YAML
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: {name: $name, namespace: $NS, labels: {role: oneoff}}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: $IMAGE
|
||||
imagePullPolicy: Never
|
||||
command: $cmd
|
||||
resources: $resources
|
||||
YAML
|
||||
kubectl -n "$NS" wait --for=jsonpath='{.status.phase}'=Succeeded "pod/$name" --timeout=120s > /dev/null
|
||||
kubectl -n "$NS" logs "$name"
|
||||
kubectl -n "$NS" delete pod "$name" --wait=false > /dev/null
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env python3
|
||||
"""From the raw GC logs of demo-gc-throttling.sh: CPU time the collector got during pauses vs the
|
||||
pauses' wall time (-Xlog:gc+cpu prints User/Sys/Real per collection, at 10 ms resolution).
|
||||
A single GC thread that is never descheduled gives a ratio near 1.0; N parallel threads near N.
|
||||
Well below that means the collector was waiting for CPU - throttled - inside its own pauses.
|
||||
-> docs/output/gc-cpu-ratio.txt"""
|
||||
import os, re, sys
|
||||
out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "docs", "output")
|
||||
rows = []
|
||||
for v in "ABCDE":
|
||||
path = os.path.join(out, f"gc-throttling-{v}-gclog.txt")
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
cpu = real = 0.0
|
||||
n = 0
|
||||
for m in re.finditer(r"User=(\d+\.\d+)s Sys=(\d+\.\d+)s Real=(\d+\.\d+)s", open(path).read()):
|
||||
u, s, r = map(float, m.groups())
|
||||
cpu += u + s
|
||||
real += r
|
||||
n += 1
|
||||
rows.append((v, n, cpu, real, cpu / real if real else 0))
|
||||
with open(os.path.join(out, "gc-cpu-ratio.txt"), "w") as f:
|
||||
f.write("# CPU the collector received during its pauses, from -Xlog:gc+cpu (sum over every collection)\n\n")
|
||||
f.write("%-2s %12s %12s %12s %10s\n" % ("", "collections", "user+sys", "real", "cpu/real"))
|
||||
for v, n, cpu, real, ratio in rows:
|
||||
f.write("%-2s %12d %11.2fs %11.2fs %10.2f\n" % (v, n, cpu, real, ratio))
|
||||
print(open(os.path.join(out, "gc-cpu-ratio.txt")).read())
|
||||
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run the in-cluster load generator as a pod and wait for it; prints its report.
|
||||
# [METHOD=POST] ./scripts/loadgen.sh <name> <url> <concurrency-or-schedule> <seconds>
|
||||
set -uo pipefail
|
||||
source "$(dirname "$0")/env.sh"
|
||||
name="$1" url="$2" conc="$3" secs="$4"
|
||||
kubectl -n "$NS" delete pod "$name" --ignore-not-found --wait=true > /dev/null
|
||||
kubectl -n "$NS" run "$name" --image="$IMAGE" --image-pull-policy=Never --restart=Never --labels=role=loadgen \
|
||||
--overrides='{"spec":{"containers":[{"name":"'"$name"'","image":"'"$IMAGE"'","imagePullPolicy":"Never","resources":{"requests":{"cpu":"100m","memory":"256Mi"},"limits":{"memory":"512Mi"}},"command":["java","-Dloadgen.method='"${METHOD:-GET}"'","-cp","application.jar","com.ankurm.k8s.loadgen.LoadGen","'"$url"'","'"$conc"'","'"$secs"'"]}]}}' > /dev/null
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.ankurm.k8s;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication
|
||||
public class K8sApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(K8sApplication.class, args);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package com.ankurm.k8s.diag;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.lang.management.GarbageCollectorMXBean;
|
||||
import java.lang.management.ManagementFactory;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.sun.management.HotSpotDiagnosticMXBean;
|
||||
import com.sun.management.OperatingSystemMXBean;
|
||||
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
/**
|
||||
* What the JVM decided about the container it is in, and what the kernel did to it.
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code /diag/jvm} - the ergonomic choices (processors, collector, heap, thread counts) next
|
||||
* to the cgroup limits they were derived from.</li>
|
||||
* <li>{@code /diag/gc} - cumulative GC counts/time and the CFS throttling counters. Take it
|
||||
* before and after a load run and subtract (scripts/demo-gc-throttling.sh does).</li>
|
||||
* </ul>
|
||||
* Reads both cgroup v1 and v2 layouts. Delete before shipping anything real.
|
||||
*/
|
||||
@RestController
|
||||
public class JvmController {
|
||||
|
||||
private static final List<String> FLAGS = List.of("UseSerialGC", "UseParallelGC", "UseG1GC", "UseZGC",
|
||||
"ActiveProcessorCount", "ParallelGCThreads", "ConcGCThreads", "CICompilerCount", "MaxHeapSize",
|
||||
"MaxRAMPercentage", "InitialHeapSize");
|
||||
|
||||
@GetMapping("/diag/jvm")
|
||||
public Map<String, Object> jvm() {
|
||||
Map<String, Object> out = new LinkedHashMap<>();
|
||||
Runtime rt = Runtime.getRuntime();
|
||||
out.put("availableProcessors", rt.availableProcessors());
|
||||
out.put("maxHeapMiB", rt.maxMemory() / (1024 * 1024));
|
||||
out.put("collectors", ManagementFactory.getGarbageCollectorMXBeans().stream()
|
||||
.map(GarbageCollectorMXBean::getName).toList());
|
||||
HotSpotDiagnosticMXBean hs = ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class);
|
||||
Map<String, String> flags = new LinkedHashMap<>();
|
||||
for (String flag : FLAGS) {
|
||||
var option = hs.getVMOption(flag);
|
||||
flags.put(flag, option.getValue() + (option.getOrigin().name().equals("DEFAULT") ? "" : " (" + option.getOrigin() + ")"));
|
||||
}
|
||||
out.put("flags", flags);
|
||||
OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(OperatingSystemMXBean.class);
|
||||
out.put("osTotalMemoryMiB (container-aware)", os.getTotalMemorySize() / (1024 * 1024));
|
||||
out.put("cgroup", cgroup());
|
||||
return out;
|
||||
}
|
||||
|
||||
@GetMapping("/diag/gc")
|
||||
public Map<String, Object> gc() {
|
||||
Map<String, Object> out = new LinkedHashMap<>();
|
||||
for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) {
|
||||
out.put(gc.getName(), Map.of("count", gc.getCollectionCount(), "timeMs", gc.getCollectionTime()));
|
||||
}
|
||||
out.put("cpuStat", cpuStat());
|
||||
OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(OperatingSystemMXBean.class);
|
||||
out.put("processCpuTimeMs", os.getProcessCpuTime() / 1_000_000);
|
||||
return out;
|
||||
}
|
||||
|
||||
private static Map<String, Object> cgroup() {
|
||||
Map<String, Object> out = new LinkedHashMap<>();
|
||||
if (Files.exists(Path.of("/sys/fs/cgroup/cpu.max"))) {
|
||||
out.put("version", 2);
|
||||
out.put("cpu.max", read("/sys/fs/cgroup/cpu.max"));
|
||||
out.put("memory.max", read("/sys/fs/cgroup/memory.max"));
|
||||
}
|
||||
else {
|
||||
out.put("version", 1);
|
||||
out.put("cpu.cfs_quota_us", read("/sys/fs/cgroup/cpu/cpu.cfs_quota_us"));
|
||||
out.put("cpu.cfs_period_us", read("/sys/fs/cgroup/cpu/cpu.cfs_period_us"));
|
||||
out.put("cpu.shares", read("/sys/fs/cgroup/cpu/cpu.shares"));
|
||||
out.put("memory.limit_in_bytes", read("/sys/fs/cgroup/memory/memory.limit_in_bytes"));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private static Map<String, Long> cpuStat() {
|
||||
Path v2 = Path.of("/sys/fs/cgroup/cpu.stat");
|
||||
Path v1 = Path.of("/sys/fs/cgroup/cpu/cpu.stat");
|
||||
Path file = Files.exists(Path.of("/sys/fs/cgroup/cpu.max")) ? v2 : v1;
|
||||
Map<String, Long> out = new LinkedHashMap<>();
|
||||
try {
|
||||
for (String line : Files.readAllLines(file)) {
|
||||
String[] kv = line.trim().split("\\s+");
|
||||
if (kv.length == 2) {
|
||||
out.put(kv[0], Long.parseLong(kv[1]));
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (IOException | NumberFormatException ex) {
|
||||
out.put("unreadable", -1L);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private static String read(String file) {
|
||||
try {
|
||||
return Files.readString(Path.of(file)).trim();
|
||||
}
|
||||
catch (IOException ex) {
|
||||
return "n/a";
|
||||
}
|
||||
}
|
||||
}
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
package com.ankurm.k8s.health;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.boot.health.contributor.Health;
|
||||
import org.springframework.boot.health.contributor.HealthIndicator;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* Reports whether a downstream HTTP service answers. Which probe it belongs to is decided by
|
||||
* configuration, not code - and putting it in the liveness group is the mistake that turns one
|
||||
* dependency outage into a restart storm across every replica (docs/02-probes.md).
|
||||
*/
|
||||
@Component("downstream")
|
||||
public class DownstreamHealthIndicator implements HealthIndicator {
|
||||
|
||||
private final HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofMillis(500)).build();
|
||||
private final URI url;
|
||||
|
||||
public DownstreamHealthIndicator(@Value("${demo.downstream-url:}") String url) {
|
||||
this.url = url.isBlank() ? null : URI.create(url);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Health health() {
|
||||
if (url == null) {
|
||||
return Health.up().withDetail("downstream", "not configured").build();
|
||||
}
|
||||
try {
|
||||
HttpResponse<Void> response = client.send(
|
||||
HttpRequest.newBuilder(url).timeout(Duration.ofMillis(800)).GET().build(),
|
||||
HttpResponse.BodyHandlers.discarding());
|
||||
return response.statusCode() < 500
|
||||
? Health.up().withDetail("status", response.statusCode()).build()
|
||||
: Health.down().withDetail("status", response.statusCode()).build();
|
||||
}
|
||||
catch (Exception ex) {
|
||||
return Health.down().withDetail("error", ex.getClass().getSimpleName()).build();
|
||||
}
|
||||
finally {
|
||||
if (Thread.interrupted()) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package com.ankurm.k8s.loadgen;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
/**
|
||||
* A closed-loop load generator that runs inside the cluster, from the same image:
|
||||
* {@code java -cp application.jar com.ankurm.k8s.loadgen.LoadGen <url> <concurrency> <seconds>}.
|
||||
*
|
||||
* <p>Prints one line per second - successes, and failures grouped by cause - then a summary with
|
||||
* latency percentiles. A "failure" is anything that is not HTTP 200: a refused or reset
|
||||
* connection, a 5xx, a timeout. That is exactly what a rolling update is not supposed to produce.
|
||||
* Concurrency can be changed mid-run by giving a schedule instead: {@code 0:5,60:40,180:0}
|
||||
* (seconds:concurrency).
|
||||
*
|
||||
* <p>{@code -Dloadgen.method=POST} sends POST instead of GET. That matters: the JDK HttpClient
|
||||
* transparently retries an idempotent GET whose connection was closed under it, so a GET load test
|
||||
* under-counts requests the server dropped. A POST is never retried - what fails is what you see.
|
||||
*/
|
||||
public final class LoadGen {
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
URI uri = URI.create(args[0]);
|
||||
String method = System.getProperty("loadgen.method", "GET");
|
||||
String schedule = args[1];
|
||||
int seconds = Integer.parseInt(args[2]);
|
||||
HttpClient client = HttpClient.newBuilder()
|
||||
.version(HttpClient.Version.HTTP_1_1)
|
||||
.connectTimeout(Duration.ofSeconds(2))
|
||||
.build();
|
||||
|
||||
Map<Long, Map<String, AtomicInteger>> perSecond = new ConcurrentHashMap<>();
|
||||
List<Long> latencies = Collections.synchronizedList(new ArrayList<>());
|
||||
AtomicInteger target = new AtomicInteger();
|
||||
AtomicBoolean running = new AtomicBoolean(true);
|
||||
long start = System.nanoTime();
|
||||
TreeMap<Integer, Integer> plan = parse(schedule);
|
||||
|
||||
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
|
||||
AtomicInteger workers = new AtomicInteger();
|
||||
// One controller thread adjusts the number of workers to the schedule.
|
||||
while (running.get()) {
|
||||
long elapsed = (System.nanoTime() - start) / 1_000_000_000;
|
||||
if (elapsed >= seconds) {
|
||||
running.set(false);
|
||||
break;
|
||||
}
|
||||
target.set(plan.floorEntry((int) elapsed).getValue());
|
||||
while (workers.get() < target.get()) {
|
||||
int id = workers.incrementAndGet();
|
||||
executor.submit(() -> {
|
||||
while (running.get() && id <= target.get()) {
|
||||
long t0 = System.nanoTime();
|
||||
String outcome;
|
||||
try {
|
||||
HttpResponse<Void> r = client.send(HttpRequest.newBuilder(uri)
|
||||
.timeout(Duration.ofSeconds(10))
|
||||
.method(method, HttpRequest.BodyPublishers.noBody()).build(),
|
||||
HttpResponse.BodyHandlers.discarding());
|
||||
outcome = r.statusCode() == 200 ? "ok" : "http_" + r.statusCode();
|
||||
}
|
||||
catch (Exception ex) {
|
||||
outcome = ex.getClass().getSimpleName();
|
||||
}
|
||||
long t1 = System.nanoTime();
|
||||
long second = (t1 - start) / 1_000_000_000;
|
||||
perSecond.computeIfAbsent(second, k -> new ConcurrentHashMap<>())
|
||||
.computeIfAbsent(outcome, k -> new AtomicInteger()).incrementAndGet();
|
||||
if (outcome.equals("ok")) {
|
||||
latencies.add((t1 - t0) / 1_000_000);
|
||||
}
|
||||
}
|
||||
workers.decrementAndGet();
|
||||
return null;
|
||||
});
|
||||
}
|
||||
Thread.sleep(200);
|
||||
}
|
||||
}
|
||||
|
||||
Map<String, Integer> totals = new TreeMap<>();
|
||||
for (long s = 0; s <= seconds; s++) {
|
||||
Map<String, AtomicInteger> row = perSecond.getOrDefault(s, Map.of());
|
||||
StringBuilder line = new StringBuilder(String.format("t=%3ds conc=%-3d", s, plan.floorEntry((int) Math.min(s, seconds - 1)).getValue()));
|
||||
new TreeMap<>(row).forEach((k, v) -> {
|
||||
line.append(' ').append(k).append('=').append(v.get());
|
||||
totals.merge(k, v.get(), Integer::sum);
|
||||
});
|
||||
System.out.println(line);
|
||||
}
|
||||
List<Long> sorted = new ArrayList<>(latencies);
|
||||
Collections.sort(sorted);
|
||||
System.out.println("METHOD " + method);
|
||||
System.out.println("TOTAL " + totals);
|
||||
if (!sorted.isEmpty()) {
|
||||
System.out.printf("LATENCY ms p50=%d p90=%d p99=%d max=%d (n=%d)%n", pct(sorted, 50), pct(sorted, 90),
|
||||
pct(sorted, 99), sorted.get(sorted.size() - 1), sorted.size());
|
||||
}
|
||||
}
|
||||
|
||||
private static TreeMap<Integer, Integer> parse(String schedule) {
|
||||
TreeMap<Integer, Integer> plan = new TreeMap<>();
|
||||
if (!schedule.contains(":")) {
|
||||
plan.put(0, Integer.parseInt(schedule));
|
||||
return plan;
|
||||
}
|
||||
for (String step : schedule.split(",")) {
|
||||
String[] p = step.split(":");
|
||||
plan.put(Integer.parseInt(p[0]), Integer.parseInt(p[1]));
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
private static long pct(List<Long> sorted, int p) {
|
||||
return sorted.get(Math.min(sorted.size() - 1, (int) Math.ceil(p / 100.0 * sorted.size()) - 1));
|
||||
}
|
||||
|
||||
private LoadGen() {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package com.ankurm.k8s.web;
|
||||
|
||||
import jakarta.annotation.PostConstruct;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** Simulates an application that needs {@code demo.startup-delay} to become ready (cache warm-up, migrations). */
|
||||
@Component
|
||||
public class SlowStartup {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(SlowStartup.class);
|
||||
|
||||
private final long seconds;
|
||||
|
||||
public SlowStartup(@Value("${demo.startup-delay:0s}") java.time.Duration delay) {
|
||||
this.seconds = delay.toSeconds();
|
||||
}
|
||||
|
||||
@PostConstruct
|
||||
void warmUp() throws InterruptedException {
|
||||
if (seconds > 0) {
|
||||
log.info("Warming up for {} s", seconds);
|
||||
Thread.sleep(seconds * 1000);
|
||||
log.info("Warm-up complete");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package com.ankurm.k8s.web;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ThreadLocalRandom;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import io.micrometer.core.instrument.Gauge;
|
||||
import io.micrometer.core.instrument.MeterRegistry;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMethod;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
/**
|
||||
* The two kinds of work the Kubernetes experiments need.
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code /work?ms=300} - a request that takes a known time. In-flight requests are exported
|
||||
* as the gauge {@code app.inflight.requests}, which Prometheus sees as
|
||||
* {@code app_inflight_requests} and the HorizontalPodAutoscaler scales on
|
||||
* (docs/06-hpa-custom-metrics.md).</li>
|
||||
* <li>{@code /alloc?mb=64} - allocation-heavy work that keeps the garbage collector busy, for the
|
||||
* CPU-limit experiments (docs/04-cpu-limits-and-gc.md).</li>
|
||||
* </ul>
|
||||
*/
|
||||
@RestController
|
||||
public class WorkController {
|
||||
|
||||
private final AtomicInteger inFlight = new AtomicInteger();
|
||||
private final String pod;
|
||||
private final String revision;
|
||||
/** A slowly churning old generation, so collections have live data to trace, not just garbage. */
|
||||
private final List<byte[]> retained = new ArrayList<>();
|
||||
|
||||
public WorkController(MeterRegistry registry, @Value("${HOSTNAME:local}") String pod,
|
||||
@Value("${demo.revision:1}") String revision) {
|
||||
this.pod = pod;
|
||||
this.revision = revision;
|
||||
Gauge.builder("app.inflight.requests", inFlight, AtomicInteger::get)
|
||||
.description("Requests currently being processed by /work")
|
||||
.register(registry);
|
||||
}
|
||||
|
||||
/** GET and POST: POST is what the shutdown experiment uses, because HTTP clients retry GET. */
|
||||
@RequestMapping(path = "/work", method = {RequestMethod.GET, RequestMethod.POST})
|
||||
public Map<String, Object> work(@RequestParam(defaultValue = "100") long ms) throws InterruptedException {
|
||||
inFlight.incrementAndGet();
|
||||
try {
|
||||
Thread.sleep(ms);
|
||||
return Map.of("pod", pod, "revision", revision, "ms", ms);
|
||||
}
|
||||
finally {
|
||||
inFlight.decrementAndGet();
|
||||
}
|
||||
}
|
||||
|
||||
@GetMapping("/alloc")
|
||||
public Map<String, Object> alloc(@RequestParam(defaultValue = "64") int mb) {
|
||||
long start = System.nanoTime();
|
||||
long checksum = 0;
|
||||
ThreadLocalRandom random = ThreadLocalRandom.current();
|
||||
for (int i = 0; i < mb * 16; i++) { // 64 KiB chunks
|
||||
byte[] chunk = new byte[64 * 1024];
|
||||
chunk[random.nextInt(chunk.length)] = 1;
|
||||
checksum += chunk[0];
|
||||
if (random.nextInt(64) == 0) {
|
||||
synchronized (retained) {
|
||||
retained.add(chunk);
|
||||
if (retained.size() > 400) { // ~25 MiB of long-lived data, replaced over time
|
||||
retained.remove(random.nextInt(retained.size()));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return Map.of("pod", pod, "mb", mb, "ms", (System.nanoTime() - start) / 1_000_000, "checksum", checksum);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
spring:
|
||||
application:
|
||||
name: orders
|
||||
threads:
|
||||
virtual:
|
||||
enabled: true
|
||||
management:
|
||||
endpoints:
|
||||
web:
|
||||
exposure:
|
||||
include: health,info,prometheus
|
||||
endpoint:
|
||||
health:
|
||||
probes:
|
||||
# Auto-enabled when Spring Boot detects Kubernetes (the *_SERVICE_HOST/_PORT variables);
|
||||
# set explicitly so the same groups exist when you run the jar on a laptop.
|
||||
enabled: true
|
||||
show-details: always
|
||||
metrics:
|
||||
tags:
|
||||
application: ${spring.application.name}
|
||||
@@ -0,0 +1,31 @@
|
||||
package com.ankurm.k8s;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.web.server.LocalServerPort;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* The probe endpoints and the custom gauge exist with the names the Kubernetes manifests and the
|
||||
* Prometheus adapter rule depend on. If a Spring Boot upgrade renames any of them, this fails
|
||||
* before a rollout does.
|
||||
*/
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
class ProbesContractTest {
|
||||
|
||||
@LocalServerPort
|
||||
int port;
|
||||
|
||||
@Test
|
||||
void probeGroupsAndGaugeAreExposed() {
|
||||
RestClient client = RestClient.create("http://localhost:" + port);
|
||||
assertThat(client.get().uri("/actuator/health/liveness").retrieve().body(String.class)).contains("\"UP\"");
|
||||
assertThat(client.get().uri("/actuator/health/readiness").retrieve().body(String.class)).contains("\"UP\"");
|
||||
assertThat(client.get().uri("/actuator/prometheus").retrieve().body(String.class))
|
||||
.contains("app_inflight_requests{application=\"orders\"}");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user