Add shenandoah-jdk28: generational vs SATB Shenandoah, G1 and ZGC on a Spring Boot service (JEP 535 status probe included)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HmvoHH3ZnCEaPNXRpDxDZe
2026-10-24 10:53:17 +00:00
parent df13a904d7
commit b25bd65d4b
14 changed files with 460 additions and 0 deletions
@@ -0,0 +1,62 @@
+# shenandoah-jdk28 — generational Shenandoah vs SATB Shenandoah vs G1 vs ZGC on a Spring Boot service
+
+Companion module for the ankurm.com post
+**Generational Shenandoah by Default in JDK 28 (JEP 535): Benchmarked Against G1 and ZGC**.
+There is no `docs/` folder: this README and the captured files in [`output/`](output) are the documentation.
+
+## The honest status
+
+JEP 535 ("Shenandoah: Generational Mode by Default", which also deprecates the SATB mode) is **Targeted** to JDK 28 on
+openjdk.org, but it is **not integrated in Temurin 28+21 EA**. [`output/01-gc-mode-probe.txt`](output/01-gc-mode-probe.txt)
+shows `-XX:+UseShenandoahGC` still starts in *Snapshot-At-The-Beginning (SATB)* mode on both JDK 25.0.5+8 and 28+21.
+Generational mode is available on both as an explicit opt-in (`-XX:ShenandoahGCMode=generational`).
+So this module benchmarks **explicit generational vs explicit SATB**, which is what the default will switch between.
+Oracle's jdk.java.net builds do not contain Shenandoah at all; use Temurin (or another build that ships it).
+
+## Versions
+
+| Component | Version |
+|---|---|
+| JDK 25 | Temurin 25.0.5+8 (probe only) |
+| JDK 28 | Temurin 28+21 EA (28-beta+21-ea, build 202610221655) |
+| Spring Boot | 4.1.2 (`spring-boot-starter-webmvc`) |
+| Machine | 2 vCPU, 8 GB RAM Linux sandbox. A small box: absolute numbers are illustrative |
+| Heap | `-Xms1g -Xmx1g` for every configuration |
+
+## Quick start
+
+```console
+mvn -B -DskipTests package
+export J25=/path/to/temurin-25 J28=/path/to/temurin-28-ea
+scripts/run-all.sh                    # probe + 3 interleaved reps of 4 configurations, ~2.5 min per run
+scripts/more-reps.sh 4 5              # append more reps and re-summarise
+```
+
+## What is measured
+
+| File | What it is |
+|---|---|
+| [`src/main/java/com/ankurm/shenandoah/ShenandoahApplication.java`](src/main/java/com/ankurm/shenandoah/ShenandoahApplication.java) | `/work` replaces 8 of 200,000 1 KiB cache slots and allocates 100 short-lived 1 KiB arrays; `/info` reports the heap |
+| [`scripts/LoadGen.java`](scripts/LoadGen.java) | `open`: fixed-rate load (3,000 req/s, 64 virtual threads), latency measured from the *intended* send time; `closed`: peak throughput |
+| [`scripts/bench-one.sh`](scripts/bench-one.sh) | One run: 30 s warm-up, 60 s open-loop window, 20 s closed loop; pauses from `-Xlog:gc*`; peak RSS from `VmHWM` |
+| [`scripts/gc_pauses.py`](scripts/gc_pauses.py) | Stop-the-world pauses inside the measure window |
+| [`scripts/summarize.py`](scripts/summarize.py) | Median per configuration |
+
+## Results (median of 5 interleaved runs, JDK 28+21 EA)
+
+[`output/03-summary.txt`](output/03-summary.txt), from the per-run lines in [`output/02-runs.txt`](output/02-runs.txt):
+
+| Configuration | p50 ms | p99 ms | p99.9 ms | max ms | peak req/s | pause max ms | RSS MB |
+|---|---|---|---|---|---|---|---|
+| G1 | 0.63 | 29.2 | 128.1 | 179.5 | 8,900 | 111.6 | 1,205 |
+| ZGC (generational) | 0.60 | 7.4 | 17.3 | 36.4 | 7,117 | 0.17 | 1,250 |
+| Shenandoah SATB | 0.78 | 14.9 | 43.6 | 86.4 | 5,026 | 1.5 | 1,212 |
+| Shenandoah generational | 0.82 | 1,557.5 | 1,820.4 | 1,864.7 | 5,498 | 2.6 | 1,227 |
+
+**Read the last row carefully.** Per-run p99 for generational Shenandoah was 1,557, 3,077, 24.5, 1,658 and 692 ms: four of five
+runs contain a multi-second latency episode while the recorded stop-the-world pauses stay at a few milliseconds
+(one run had a 209 ms `Pause Full`). The GC logs do not show allocation failures (`Trigger: Failed to allocate` count is 0), and
+the `Cancelling GC: Concurrent GC` lines are old-generation marking being preempted for a young collection, which is normal.
+So the cause of the stalls is **not established** here. One SATB run (811 ms p99) also stalled with no GC event at all,
+which says this 2-vCPU machine produces stalls of its own. Treat the generational result as "did not behave on this box
+and this workload", not as a general verdict. A larger machine and a bigger heap are the obvious next experiment.
@@ -0,0 +1,20 @@
+== JDK 25: OpenJDK 64-Bit Server VM Temurin-25.0.5+8 (build 25.0.5+8-LTS, mixed mode, sharing)
+-- java -XX:+UseShenandoahGC -Xlog:gc+init -version
+[0.009s][info][gc,init] Mode: Snapshot-At-The-Beginning (SATB)
+-- java -XX:+UseShenandoahGC -XX:+PrintFlagsFinal -version  (ShenandoahGCMode)
+    ccstr ShenandoahGCMode                         = satb                                      {product} {default}
+-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:gc+init -version
+[0.009s][info][gc,init] Mode: Generational
+-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb -Xlog:gc+init -version
+[0.007s][info][gc,init] Mode: Snapshot-At-The-Beginning (SATB)
+
+== JDK 28 EA: OpenJDK 64-Bit Server VM Temurin-28+21-202610221655 (build 28-beta+21-ea, mixed mode, sharing)
+-- java -XX:+UseShenandoahGC -Xlog:gc+init -version
+[0.006s][info][gc,init] Mode: Snapshot-At-The-Beginning (SATB)
+-- java -XX:+UseShenandoahGC -XX:+PrintFlagsFinal -version  (ShenandoahGCMode)
+    ccstr ShenandoahGCMode                         = satb                                      {product} {default}
+-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:gc+init -version
+[0.008s][info][gc,init] Mode: Generational
+-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb -Xlog:gc+init -version
+[0.007s][info][gc,init] Mode: Snapshot-At-The-Beginning (SATB)
+
@@ -0,0 +1,20 @@
+label=g1 rep=1 target=3000 achieved=3000.0 requests=180000 p50_ms=0.61 p99_ms=28.69 p999_ms=95.48 max_ms=154.41 errors=0 peak_rps=8399.5 pauses=65 pause_p50_ms=23.459 pause_p99_ms=115.550 pause_max_ms=115.550 pause_total_ms=1349.5 rss_peak_mb=1205 heap_committed_mb=1024
+label=zgc rep=1 target=3000 achieved=3000.0 requests=180000 p50_ms=0.60 p99_ms=7.32 p999_ms=65.10 max_ms=108.85 errors=0 peak_rps=7540.4 pauses=158 pause_p50_ms=0.014 pause_p99_ms=0.060 pause_max_ms=0.141 pause_total_ms=2.6 rss_peak_mb=1250 heap_committed_mb=1024
+label=shenandoah-satb rep=1 target=3000 achieved=3000.0 requests=180000 p50_ms=0.80 p99_ms=14.94 p999_ms=29.12 max_ms=53.70 errors=0 peak_rps=5025.8 pauses=204 pause_p50_ms=0.165 pause_p99_ms=0.945 pause_max_ms=1.125 pause_total_ms=44.1 rss_peak_mb=1213 heap_committed_mb=1024
+label=shenandoah-generational rep=1 target=3000 achieved=3000.0 requests=180000 p50_ms=0.89 p99_ms=1557.51 p999_ms=1866.05 max_ms=1906.45 errors=0 peak_rps=5484.3 pauses=319 pause_p50_ms=0.170 pause_p99_ms=1.236 pause_max_ms=1.418 pause_total_ms=76.5 rss_peak_mb=1227 heap_committed_mb=1024
+label=g1 rep=2 target=3000 achieved=3000.0 requests=180000 p50_ms=0.66 p99_ms=29.17 p999_ms=128.06 max_ms=179.49 errors=0 peak_rps=8899.7 pauses=63 pause_p50_ms=24.557 pause_p99_ms=110.232 pause_max_ms=110.232 pause_total_ms=1366.7 rss_peak_mb=1207 heap_committed_mb=1024
+label=zgc rep=2 target=3000 achieved=3000.0 requests=180000 p50_ms=0.60 p99_ms=6.98 p999_ms=15.80 max_ms=33.36 errors=0 peak_rps=7060.8 pauses=154 pause_p50_ms=0.014 pause_p99_ms=0.090 pause_max_ms=0.183 pause_total_ms=2.6 rss_peak_mb=1241 heap_committed_mb=1024
+label=shenandoah-satb rep=2 target=3000 achieved=3000.0 requests=180000 p50_ms=0.78 p99_ms=14.58 p999_ms=27.28 max_ms=56.25 errors=0 peak_rps=4542.2 pauses=204 pause_p50_ms=0.160 pause_p99_ms=0.981 pause_max_ms=1.067 pause_total_ms=46.0 rss_peak_mb=1212 heap_committed_mb=1024
+label=shenandoah-generational rep=2 target=3000 achieved=3000.0 requests=180000 p50_ms=0.93 p99_ms=3077.07 p999_ms=3222.87 max_ms=3284.93 errors=0 peak_rps=5497.7 pauses=328 pause_p50_ms=0.184 pause_p99_ms=1.544 pause_max_ms=209.368 pause_total_ms=292.7 rss_peak_mb=1219 heap_committed_mb=1024
+label=g1 rep=3 target=3000 achieved=3000.0 requests=180000 p50_ms=0.65 p99_ms=29.18 p999_ms=137.90 max_ms=194.77 errors=0 peak_rps=9385.5 pauses=65 pause_p50_ms=24.353 pause_p99_ms=123.149 pause_max_ms=123.149 pause_total_ms=1383.9 rss_peak_mb=1203 heap_committed_mb=1024
+label=zgc rep=3 target=3000 achieved=3000.0 requests=180000 p50_ms=0.62 p99_ms=7.37 p999_ms=15.73 max_ms=36.38 errors=0 peak_rps=7816.9 pauses=156 pause_p50_ms=0.014 pause_p99_ms=0.099 pause_max_ms=0.184 pause_total_ms=2.6 rss_peak_mb=1257 heap_committed_mb=1024
+label=shenandoah-satb rep=3 target=3000 achieved=3000.0 requests=180000 p50_ms=0.69 p99_ms=811.54 p999_ms=1116.58 max_ms=1186.35 errors=0 peak_rps=5402.7 pauses=213 pause_p50_ms=0.144 pause_p99_ms=1.012 pause_max_ms=2.851 pause_total_ms=43.8 rss_peak_mb=1212 heap_committed_mb=1024
+label=shenandoah-generational rep=3 target=3000 achieved=3000.0 requests=180000 p50_ms=0.82 p99_ms=24.49 p999_ms=103.01 max_ms=150.61 errors=0 peak_rps=5270.0 pauses=282 pause_p50_ms=0.172 pause_p99_ms=1.398 pause_max_ms=2.988 pause_total_ms=72.0 rss_peak_mb=1229 heap_committed_mb=1024
+label=g1 rep=4 target=3000 achieved=3000.0 requests=180000 p50_ms=0.63 p99_ms=42.72 p999_ms=117.79 max_ms=146.07 errors=0 peak_rps=8847.7 pauses=55 pause_p50_ms=25.266 pause_p99_ms=111.576 pause_max_ms=111.576 pause_total_ms=1693.4 rss_peak_mb=1205 heap_committed_mb=1024
+label=zgc rep=4 target=3000 achieved=3000.0 requests=180000 p50_ms=0.67 p99_ms=7.98 p999_ms=17.28 max_ms=31.52 errors=0 peak_rps=6923.5 pauses=155 pause_p50_ms=0.015 pause_p99_ms=0.097 pause_max_ms=0.168 pause_total_ms=3.0 rss_peak_mb=1250 heap_committed_mb=1024
+label=shenandoah-satb rep=4 target=3000 achieved=3000.0 requests=180000 p50_ms=0.87 p99_ms=21.25 p999_ms=43.63 max_ms=86.39 errors=0 peak_rps=4639.2 pauses=210 pause_p50_ms=0.163 pause_p99_ms=0.743 pause_max_ms=1.475 pause_total_ms=44.1 rss_peak_mb=1209 heap_committed_mb=1024
+label=shenandoah-generational rep=4 target=3000 achieved=3000.0 requests=180000 p50_ms=0.75 p99_ms=1657.87 p999_ms=1820.38 max_ms=1864.68 errors=0 peak_rps=5519.6 pauses=293 pause_p50_ms=0.177 pause_p99_ms=2.230 pause_max_ms=2.553 pause_total_ms=76.6 rss_peak_mb=1220 heap_committed_mb=1024
+label=g1 rep=5 target=3000 achieved=3000.0 requests=180000 p50_ms=0.60 p99_ms=66.31 p999_ms=379.56 max_ms=423.34 errors=0 peak_rps=9775.1 pauses=70 pause_p50_ms=25.645 pause_p99_ms=101.489 pause_max_ms=101.489 pause_total_ms=1443.9 rss_peak_mb=1202 heap_committed_mb=1024
+label=zgc rep=5 target=3000 achieved=3000.0 requests=180000 p50_ms=0.59 p99_ms=7.49 p999_ms=48.76 max_ms=83.78 errors=0 peak_rps=7117.4 pauses=147 pause_p50_ms=0.014 pause_p99_ms=0.080 pause_max_ms=0.112 pause_total_ms=2.4 rss_peak_mb=1243 heap_committed_mb=1024
+label=shenandoah-satb rep=5 target=3000 achieved=3000.0 requests=180000 p50_ms=0.64 p99_ms=12.71 p999_ms=62.14 max_ms=101.88 errors=0 peak_rps=5448.0 pauses=191 pause_p50_ms=0.147 pause_p99_ms=1.192 pause_max_ms=1.512 pause_total_ms=36.9 rss_peak_mb=1214 heap_committed_mb=1024
+label=shenandoah-generational rep=5 target=3000 achieved=3000.0 requests=180000 p50_ms=0.72 p99_ms=691.58 p999_ms=916.21 max_ms=959.93 errors=0 peak_rps=5745.0 pauses=277 pause_p50_ms=0.176 pause_p99_ms=1.283 pause_max_ms=1.359 pause_total_ms=67.2 rss_peak_mb=1231 heap_committed_mb=1024
@@ -0,0 +1,7 @@
+median of 5 runs per configuration (latency from a fixed-rate open loop; throughput from a closed loop)
+
+configuration                      p50 ms         p99 ms       p99.9 ms         max ms     peak req/s         pauses   pause p99 ms   pause max ms pause total ms         RSS MB
+g1                                   0.63           29.2          128.1          179.5           8900             65        111.576        111.576           1384           1205
+zgc                                  0.60            7.4           17.3           36.4           7117            155          0.090          0.168              3           1250
+shenandoah-satb                      0.78           14.9           43.6           86.4           5026            204          0.981          1.475             44           1212
+shenandoah-generational              0.82         1557.5         1820.4         1864.7           5498            293          1.398          2.553             76           1227
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-parent</artifactId>
+    <version>4.1.2</version>
+    <relativePath/>
+  </parent>
+  <groupId>com.ankurm</groupId>
+  <artifactId>shenandoah-jdk28</artifactId>
+  <version>1.0.0</version>
+  <name>shenandoah-jdk28</name>
+  <description>Same Spring Boot load on G1, generational ZGC and Shenandoah (SATB and generational) on JDK 28 early access</description>
+  <properties>
+    <java.version>25</java.version>
+  </properties>
+  <dependencies>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-webmvc</artifactId>
+    </dependency>
+  </dependencies>
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.springframework.boot</groupId>
+        <artifactId>spring-boot-maven-plugin</artifactId>
+      </plugin>
+    </plugins>
+  </build>
+</project>
@@ -0,0 +1,94 @@
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.LockSupport;
+
+/**
+ * Load generator, run with the source launcher: java LoadGen.java open|closed URL RATE_OR_0 CONCURRENCY WARMUP_S MEASURE_S
+ *
+ * open   : requests are SCHEDULED at a fixed rate and latency is measured from the scheduled time, so a server
+ *          that stalls is charged for the requests that queued behind the stall (no coordinated omission).
+ * closed : CONCURRENCY clients send back to back; the result is the peak throughput.
+ *
+ * Prints one line: mode=... achieved=... p50_ms=... p99_ms=... p999_ms=... max_ms=... errors=... window_start_ms=... window_end_ms=...
+ * (the window is the measured interval in epoch milliseconds, so GC pauses can be cut to exactly that interval).
+ */
+public class LoadGen {
+
+    public static void main(String[] a) throws Exception {
+        String mode = a[0];
+        URI uri = URI.create(a[1]);
+        double rate = Double.parseDouble(a[2]);
+        int concurrency = Integer.parseInt(a[3]);
+        int warmup = Integer.parseInt(a[4]);
+        int measure = Integer.parseInt(a[5]);
+        boolean open = mode.equals("open");
+
+        HttpClient client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1)
+                .connectTimeout(Duration.ofSeconds(5)).build();
+        HttpRequest request = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(30)).GET().build();
+
+        long start = System.nanoTime();
+        long epochStartMs = System.currentTimeMillis();
+        long measureFrom = start + warmup * 1_000_000_000L;
+        long end = measureFrom + measure * 1_000_000_000L;
+        int capacity = (int) (open ? rate * measure * 2 + 1000 : 2_000_000);
+        long[] latencies = new long[capacity];
+        AtomicInteger count = new AtomicInteger();
+        AtomicLong errors = new AtomicLong();
+        AtomicLong next = new AtomicLong();
+        long intervalNanos = open ? (long) (1e9 / rate) : 0;
+
+        ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor();
+        for (int t = 0; t < concurrency; t++) {
+            pool.submit(() -> {
+                while (true) {
+                    long intended;
+                    if (open) {
+                        intended = start + next.getAndIncrement() * intervalNanos;
+                        if (intended >= end) return;
+                        long wait = intended - System.nanoTime();
+                        if (wait > 0) LockSupport.parkNanos(wait);
+                    } else {
+                        intended = System.nanoTime();
+                        if (intended >= end) return;
+                    }
+                    try {
+                        HttpResponse<Void> r = client.send(request, HttpResponse.BodyHandlers.discarding());
+                        long done = System.nanoTime();
+                        if (r.statusCode() != 200) errors.incrementAndGet();
+                        if (intended >= measureFrom) {
+                            int i = count.getAndIncrement();
+                            if (i < latencies.length) latencies[i] = done - intended;
+                        }
+                    } catch (Exception e) {
+                        errors.incrementAndGet();
+                    }
+                }
+            });
+        }
+        pool.shutdown();
+        pool.awaitTermination(measure + warmup + 120, TimeUnit.SECONDS);
+
+        int n = Math.min(count.get(), latencies.length);
+        long[] sorted = Arrays.copyOf(latencies, n);
+        Arrays.sort(sorted);
+        System.out.printf("mode=%s target=%.0f achieved=%.1f requests=%d p50_ms=%.2f p99_ms=%.2f p999_ms=%.2f max_ms=%.2f errors=%d window_start_ms=%d window_end_ms=%d%n",
+                mode, rate, n / (double) measure, n, ms(sorted, 0.50), ms(sorted, 0.99), ms(sorted, 0.999),
+                n == 0 ? 0 : sorted[n - 1] / 1e6, errors.get(),
+                epochStartMs + warmup * 1000L, epochStartMs + (warmup + measure) * 1000L);
+    }
+
+    static double ms(long[] sorted, double q) {
+        if (sorted.length == 0) return 0;
+        return sorted[Math.min(sorted.length - 1, (int) Math.ceil(q * sorted.length) - 1)] / 1e6;
+    }
+}
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# One benchmark run: start the server with the given JVM flags, warm up, measure at a fixed request rate
+# (latency), then measure peak throughput, and print ONE result line.
+#
+#   bench-one.sh <java-home> <label> <rep> [jvm flags...]
+#
+# Env: RATE (default 3000 req/s), WARMUP (30 s), MEASURE (60 s), PORT (18090)
+set -u
+JH=$1; LABEL=$2; REP=$3; shift 3
+HERE="$(cd "$(dirname "$0")" && pwd)"
+JAR="$HERE/../target/shenandoah-jdk28-1.0.0.jar"
+RATE=${RATE:-3000}; WARMUP=${WARMUP:-30}; MEASURE=${MEASURE:-60}; PORT=${PORT:-18090}
+OUT="$HERE/../target/gclogs"; mkdir -p "$OUT"
+LOG="$OUT/$LABEL-$REP.log"
+
+env -u JAVA_TOOL_OPTIONS "$JH/bin/java" -Xms1g -Xmx1g "$@" \
+    -Xlog:gc*:file="$LOG":time,uptime \
+    -jar "$JAR" --server.port=$PORT --spring.main.banner-mode=off --logging.level.root=WARN >/dev/null 2>&1 &
+PID=$!
+for _ in $(seq 1 200); do
+  curl -s -o /dev/null -m 1 "localhost:$PORT/work" && break
+  kill -0 $PID 2>/dev/null || { echo "label=$LABEL rep=$REP FAILED: JVM exited"; exit 1; }
+  sleep 0.3
+done
+
+OPEN=$(env -u JAVA_TOOL_OPTIONS "$JH/bin/java" "$HERE/LoadGen.java" open "http://localhost:$PORT/work" "$RATE" 64 "$WARMUP" "$MEASURE")
+CLOSED=$(env -u JAVA_TOOL_OPTIONS "$JH/bin/java" "$HERE/LoadGen.java" closed "http://localhost:$PORT/work" 0 16 10 20)
+RSS_PEAK_MB=$(awk '/VmHWM/ {printf "%d", $2/1024}' /proc/$PID/status)
+COMMITTED=$(curl -s "localhost:$PORT/info" | sed -n 's/.*"committedMiB":\([0-9]*\).*/\1/p')
+kill $PID 2>/dev/null; wait $PID 2>/dev/null
+
+W0=$(sed -n 's/.*window_start_ms=\([0-9]*\).*/\1/p' <<<"$OPEN"); W1=$(sed -n 's/.*window_end_ms=\([0-9]*\).*/\1/p' <<<"$OPEN")
+PAUSES=$(python3 "$HERE/gc_pauses.py" "$LOG" "$W0" "$W1")
+CLOSED_TPUT=$(sed -n 's/.*achieved=\([0-9.]*\).*/\1/p' <<<"$CLOSED")
+echo "label=$LABEL rep=$REP $(sed 's/ window_start_ms=.*//; s/^mode=open //' <<<"$OPEN") peak_rps=$CLOSED_TPUT $PAUSES rss_peak_mb=$RSS_PEAK_MB heap_committed_mb=$COMMITTED"
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""Stop-the-world pauses from a -Xlog:gc*:...:time,uptime log, restricted to an epoch-ms window.
+
+usage: gc_pauses.py LOG WINDOW_START_MS WINDOW_END_MS
+
+A pause is a log line that contains "Pause" and ends in "<n>ms" (G1 "Pause Young ... 4.2ms", ZGC
+"Y: Pause Mark Start 0.018ms", Shenandoah "Pause Init Mark 0.052ms"). The line without a duration that
+G1 and Shenandoah print when a pause *starts* is ignored, so nothing is counted twice.
+"""
+import re
+import sys
+from datetime import datetime
+
+LINE = re.compile(r"^\[(\d{4}-\d\d-\d\dT[\d:.]+[+-]\d{4})\].*?\bGC\(\d+\) .*Pause .* ([0-9.]+)ms$")
+
+log, lo, hi = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
+pauses = []
+for line in open(log, errors="replace"):
+    m = LINE.match(line.rstrip("\n"))
+    if not m:
+        continue
+    ts = datetime.strptime(m.group(1), "%Y-%m-%dT%H:%M:%S.%f%z").timestamp() * 1000
+    if lo <= ts <= hi:
+        pauses.append(float(m.group(2)))
+pauses.sort()
+n = len(pauses)
+
+
+def q(p):
+    return pauses[min(n - 1, int(p * n))] if n else 0.0
+
+
+print(f"pauses={n} pause_p50_ms={q(0.50):.3f} pause_p99_ms={q(0.99):.3f} pause_max_ms={pauses[-1] if n else 0.0:.3f} pause_total_ms={sum(pauses):.1f}")
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Appends extra interleaved repetitions to output/02-runs.txt (same protocol as run-all.sh), then re-summarises.
+#   scripts/more-reps.sh FIRST_REP LAST_REP        e.g. scripts/more-reps.sh 4 5
+set -eu
+cd "$(dirname "$0")/.."
+: "${J28:?set J28}"
+for rep in $(seq "$1" "$2"); do
+  ./scripts/bench-one.sh "$J28" g1                      "$rep" -XX:+UseG1GC                                           | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" zgc                     "$rep" -XX:+UseZGC                                            | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" shenandoah-satb         "$rep" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb         | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" shenandoah-generational "$rep" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational | tee -a output/02-runs.txt
+done
+python3 scripts/summarize.py output/02-runs.txt | tee output/03-summary.txt
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# What does plain -XX:+UseShenandoahGC select on each JDK? Needs J25 and J28 pointing at JDKs that include Shenandoah
+# (Temurin does; the Oracle builds from jdk.java.net do not).
+set -u
+for entry in "JDK 25:$J25" "JDK 28 EA:$J28"; do
+  name=${entry%%:*}; home=${entry#*:}
+  echo "== $name: $(env -u JAVA_TOOL_OPTIONS "$home/bin/java" -version 2>&1 | sed -n 3p)"
+  echo "-- java -XX:+UseShenandoahGC -Xlog:gc+init -version"
+  env -u JAVA_TOOL_OPTIONS "$home/bin/java" -XX:+UseShenandoahGC -Xlog:gc+init -Xmx256m -version 2>&1 | grep -E 'Mode:|warning|WARNING|deprecat' 
+  echo "-- java -XX:+UseShenandoahGC -XX:+PrintFlagsFinal -version  (ShenandoahGCMode)"
+  env -u JAVA_TOOL_OPTIONS "$home/bin/java" -XX:+UseShenandoahGC -XX:+PrintFlagsFinal -Xmx256m -version 2>&1 | grep -E 'ShenandoahGCMode'
+  echo "-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:gc+init -version"
+  env -u JAVA_TOOL_OPTIONS "$home/bin/java" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:gc+init -Xmx256m -version 2>&1 | grep -E 'Mode:|warning|WARNING|deprecat'
+  echo "-- java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb -Xlog:gc+init -version"
+  env -u JAVA_TOOL_OPTIONS "$home/bin/java" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb -Xlog:gc+init -Xmx256m -version 2>&1 | grep -E 'Mode:|warning|WARNING|deprecat'
+  echo
+done
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Regenerates everything in output/.
+#   export J25=/path/to/temurin-25 J28=/path/to/temurin-28-ea     (both must include Shenandoah: use Temurin, not the jdk.java.net builds)
+#   mvn -B -DskipTests package
+#   scripts/run-all.sh                  # 4 configurations x REPS runs, interleaved, about 2.5 minutes per run
+#
+# Env: REPS (default 3), RATE / WARMUP / MEASURE (see bench-one.sh)
+set -eu
+cd "$(dirname "$0")/.."
+: "${J25:?set J25}" "${J28:?set J28}"
+REPS=${REPS:-3}
+./scripts/probe.sh > output/01-gc-mode-probe.txt 2>&1
+: > output/02-runs.txt
+for rep in $(seq 1 "$REPS"); do
+  # interleaved so slow drift on the machine hits every configuration alike
+  ./scripts/bench-one.sh "$J28" g1                      "$rep" -XX:+UseG1GC                                                  | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" zgc                     "$rep" -XX:+UseZGC                                                   | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" shenandoah-satb         "$rep" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=satb                | tee -a output/02-runs.txt
+  ./scripts/bench-one.sh "$J28" shenandoah-generational "$rep" -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational        | tee -a output/02-runs.txt
+done
+python3 scripts/summarize.py output/02-runs.txt | tee output/03-summary.txt
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""Median across repetitions, one row per configuration. usage: summarize.py output/02-runs.txt"""
+import re
+import statistics
+import sys
+from collections import OrderedDict
+
+rows = OrderedDict()
+for line in open(sys.argv[1]):
+    if "FAILED" in line or not line.strip():
+        continue
+    kv = dict(re.findall(r"(\w+)=([^\s]+)", line))
+    rows.setdefault(kv["label"], []).append({k: float(v) for k, v in kv.items() if k not in ("label",)})
+
+cols = [
+    ("p50 ms", "p50_ms", "{:.2f}"),
+    ("p99 ms", "p99_ms", "{:.1f}"),
+    ("p99.9 ms", "p999_ms", "{:.1f}"),
+    ("max ms", "max_ms", "{:.1f}"),
+    ("peak req/s", "peak_rps", "{:.0f}"),
+    ("pauses", "pauses", "{:.0f}"),
+    ("pause p99 ms", "pause_p99_ms", "{:.3f}"),
+    ("pause max ms", "pause_max_ms", "{:.3f}"),
+    ("pause total ms", "pause_total_ms", "{:.0f}"),
+    ("RSS MB", "rss_peak_mb", "{:.0f}"),
+]
+reps = max(len(v) for v in rows.values())
+print(f"median of {reps} runs per configuration (latency from a fixed-rate open loop; throughput from a closed loop)")
+print()
+print(f"{'configuration':<26}" + "".join(f"{c[0]:>15}" for c in cols))
+for label, runs in rows.items():
+    cells = [c[2].format(statistics.median(r[c[1]] for r in runs)) for c in cols]
+    print(f"{label:<26}" + "".join(f"{x:>15}" for x in cells))
@@ -0,0 +1,71 @@
+package com.ankurm.shenandoah;
+
+import java.lang.management.ManagementFactory;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicReferenceArray;
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+/**
+ * A server with the shape a generational collector is built for: a large, slowly changing live set
+ * (the cache) plus a steady stream of short-lived garbage (each request).
+ */
+@SpringBootApplication
+public class ShenandoahApplication {
+
+    public static void main(String[] args) {
+        SpringApplication.run(ShenandoahApplication.class, args);
+    }
+
+    @RestController
+    static class Api {
+
+        /** 200,000 slots of 1 KiB is a live set of roughly 210 MiB. */
+        static final int SLOTS = 200_000;
+        /** Slots replaced per request: the old value becomes old-generation garbage. */
+        static final int REPLACED_PER_REQUEST = 8;
+        /** Short-lived 1 KiB arrays allocated per request: about 100 KiB of young garbage. */
+        static final int TEMPORARIES = 100;
+
+        private final AtomicReferenceArray<byte[]> cache = new AtomicReferenceArray<>(SLOTS);
+        /** Written on every request so the JIT cannot discard the temporaries. */
+        private volatile int sink;
+
+        Api() {
+            for (int i = 0; i < SLOTS; i++) {
+                cache.set(i, new byte[1024]);
+            }
+        }
+
+        @GetMapping("/work")
+        String work() {
+            ThreadLocalRandom r = ThreadLocalRandom.current();
+            for (int i = 0; i < REPLACED_PER_REQUEST; i++) {
+                cache.set(r.nextInt(SLOTS), new byte[1024]);
+            }
+            byte[][] temp = new byte[TEMPORARIES][];
+            for (int i = 0; i < TEMPORARIES; i++) {
+                temp[i] = new byte[1024];
+                temp[i][0] = (byte) i;
+            }
+            sink = temp[r.nextInt(TEMPORARIES)][0] + cache.get(r.nextInt(SLOTS))[0];
+            return "ok";
+        }
+
+        @GetMapping("/info")
+        Map<String, Object> info() {
+            Runtime rt = Runtime.getRuntime();
+            Map<String, Object> m = new LinkedHashMap<>();
+            m.put("javaVersion", System.getProperty("java.version"));
+            m.put("committedMiB", rt.totalMemory() >> 20);
+            m.put("maxMiB", rt.maxMemory() >> 20);
+            m.put("gc", ManagementFactory.getGarbageCollectorMXBeans().stream().map(b -> b.getName()).toList());
+            m.put("jvmUptimeMs", ManagementFactory.getRuntimeMXBean().getUptime());
+            return m;
+        }
+    }
+}