diff --git a/README.md b/README.md index 20538d2..3844673 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ files. | [`configuration-properties/`](configuration-properties) | [@ConfigurationProperties vs @Value in Spring Boot 4](https://ankurm.com/) | relaxed binding measured three ways, record constructor binding, validation, IDE metadata generation | | [`profiles-and-config/`](profiles-and-config) | [Spring Boot Profiles Done Right](https://ankurm.com/) | the precedence stack made visible, config trees and ConfigMaps, why a profile file loses to an environment variable | | [`spring-aop/`](spring-aop) | [Spring AOP Explained](https://ankurm.com/) | every pointcut designator with real matches, JDK vs CGLIB proxies, six aspects that do not fire | +| [`docker-images/`](docker-images) | [Dockerizing Spring Boot 4: Layered Jars, Buildpacks, Distroless and Image Size Benchmarks](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/) | one service packaged nine ways and measured: size on disk and pushed, rebuild delta, startup, PID 1 and signals, jlink, the JDK 25 AOT cache | Articles whose text is kept here rather than only on the blog have it under `/post/` — `post.md` for the body and `meta.md` for the title, excerpt and @@ -21,7 +22,8 @@ categories. ## Running any of them -Each project needs a JDK 25 and Maven 3.9: +Each project needs a JDK 25 and Maven 3.9. `docker-images` also needs Docker; its README lists +the rest: ```bash cd diff --git a/docker-images/.gitignore b/docker-images/.gitignore new file mode 100644 index 0000000..16ae243 --- /dev/null +++ b/docker-images/.gitignore @@ -0,0 +1,2 @@ +bindings/ca-certificates/*.pem +bindings/dependency-mapping/*.jar diff --git a/docker-images/README.md b/docker-images/README.md new file mode 100644 index 0000000..09cab49 --- /dev/null +++ b/docker-images/README.md @@ -0,0 +1,99 @@ +# Dockerizing Spring Boot 4: layered jars, buildpacks, distroless - measured + +Companion project for [**Dockerizing Spring Boot 4: Layered Jars, Buildpacks, Distroless and Image Size Benchmarks**](https://ankurm.com/dockerizing-spring-boot-4-layered-jars-buildpacks-distroless/) +on ankurm.com. + +One ordinary Spring Boot 4 service (web, validation, Actuator, Prometheus - a 25 MB fat jar) packaged +nine ways and measured the same way: size on disk, size over the wire, what a one-line code change +costs to push, who the process runs as, whether there is a shell, and how long it takes to become +ready. Every number in the article is in [`docs/output/`](docs/output). + +## Versions + +| | | +|---|---| +| Spring Boot | 4.1.1 | +| JDK in the images | Temurin 25.0.4 (`eclipse-temurin:25-*`), Temurin 25.0.4.1 (distroless), Liberica 25.0.4 (buildpacks) | +| Docker Engine | 29.4.3, classic `overlay2` image store | +| Paketo builder | `paketobuildpacks/builder-noble-java-tiny` (the Boot 4.1.1 default) | +| Jib | 3.5.2 | + +Base image digests the committed output was measured with: + +| Image | Index digest | +|---|---| +| `eclipse-temurin:25-jre` | `sha256:15090d159279e5c158473eccb48cd87f57b3e3a47511a797eb5a7a7ea6f86b0f` | +| `eclipse-temurin:25-jdk` | `sha256:dcf835e52330939b6c9f90ecab8aafcbcaa8fbf48423db44de884cf978c10144` | +| `eclipse-temurin:25-jre-alpine` | `sha256:3137541deb3cac6626b5d9a4a2187bc0d6a34312f858bd2c67dd01e732e6b682` | +| `gcr.io/distroless/java25-debian13:nonroot` | `sha256:fce4a1d66284e8866c46113d9bdc286c46fb8c3c3f0a098f877034349e88debe` | +| `gcr.io/distroless/java-base-debian13:nonroot` | `sha256:40a4046b8663ac0226eae964d5844c44999e006025d44ddc786499e3d6cac1a5` | +| `paketobuildpacks/builder-noble-java-tiny:latest` | `sha256:a836fcd7b0e4c047673e57039f243bc6d6d505fee86bce690579359839fdf7ea` | + +## Quickstart + +```bash +export JAVA_HOME=/path/to/jdk-25 +mvn -DskipTests package +./scripts/build-images.sh # all nine variants, tagged sbd/docker-images: +docker run --rm -p 8080:8080 sbd/docker-images:layered-distroless +./scripts/run-all.sh # every measurement (~15 min, needs crane for a local registry) +``` + +## The variants + +| Tag | Built from | | +|---|---|---| +| `fatjar-jdk` | [`Dockerfile.fatjar-jdk`](docker/Dockerfile.fatjar-jdk) | the tutorial default | +| `fatjar-jre` | [`Dockerfile.fatjar-jre`](docker/Dockerfile.fatjar-jre) | | +| `layered-jre` | [`Dockerfile.layered-jre`](docker/Dockerfile.layered-jre) | `jarmode=tools extract --layers` | +| `layered-alpine` | [`Dockerfile.layered-alpine`](docker/Dockerfile.layered-alpine) | musl | +| `layered-distroless` | [`Dockerfile.layered-distroless`](docker/Dockerfile.layered-distroless) | **the recommendation** | +| `jlink-distroless` | [`Dockerfile.jlink-distroless`](docker/Dockerfile.jlink-distroless) | custom runtime on `java-base` | +| `aot-cache` | [`Dockerfile.aot-cache`](docker/Dockerfile.aot-cache) | JDK 25 AOT cache training run | +| `buildpacks` | `mvn spring-boot:build-image` | no Dockerfile | +| `jib` | `mvn jib:dockerBuild` | no Dockerfile, no daemon needed for `jib:build` | + +Plus three shell-form `ENTRYPOINT` traps (`shell-form*`), an AOT cache on the wrong JVM +(`aot-cache-mismatch`) and a jlink image built from `jdeps` output alone (`jlink-jdeps-only`). + +## Documentation + +1. [The variants and how each is built](docs/01-the-variants.md) +2. [Measuring size: on disk, over the wire, and why `docker images` disagrees with itself](docs/02-measuring-size.md) +3. [Layered jars: the win is the second push, not the first](docs/03-layered-jars.md) +4. [Buildpacks: what the builder decides for you](docs/04-buildpacks.md) +5. [jlink: 133 MB smaller, 15 metrics quieter](docs/05-jlink.md) +6. [The JDK 25 AOT cache: faster start, bigger pushes, and the silent fallback](docs/06-aot-cache.md) +7. [PID 1 and signals: shell form, dash and BusyBox](docs/07-pid1-and-signals.md) +8. [Living with distroless](docs/08-distroless-in-practice.md) + +## Captured output + +| File | Produced by | +|---|---| +| [`image-matrix.txt`](docs/output/image-matrix.txt) | `scripts/measure.sh` | +| [`rebuild-delta.txt`](docs/output/rebuild-delta.txt) | `scripts/measure-rebuild.sh` | +| [`pid1-and-signals.txt`](docs/output/pid1-and-signals.txt) | `scripts/demo-signals.sh` | +| [`jlink-metrics.txt`](docs/output/jlink-metrics.txt) | `scripts/demo-jlink-metrics.sh` | +| [`aot-cache-mismatch.txt`](docs/output/aot-cache-mismatch.txt) | `scripts/demo-aot-mismatch.sh` | +| [`buildpacks-memory.txt`](docs/output/buildpacks-memory.txt) | `scripts/demo-buildpacks-memory.sh` | +| [`image-store-size-difference.txt`](docs/output/image-store-size-difference.txt) | by hand - needs a daemon restart | + +## Findings worth the trip + +- **Layering does not make the image smaller - it makes the next push roughly 4,000 times smaller**: 23.4 MB + for the fat jar against 6 KB for the application layer. +- **The AOT cache halves startup (3.36 s → 1.64 s) and adds 15 MB to every push**, because the + training run is invalidated by every code change - and on a different JVM build it fails with + `[error][aot]` lines and the application starts anyway, at the old speed. +- **`jdeps` misses `jdk.management`.** The jlink image runs, and 15 metrics - GC pauses, CPU usage - + quietly disappear from `/actuator/prometheus`. +- **Shell-form `ENTRYPOINT` on the Temurin (Ubuntu, dash) images swallows SIGTERM**: `docker stop` + takes 10.2 s and ends in SIGKILL. On Alpine, BusyBox execs the command and it works - so the same + Dockerfile behaves differently by base image. +- **The buildpacks image does not start under a 512 MiB memory limit**: the memory calculator reserves + stacks for 250 threads and a 240 MiB code cache first, and exits with code 82. +- **The buildpack's JRE layer is 276 MB against Temurin's 200 MB**: BellSoft's JRE ships a 74 MB + client VM (`lib/client`, with its own CDS archives) next to the server VM. +- **`docker images` reports 496 MB or 352 MB for the same `eclipse-temurin:25-jre`** depending on the + daemon's image store. diff --git a/docker-images/bindings/ca-certificates/README.md b/docker-images/bindings/ca-certificates/README.md new file mode 100644 index 0000000..23c84ec --- /dev/null +++ b/docker-images/bindings/ca-certificates/README.md @@ -0,0 +1,8 @@ +# ca-certificates binding + +Used only with `-Pcorporate-proxy`. Put your proxy's CA certificate here as a `.pem` file next to +`type`; the Paketo CA Certificates buildpack adds it to the build container's trust store so the +Java buildpack can download its JRE through a TLS-intercepting proxy. + +`*.pem` files in this directory are git-ignored on purpose - a CA certificate from one network does +not belong in a repository. See [docs/04-buildpacks.md](../../docs/04-buildpacks.md). diff --git a/docker-images/bindings/ca-certificates/type b/docker-images/bindings/ca-certificates/type new file mode 100644 index 0000000..54619ed --- /dev/null +++ b/docker-images/bindings/ca-certificates/type @@ -0,0 +1 @@ +ca-certificates \ No newline at end of file diff --git a/docker-images/bindings/dependency-mapping/32e47c2139d6379836910f0d1cc253a019ac282f3aeea12237069f00046279ad b/docker-images/bindings/dependency-mapping/32e47c2139d6379836910f0d1cc253a019ac282f3aeea12237069f00046279ad new file mode 100644 index 0000000..2f7baa8 --- /dev/null +++ b/docker-images/bindings/dependency-mapping/32e47c2139d6379836910f0d1cc253a019ac282f3aeea12237069f00046279ad @@ -0,0 +1 @@ +file:///platform/bindings/dependency-mapping/spring-cloud-bindings-2.0.4.jar \ No newline at end of file diff --git a/docker-images/bindings/dependency-mapping/README.md b/docker-images/bindings/dependency-mapping/README.md new file mode 100644 index 0000000..0524270 --- /dev/null +++ b/docker-images/bindings/dependency-mapping/README.md @@ -0,0 +1,10 @@ +# dependency-mapping binding + +Used only with `-Pno-maven-central`. The file named +`32e47c2139d6379836910f0d1cc253a019ac282f3aeea12237069f00046279ad` - the sha256 the Spring Boot +buildpack 5.36.7 expects for Spring Cloud Bindings 2.0.4 - contains a `file://` URI pointing at a +local copy of that jar, so the build never contacts repo1.maven.org. + +Run `../../scripts/fetch-buildpack-deps.sh` to download and verify the jar (it is git-ignored). +A newer buildpack version will expect a different jar and hash; read them from the builder's +`/cnb/buildpacks/paketo-buildpacks_spring-boot//buildpack.toml`. diff --git a/docker-images/bindings/dependency-mapping/type b/docker-images/bindings/dependency-mapping/type new file mode 100644 index 0000000..c722776 --- /dev/null +++ b/docker-images/bindings/dependency-mapping/type @@ -0,0 +1 @@ +dependency-mapping \ No newline at end of file diff --git a/docker-images/docker/Dockerfile.aot-cache b/docker-images/docker/Dockerfile.aot-cache new file mode 100644 index 0000000..2aa1eaf --- /dev/null +++ b/docker-images/docker/Dockerfile.aot-cache @@ -0,0 +1,15 @@ +# 7. Layered plus a JDK 25 AOT cache (JEP 483/514/515), built by a training run at image build time. +# Faster start - and one more large layer that changes on every code change (docs/06-aot-cache.md). +FROM eclipse-temurin:25-jre AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted + +FROM eclipse-temurin:25-jre +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +RUN java -XX:AOTCacheOutput=app.aot -Dspring.context.exit=onRefresh -jar application.jar +ENTRYPOINT ["java", "-XX:AOTCache=app.aot", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.aot-cache-mismatch b/docker-images/docker/Dockerfile.aot-cache-mismatch new file mode 100644 index 0000000..0e7f259 --- /dev/null +++ b/docker-images/docker/Dockerfile.aot-cache-mismatch @@ -0,0 +1,9 @@ +# 7b. A trap: the AOT cache from image 7, run by a different JVM build (distroless ships Temurin +# 25.0.4.1, the cache was trained on 25.0.4). What happens when a base-image bump changes the JVM +# under a cache built earlier - measured in docs/06-aot-cache.md. +FROM sbd/docker-images:aot-cache AS trained + +FROM gcr.io/distroless/java25-debian13:nonroot +WORKDIR /application +COPY --from=trained /application/ ./ +ENTRYPOINT ["java", "-XX:AOTCache=app.aot", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.fatjar-jdk b/docker-images/docker/Dockerfile.fatjar-jdk new file mode 100644 index 0000000..f0ba153 --- /dev/null +++ b/docker-images/docker/Dockerfile.fatjar-jdk @@ -0,0 +1,4 @@ +# 1. The Dockerfile most tutorials start with: the whole JDK, the whole fat jar, one layer for both. +FROM eclipse-temurin:25-jdk +COPY target/app.jar /app/app.jar +ENTRYPOINT ["java", "-jar", "/app/app.jar"] diff --git a/docker-images/docker/Dockerfile.fatjar-jre b/docker-images/docker/Dockerfile.fatjar-jre new file mode 100644 index 0000000..1c470d1 --- /dev/null +++ b/docker-images/docker/Dockerfile.fatjar-jre @@ -0,0 +1,4 @@ +# 2. Same, on the JRE image. Smaller base, but every code change still re-ships the whole fat jar. +FROM eclipse-temurin:25-jre +COPY target/app.jar /app/app.jar +ENTRYPOINT ["java", "-jar", "/app/app.jar"] diff --git a/docker-images/docker/Dockerfile.jlink-distroless b/docker-images/docker/Dockerfile.jlink-distroless new file mode 100644 index 0000000..1f495db --- /dev/null +++ b/docker-images/docker/Dockerfile.jlink-distroless @@ -0,0 +1,26 @@ +# 6. A custom runtime: jdeps works out which JDK modules the application needs, jlink builds a +# JRE containing only those, and it goes onto distroless java-base (glibc, no JRE of its own). +# See docs/05-jlink.md for the module list this produced and the one it initially missed. +FROM eclipse-temurin:25-jdk AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted +# jdeps cannot see modules loaded reflectively. jdk.management is the one this application +# needs: without it Micrometer silently drops 15 metrics, GC pauses and CPU usage among them. +ARG EXTRA_MODULES="jdk.management" +RUN MODULES="$(jdeps --ignore-missing-deps -q --recursive --multi-release 25 --print-module-deps \ + --class-path 'extracted/dependencies/lib/*' extracted/application/application.jar)" \ + && MODULES="${MODULES}${EXTRA_MODULES:+,$EXTRA_MODULES}" \ + && echo "jlink modules: $MODULES" | tee /builder/modules.txt \ + && jlink --add-modules "$MODULES" --strip-debug --no-man-pages --no-header-files \ + --compress=zip-6 --output /builder/jre + +FROM gcr.io/distroless/java-base-debian13:nonroot +COPY --from=builder /builder/jre /opt/jre +COPY --from=builder /builder/modules.txt /opt/jre/modules.txt +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +ENTRYPOINT ["/opt/jre/bin/java", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.layered-alpine b/docker-images/docker/Dockerfile.layered-alpine new file mode 100644 index 0000000..fed5a03 --- /dev/null +++ b/docker-images/docker/Dockerfile.layered-alpine @@ -0,0 +1,13 @@ +# 4. Layered, on the Alpine (musl) JRE image. +FROM eclipse-temurin:25-jre-alpine AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted + +FROM eclipse-temurin:25-jre-alpine +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +ENTRYPOINT ["java", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.layered-distroless b/docker-images/docker/Dockerfile.layered-distroless new file mode 100644 index 0000000..8f40e8e --- /dev/null +++ b/docker-images/docker/Dockerfile.layered-distroless @@ -0,0 +1,15 @@ +# 5. Layered, on distroless: no shell, no package manager, runs as uid 65532 (the :nonroot tag). +# The extraction has to happen in a stage that HAS a shell - distroless cannot RUN anything. +FROM eclipse-temurin:25-jre AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted + +FROM gcr.io/distroless/java25-debian13:nonroot +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +# Exec form is mandatory here: there is no /bin/sh to run a shell-form command. +ENTRYPOINT ["java", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.layered-jre b/docker-images/docker/Dockerfile.layered-jre new file mode 100644 index 0000000..052cda6 --- /dev/null +++ b/docker-images/docker/Dockerfile.layered-jre @@ -0,0 +1,14 @@ +# 3. Spring Boot's layered extraction: dependencies, loader, snapshots and your code become four +# separate layers, ordered from least to most likely to change. +FROM eclipse-temurin:25-jre AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted + +FROM eclipse-temurin:25-jre +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +ENTRYPOINT ["java", "-jar", "application.jar"] diff --git a/docker-images/docker/Dockerfile.shell-form b/docker-images/docker/Dockerfile.shell-form new file mode 100644 index 0000000..ba81f7c --- /dev/null +++ b/docker-images/docker/Dockerfile.shell-form @@ -0,0 +1,5 @@ +# 8. A trap, not a recommendation: shell-form ENTRYPOINT. Whether SIGTERM reaches the JVM depends +# on what /bin/sh does with "-c" - measured, not assumed, in docs/07-pid1-and-signals.md. +FROM eclipse-temurin:25-jre +COPY target/app.jar /app/app.jar +ENTRYPOINT java -jar /app/app.jar diff --git a/docker-images/docker/Dockerfile.shell-form-alpine b/docker-images/docker/Dockerfile.shell-form-alpine new file mode 100644 index 0000000..20948c9 --- /dev/null +++ b/docker-images/docker/Dockerfile.shell-form-alpine @@ -0,0 +1,4 @@ +# 8b. The same shell-form ENTRYPOINT where /bin/sh is BusyBox ash instead of dash. +FROM eclipse-temurin:25-jre-alpine +COPY target/app.jar /app/app.jar +ENTRYPOINT java -jar /app/app.jar diff --git a/docker-images/docker/Dockerfile.shell-form-wrapper b/docker-images/docker/Dockerfile.shell-form-wrapper new file mode 100644 index 0000000..20ed6a5 --- /dev/null +++ b/docker-images/docker/Dockerfile.shell-form-wrapper @@ -0,0 +1,4 @@ +# 8c. What people actually write: a shell form with a second command in front of java. +FROM eclipse-temurin:25-jre +COPY target/app.jar /app/app.jar +ENTRYPOINT echo "starting revision $(date +%s)" && java -jar /app/app.jar diff --git a/docker-images/docs/01-the-variants.md b/docker-images/docs/01-the-variants.md new file mode 100644 index 0000000..1d113de --- /dev/null +++ b/docker-images/docs/01-the-variants.md @@ -0,0 +1,42 @@ +# 1. The variants and how each is built + +[Index](../README.md) · Next: [2. Measuring size →](02-measuring-size.md) + +All nine start from the same `target/app.jar` (25 MB). The Dockerfiles are in [`docker/`](../docker), +each with a comment saying what it is for. + +## The layered pattern + +```dockerfile +FROM eclipse-temurin:25-jre AS builder +WORKDIR /builder +COPY target/app.jar application.jar +RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted + +FROM gcr.io/distroless/java25-debian13:nonroot +WORKDIR /application +COPY --from=builder /builder/extracted/dependencies/ ./ +COPY --from=builder /builder/extracted/spring-boot-loader/ ./ +COPY --from=builder /builder/extracted/snapshot-dependencies/ ./ +COPY --from=builder /builder/extracted/application/ ./ +ENTRYPOINT ["java", "-jar", "application.jar"] +``` + +`extract` without `--launcher` writes a thin `application.jar` whose manifest `Class-Path` lists +`lib/*.jar`, so the runtime stage launches with plain `java -jar` - no `JarLauncher`, no nested-jar +class loading. `list-layers` on this jar prints `dependencies`, `spring-boot-loader`, +`snapshot-dependencies`, `application`, in that order: least likely to change first. + +The builder stage uses the Temurin JRE, not the JDK: `extract` needs only a JVM. Only the jlink +variant needs the JDK, for `jdeps` and `jlink`. + +## Buildpacks and Jib in a restricted network + +The committed output was produced in a sandbox with no route to Docker Hub or Maven Central. The +flags that made that work are in [`run-all.sh`](../scripts/run-all.sh): + +- Buildpacks: `-Pcorporate-proxy,no-maven-central -Dspring-boot.build-image.pullPolicy=IF_NOT_PRESENT` + ([chapter 4](04-buildpacks.md)) +- Jib: `-Djib.from.image=docker://eclipse-temurin:25-jre` - take the base image from the local daemon + +With ordinary internet access, `mvn spring-boot:build-image` and `mvn jib:dockerBuild` need no flags. diff --git a/docker-images/docs/02-measuring-size.md b/docker-images/docs/02-measuring-size.md new file mode 100644 index 0000000..0c3cc71 --- /dev/null +++ b/docker-images/docs/02-measuring-size.md @@ -0,0 +1,58 @@ +# 2. Measuring size + +[← 1. The variants](01-the-variants.md) · [Index](../README.md) · Next: [3. Layered jars →](03-layered-jars.md) + +"Image size" means at least three different numbers: + +| Number | What it is | How it is measured here | +|---|---|---| +| on disk | uncompressed layers, what a node stores | `docker image inspect -f '{{.Size}}'` on the classic `overlay2` store | +| pushed | compressed layer blobs + config, what a registry stores and a cold node downloads | sum of `layers[].size` + `config.size` from `crane manifest` | +| rebuild delta | the blobs a push actually uploads after a change | [chapter 3](03-layered-jars.md) | + +[`measure.sh`](../scripts/measure.sh) pushes every variant to a throwaway registry +(`crane registry serve --address localhost:5000` - in-memory, no Docker Hub account needed) and +reads the manifests back. + +## `docker images` disagrees with itself + +Docker Engine 29 can use two image stores. The same eleven tarballs, loaded into each +([`image-store-size-difference.txt`](output/image-store-size-difference.txt)): + +| Image | containerd store | classic overlay2 | +|---|---|---| +| `eclipse-temurin:25-jre` | 496 MB | 352 MB | +| `gcr.io/distroless/java25-debian13:nonroot` | 305 MB | 226 MB | +| `paketobuildpacks/builder-noble-java-tiny` | 1.26 GB | 868 MB | + +The containerd store's figure is larger across the board. Size comparisons from blog posts - or +between two colleagues' laptops - are only comparable if they used the same store, and the +containerd store is the default for new Docker installations. This project measures on the classic +store because its number is the unpacked size alone. + +## The result + +[`image-matrix.txt`](output/image-matrix.txt): + +``` +variant on disk pushed layers user shell packages ready ms Started RSS JVM +fatjar-jdk 456M 178M 7 root yes 115 (dpkg) 5362 3.932s 169.5MiB Eclipse 25.0.4 +fatjar-jre 377M 144M 7 root yes 106 (dpkg) 5091 3.944s 182.1MiB Eclipse 25.0.4 +layered-jre 377M 144M 11 root yes 106 (dpkg) 4272 3.356s 169.5MiB Eclipse 25.0.4 +layered-alpine 251M 98M 10 root yes 44 (apk) 4634 3.775s 145.9MiB Eclipse 25.0.4 +layered-distroless 252M 97M 39 65532 no 25 (status.d) 4118 3.254s 164.3MiB Eclipse 25.0.4.1 +jlink-distroless 119M 76M 39 65532 no 24 (status.d) 4386 3.39s 165.7MiB Eclipse 25.0.4 +aot-cache 440M 160M 12 root yes 106 (dpkg) 2412 1.638s 174.6MiB Eclipse 25.0.4 +buildpacks 345M 125M 20 1002:1001 no 10 (status.d) 4726 3.615s 183.6MiB BellSoft 25.0.4 +jib 377M 144M 10 root yes 106 (dpkg) 4323 3.457s 162.5MiB Eclipse 25.0.4 +``` + +- *ready ms* is `docker run` to the first 200 from `/actuator/health/readiness`; *Started* is + Spring Boot's own figure. Median of three on a 2-vCPU VM - treat differences under ~0.3 s as noise. +- *packages* counts OS packages from the image filesystem without running anything in it (dpkg + `status`, distroless/Paketo `status.d/`, or `apk`), which works for images with no shell. +- Every Temurin-based variant runs as **root**: the official images set no `USER`. + +A note on method: the first run of this table had distroless starting two seconds slower than +everything else. A k3s image import was running on the same two CPUs at the time. Nothing else runs +during `measure.sh` now. diff --git a/docker-images/docs/03-layered-jars.md b/docker-images/docs/03-layered-jars.md new file mode 100644 index 0000000..20874d0 --- /dev/null +++ b/docker-images/docs/03-layered-jars.md @@ -0,0 +1,33 @@ +# 3. Layered jars: the win is the second push + +[← 2. Measuring size](02-measuring-size.md) · [Index](../README.md) · Next: [4. Buildpacks →](04-buildpacks.md) + +`fatjar-jre` and `layered-jre` are the same size: 377 MB on disk, 144 MB pushed. The layers are the +same bytes, cut differently. + +What differs is the next build. [`measure-rebuild.sh`](../scripts/measure-rebuild.sh) changes one +string constant, rebuilds every variant, and counts the layer digests that did not exist before +([`rebuild-delta.txt`](output/rebuild-delta.txt)): + +``` +variant new layers bytes to push image total share +fatjar-jdk 1 of 7 23422K 178M 13.1% +fatjar-jre 1 of 7 23422K 144M 16.2% +layered-jre 1 of 11 6K 144M 0.0% +layered-alpine 1 of 10 6K 98M 0.0% +layered-distroless 1 of 39 6K 97M 0.0% +jlink-distroless 1 of 39 6K 76M 0.0% +aot-cache 2 of 12 15323K 160M 9.6% +buildpacks 2 of 20 64K 125M 0.1% +jib 1 of 10 2K 144M 0.0% +``` + +A fat jar re-ships all 23 MB of dependencies on every commit; a layered image ships 6 KB. Multiply +by deployments per day and nodes per cluster - every node that already has revision 1 downloads +only the changed layers. + +Layers only help if their order matches change frequency. Put your own `COPY` of configuration +files *after* the dependency layers, never before, or every change re-ships everything below it. +Snapshot dependencies get their own layer because they change without a version bump. + +Jib and buildpacks layer the same way without being asked. diff --git a/docker-images/docs/04-buildpacks.md b/docker-images/docs/04-buildpacks.md new file mode 100644 index 0000000..9a6f376 --- /dev/null +++ b/docker-images/docs/04-buildpacks.md @@ -0,0 +1,80 @@ +# 4. Buildpacks: what the builder decides for you + +[← 3. Layered jars](03-layered-jars.md) · [Index](../README.md) · Next: [5. jlink →](05-jlink.md) + +`mvn spring-boot:build-image` with no configuration used `paketobuildpacks/builder-noble-java-tiny` +- the Spring Boot 4.1.1 default - and six of its 26 buildpacks took part: `ca-certificates`, +`bellsoft-liberica`, `syft`, `executable-jar`, `dist-zip`, `spring-boot`. + +What you get without writing a line of Dockerfile, from the build log and +[`image-matrix.txt`](output/image-matrix.txt): + +- **Java version from the jar.** `$BP_JVM_VERSION` shows `21` as its default, then + `Using Java version 25 extracted from MANIFEST.MF` - the buildpack reads `Build-Jdk-Spec`. +- **A non-root user**, `1002:1001`, and no shell (the *tiny* run image). +- **A reproducible image.** Creation date is fixed (Docker shows "46 years ago" - 1980), so the same + input gives the same digest. +- **A memory calculator** that runs before the JVM and sets `-Xmx`, metaspace, code cache and + thread stacks from the container's memory limit - see below, it can refuse to start the app. +- **An SBOM** layer from Syft. +- **Spring Cloud Bindings** on the classpath, which reads Kubernetes service bindings into Spring + properties. `BP_SPRING_CLOUD_BINDINGS_DISABLED=true` removes it. + +## The size + +345 MB on disk, and most of it is one layer: + +``` +276MB Layer: 'jre', Created by buildpack: paketo-buildpacks/bellsoft-liberica@11.8.3 +``` + +Temurin's 25 JRE is 200 MB. BellSoft's JRE carries a second VM: `lib/client` is 74 MB (its own +`libjvm.so` plus two CDS archives) next to the 85 MB `lib/server`. `BP_JVM_JLINK_ENABLED=true` asks +the buildpack to jlink a smaller runtime - with the caveat in [chapter 5](05-jlink.md). + +## At 512 MiB the image does not start + +[`buildpacks-memory.txt`](output/buildpacks-memory.txt): + +``` +## docker run -m 512m +unable to calculate memory configuration +fixed memory regions require 595872K which is greater than 512M available for allocation: -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=83872K, -XX:ReservedCodeCacheSize=240M, -Xss1M * 250 threads +ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1 +state: exited (exit 82) +``` + +The calculator reserves 1 MiB of stack for each of **250 threads** plus a 240 MiB code cache and +metaspace sized from the class count, *before* any heap. 512 MiB - a very common Kubernetes +`limits.memory` - is not enough, and the container exits with code 82 before Java runs. The same jar +in any of the Dockerfile-built images starts fine at 512 MiB. + +Either give it more (`-m 768m` → `-Xmx190559K`, a quarter of the limit) or tell it the truth about +threads: `BPL_JVM_THREAD_COUNT=50` → starts at 512 MiB with `-Xmx133215K`. With virtual threads, +250 platform threads is a generous assumption. + +## Behind a TLS-intercepting proxy + +The Java buildpack downloads the JRE at build time, and the Spring Boot buildpack downloads Spring +Cloud Bindings from Maven Central. Inside a corporate proxy both fail. Two profiles in the +[`pom.xml`](../pom.xml) handle it: + +- `corporate-proxy` - build container on the host network, `HTTPS_PROXY` passed through, and a + `ca-certificates` [binding](../bindings/ca-certificates) so the build trusts the proxy's CA + (`Added 2 additional CA certificate(s) to system truststore` in the log). With + `BP_EMBED_CERTS=false` - the default - the CA is not baked into the runtime image. +- `no-maven-central` - a `dependency-mapping` [binding](../bindings/dependency-mapping) that points + the buildpack at a local copy of the jar, keyed by the sha256 in the buildpack's `buildpack.toml`. + +Both bindings are merged with `combine.children="append"`, so the profiles compose: +`-Pcorporate-proxy,no-maven-central`. + +## Two traps met on the way + +- **The run image is pinned by tag inside the builder.** The builder pulled as `:latest` asked for + `paketobuildpacks/ubuntu-noble-run-tiny:0.0.130`; with `pullPolicy=IF_NOT_PRESENT` and only + `:latest` present locally, the build still tried Docker Hub. Tagging the same digest as `0.0.130` + fixed it. +- **`-Dspring-boot.build-image.imageName` is ignored if the pom sets ``** - explicit + configuration beats the user property, so the "second" build silently overwrote the first image. + The pom uses a `${buildpacks.image}` property instead. diff --git a/docker-images/docs/05-jlink.md b/docker-images/docs/05-jlink.md new file mode 100644 index 0000000..e83cd7a --- /dev/null +++ b/docker-images/docs/05-jlink.md @@ -0,0 +1,45 @@ +# 5. jlink: 133 MB smaller, 15 metrics quieter + +[← 4. Buildpacks](04-buildpacks.md) · [Index](../README.md) · Next: [6. AOT cache →](06-aot-cache.md) + +[`Dockerfile.jlink-distroless`](../docker/Dockerfile.jlink-distroless) asks `jdeps` which JDK modules +the application needs and builds a runtime with only those: + +```bash +jdeps --ignore-missing-deps -q --recursive --multi-release 25 --print-module-deps \ + --class-path 'extracted/dependencies/lib/*' extracted/application/application.jar +``` + +It chose 18 modules ([`jlink-metrics.txt`](output/jlink-metrics.txt)) - including `java.desktop`, +because Spring uses `java.beans`. On `distroless/java-base` the image is **119 MB** on disk against +252 MB for distroless with the full JRE. + +## It runs. It is also missing something. + +The jdeps-only image starts, serves requests and passes its health check. Its startup log has two +warnings: + +``` +i.m.c.i.binder.jvm.JvmGcMetrics : GC notifications will not be available because com.sun.management.GarbageCollectionNotificationInfo is not present +i.m.c.i.binder.jvm.JvmGcMetrics : GC notifications will not be available because no GarbageCollectorMXBean of the JVM provides any. GCs=[G1 Young Generation, G1 Concurrent GC, G1 Old Generation] +``` + +and `/actuator/prometheus` exports 46 metric names instead of 61. Gone: `jvm_gc_pause_seconds_*`, +`jvm_gc_memory_allocated_bytes_total`, `jvm_gc_live_data_size_bytes`, `process_cpu_usage`, +`system_cpu_usage`, `process_files_open_files` and more - 15 in total. + +`com.sun.management.*` lives in the `jdk.management` module, and Micrometer touches it +reflectively, which `jdeps` cannot see. Nothing fails; the GC and CPU panels of your dashboard just +go flat after the image switch. The Dockerfile now adds it by default: + +```dockerfile +ARG EXTRA_MODULES="jdk.management" +``` + +With it, the only difference left is `jvm_gc_concurrent_phase_time_*`, which Micrometer registers +lazily inside its GC notification listener - it appears after the first G1 concurrent cycle, which +may or may not have happened a few seconds after startup, in either image. + +The general rule: diff the metric names, not just the health check, before shipping a jlink image. +Other modules commonly needed only reflectively are `jdk.crypto.cryptoki` (PKCS#11), +`jdk.localedata` (non-English locale data) and `jdk.naming.dns` (DNS lookups through JNDI). diff --git a/docker-images/docs/06-aot-cache.md b/docker-images/docs/06-aot-cache.md new file mode 100644 index 0000000..00581c4 --- /dev/null +++ b/docker-images/docs/06-aot-cache.md @@ -0,0 +1,52 @@ +# 6. The JDK 25 AOT cache + +[← 5. jlink](05-jlink.md) · [Index](../README.md) · Next: [7. PID 1 and signals →](07-pid1-and-signals.md) + +JDK 24 added the AOT cache (JEP 483: classes loaded and linked ahead of time), and JDK 25 made it +one step (JEP 514, `-XX:AOTCacheOutput`) and added method profiles to it (JEP 515). Spring Boot's +documented Dockerfile does a training run at image build time: + +```dockerfile +RUN java -XX:AOTCacheOutput=app.aot -Dspring.context.exit=onRefresh -jar application.jar +ENTRYPOINT ["java", "-XX:AOTCache=app.aot", "-jar", "application.jar"] +``` + +`spring.context.exit=onRefresh` starts the context and exits, so the training run needs no database +or network - unless your beans touch them during startup. + +## What it buys and what it costs + +| | `layered-jre` | `aot-cache` | +|---|---|---| +| Started in (median of 3) | 3.356 s | **1.638 s** | +| on disk | 377 MB | 440 MB (+63 MB) | +| pushed after a one-line change | 6 KB | **15.3 MB** | + +The last row is the one nobody mentions. The training run happens *after* the application layer +is copied, so every code change invalidates it and the cache layer is rebuilt and re-pushed. Half +the startup time costs back most of what layering saved on pushes. Whether that trade is worth it +depends on how often pods start versus how often you deploy - scale-to-zero and aggressive +autoscaling say yes; a fleet that deploys twenty times a day and rarely restarts says no. + +## A cache from a different JVM + +The cache is only valid for the exact JVM build that wrote it. +[`Dockerfile.aot-cache-mismatch`](../docker/Dockerfile.aot-cache-mismatch) copies the trained +application onto the distroless image, whose JVM is Temurin 25.0.4.1 instead of 25.0.4 +([`aot-cache-mismatch.txt`](output/aot-cache-mismatch.txt)): + +``` +[0.007s][warning][aot] The AOT cache was created by a different version or build of HotSpot +[0.007s][error ][aot] An error has occurred while processing the AOT cache. Run with -Xlog:aot for details. +[0.008s][error ][aot] Loading static archive failed. +[0.008s][error ][aot] Unable to map shared spaces +Starting ImagesApplication v1.0.0 using Java 25.0.4.1 with PID 1 (/application/application.jar started by nonroot in /ap +Started ImagesApplication in 3.564 seconds (process running for 4.05) +``` + +Four lines at `error` level, and then the application starts normally without the cache - back at +3.5 s. Nothing fails a health check. The way this happens in practice is a floating base-image tag +(`25-jre`) moving to a new JVM build between the stage that trained the cache and a later rebuild +that reused a cached training layer - or a multi-stage build that trains on one image and runs on +another, as here. Train and run on the same image, pin it by digest, and alert on `[aot]` lines at +`error`. diff --git a/docker-images/docs/07-pid1-and-signals.md b/docker-images/docs/07-pid1-and-signals.md new file mode 100644 index 0000000..dbaa2c6 --- /dev/null +++ b/docker-images/docs/07-pid1-and-signals.md @@ -0,0 +1,29 @@ +# 7. PID 1 and signals + +[← 6. AOT cache](06-aot-cache.md) · [Index](../README.md) · Next: [8. Living with distroless →](08-distroless-in-practice.md) + +`docker stop` - and Kubernetes, on pod deletion - sends SIGTERM to PID 1, waits (10 s for Docker, +`terminationGracePeriodSeconds` for Kubernetes), then SIGKILLs. Spring Boot's graceful shutdown only +runs if the JVM receives that SIGTERM. + +[`pid1-and-signals.txt`](output/pid1-and-signals.txt): + +``` +layered-jre PID 1: java -jar application.jar docker stop: 244 ms exit code: 143 graceful-shutdown log lines: 1 +shell-form PID 1: /bin/sh -c java -jar /app/app.jar docker stop: 10160 ms exit code: 137 graceful-shutdown log lines: 0 +shell-form-alpine PID 1: java -jar /app/app.jar docker stop: 219 ms exit code: 143 graceful-shutdown log lines: 1 +shell-form-wrapper PID 1: /bin/sh -c echo "starting revision $(date +%s)" && java -j docker stop: 10176 ms exit code: 137 graceful-shutdown log lines: 0 +``` + +- **Exec form** (`["java", "-jar", ...]`): the JVM is PID 1, SIGTERM arrives, graceful shutdown + runs, exit code 143, a quarter of a second. +- **Shell form on `eclipse-temurin:25-jre`**: `/bin/sh` is dash (`/usr/bin/dash`). dash stays + resident as PID 1 and does not forward SIGTERM. Docker waits the full 10 s and SIGKILLs: exit 137, + no graceful shutdown, in-flight requests cut off. +- **The same shell form on the Alpine image works**, because `/bin/sh` there is BusyBox ash, which + replaces itself with the last command of a `-c` string. The same Dockerfile line behaves + differently depending on the base image - which is how "it works on my image" arguments start. +- **Any shell form with more than one command** keeps the shell as PID 1 regardless. + +If you need a wrapper script, end it with `exec java ...`. Distroless removes the question: with no +`/bin/sh`, a shell-form `ENTRYPOINT` cannot even start. diff --git a/docker-images/docs/08-distroless-in-practice.md b/docker-images/docs/08-distroless-in-practice.md new file mode 100644 index 0000000..71bb583 --- /dev/null +++ b/docker-images/docs/08-distroless-in-practice.md @@ -0,0 +1,20 @@ +# 8. Living with distroless + +[← 7. PID 1 and signals](07-pid1-and-signals.md) · [Index](../README.md) + +`gcr.io/distroless/java25-debian13:nonroot` gives you: a JRE, glibc, CA certificates, time zone +data, and 25 OS packages in total - against 106 in `eclipse-temurin:25-jre`. No shell, no package +manager, uid 65532. It is the variant recommended in the article, and it changes a few habits. + +- **`RUN` is impossible in the final stage.** Do everything - extraction, jlink, training runs - in + a builder stage with a shell, then `COPY`. +- **`ENTRYPOINT` must be exec form.** +- **`docker exec -it ... sh` does not work.** On Kubernetes use an ephemeral debug container that + shares the process namespace: `kubectl debug -it --image=busybox:1.37 --target=app`. With + Docker: `docker run -it --pid=container: --network=container: busybox`. +- **A `preStop` hook of `exec: ["sh", "-c", "sleep 10"]` fails**, because there is no `sh`. Use the + native `sleep` action (on by default since Kubernetes 1.30, stable in 1.34). The + [Kubernetes article](https://ankurm.com/spring-boot-4-kubernetes-probes-graceful-shutdown-cpu-limits-hpa/) + measures what that failure costs during a rolling update. +- **Tags are the variant.** `:latest` runs as root, `:nonroot` as 65532, `:debug` adds a BusyBox + shell - useful for a one-off investigation, never for production. diff --git a/docker-images/docs/output/aot-cache-mismatch.txt b/docker-images/docs/output/aot-cache-mismatch.txt new file mode 100644 index 0000000..0c32e1d --- /dev/null +++ b/docker-images/docs/output/aot-cache-mismatch.txt @@ -0,0 +1,14 @@ +## sbd/docker-images:aot-cache +Starting ImagesApplication v1.0.0 using Java 25.0.4 with PID 1 (/application/application.jar started by root in /applica +Started ImagesApplication in 1.657 seconds (process running for 1.966) +exit status while running: running + +## sbd/docker-images:aot-cache-mismatch +[0.007s][warning][aot] The AOT cache was created by a different version or build of HotSpot +[0.007s][error ][aot] An error has occurred while processing the AOT cache. Run with -Xlog:aot for details. +[0.008s][error ][aot] Loading static archive failed. +[0.008s][error ][aot] Unable to map shared spaces +Starting ImagesApplication v1.0.0 using Java 25.0.4.1 with PID 1 (/application/application.jar started by nonroot in /ap +Started ImagesApplication in 3.564 seconds (process running for 4.05) +exit status while running: running + diff --git a/docker-images/docs/output/buildpacks-layers.txt b/docker-images/docs/output/buildpacks-layers.txt new file mode 100644 index 0000000..8b20a4d --- /dev/null +++ b/docker-images/docs/output/buildpacks-layers.txt @@ -0,0 +1,21 @@ +# docker history sbd/docker-images:buildpacks +69B Buildpacks Process Types +1.94kB Buildpacks Launcher Config +2.93MB Buildpacks Application Launcher +0B Application Slice: 5 +14.2kB Application Slice: 4 +0B Application Slice: 3 +402kB Application Slice: 2 +25.9MB Application Slice: 1 +577kB Software Bill-of-Materials +3B Layer: 'web-application-type', Created by buildpack: paketo-buildpacks/spring-boot@5.36.7 +77.4kB Layer: 'spring-cloud-bindings', Created by buildpack: paketo-buildpacks/spring-boot@5.36.7 +4.01MB Layer: 'helper', Created by buildpack: paketo-buildpacks/spring-boot@5.36.7 +11B Layer: 'classpath', Created by buildpack: paketo-buildpacks/executable-jar@6.15.7 +276MB Layer: 'jre', Created by buildpack: paketo-buildpacks/bellsoft-liberica@11.8.3 +214B Layer: 'java-security-properties', Created by buildpack: paketo-buildpacks/bellsoft-liberica@11.8.3 +5.48MB Layer: 'helper', Created by buildpack: paketo-buildpacks/bellsoft-liberica@11.8.3 +5.18MB Layer: 'helper', Created by buildpack: paketo-buildpacks/ca-certificates@3.12.7 +519B +191B +24.9MB diff --git a/docker-images/docs/output/buildpacks-memory.txt b/docker-images/docs/output/buildpacks-memory.txt new file mode 100644 index 0000000..40052e5 --- /dev/null +++ b/docker-images/docs/output/buildpacks-memory.txt @@ -0,0 +1,16 @@ +## docker run -m 512m +unable to calculate memory configuration +fixed memory regions require 595872K which is greater than 512M available for allocation: -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=83872K, -XX:ReservedCodeCacheSize=240M, -Xss1M * 250 threads +ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1 +state: exited (exit 82) + +## docker run -m 768m +Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx190559K -XX:MaxMetaspaceSize=83872K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 768M, Thread Count: 250, Loaded Class Count: 12394, Headroom: 0%) +Started ImagesApplication in 3.884 seconds (process running for 4.423) +state: running (exit 0) + +## docker run -m 512m -e BPL_JVM_THREAD_COUNT=50 +Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx133215K -XX:MaxMetaspaceSize=83872K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 512M, Thread Count: 50, Loaded Class Count: 12394, Headroom: 0%) +Started ImagesApplication in 3.846 seconds (process running for 4.353) +state: running (exit 0) + diff --git a/docker-images/docs/output/image-matrix.txt b/docker-images/docs/output/image-matrix.txt new file mode 100644 index 0000000..b14f1c0 --- /dev/null +++ b/docker-images/docs/output/image-matrix.txt @@ -0,0 +1,14 @@ +# Spring Boot 4.1.1 application (webmvc + actuator + validation + prometheus), fat jar 25M +# Docker Engine 29.4.3, classic overlay2 store. 3 startup runs each, median shown. +# 'on disk' = docker image inspect .Size (uncompressed). 'pushed' = layer blobs + config in the registry (compressed). + +variant on disk pushed layers user shell packages ready ms Started RSS JVM +fatjar-jdk 456M 178M 7 root yes 115 (dpkg) 5362 3.932s 169.5MiB Eclipse 25.0.4 +fatjar-jre 377M 144M 7 root yes 106 (dpkg) 5091 3.944s 182.1MiB Eclipse 25.0.4 +layered-jre 377M 144M 11 root yes 106 (dpkg) 4272 3.356s 169.5MiB Eclipse 25.0.4 +layered-alpine 251M 98M 10 root yes 44 (apk) 4634 3.775s 145.9MiB Eclipse 25.0.4 +layered-distroless 252M 97M 39 65532 no 25 (status.d) 4118 3.254s 164.3MiB Eclipse 25.0.4.1 +jlink-distroless 119M 76M 39 65532 no 24 (status.d) 4386 3.39s 165.7MiB Eclipse 25.0.4 +aot-cache 440M 160M 12 root yes 106 (dpkg) 2412 1.638s 174.6MiB Eclipse 25.0.4 +buildpacks 345M 125M 20 1002:1001 no 10 (status.d) 4726 3.615s 183.6MiB BellSoft 25.0.4 +jib 377M 144M 10 root yes 106 (dpkg) 4323 3.457s 162.5MiB Eclipse 25.0.4 diff --git a/docker-images/docs/output/image-store-size-difference.txt b/docker-images/docs/output/image-store-size-difference.txt new file mode 100644 index 0000000..3d8db10 --- /dev/null +++ b/docker-images/docs/output/image-store-size-difference.txt @@ -0,0 +1,31 @@ +# The same eleven images, loaded from the same tarballs, listed by `docker images` under the two +# Docker Engine 29.4.3 image stores. Captured once by hand (switching stores needs a daemon +# restart); the commands are in docs/02-measuring-size.md. + +## containerd image store (daemon.json: {"features":{"containerd-snapshotter":true}} - the default for new installs) +REPOSITORY:TAG IMAGE ID SIZE +paketobuildpacks/ubuntu-noble-run-tiny:latest 5fb8a81f3f56 38.7MB +eclipse-temurin:25-jre 211325bca3fd 496MB +eclipse-temurin:25-jdk 00eacca34899 608MB +eclipse-temurin:25-jre-alpine 31a7faf76302 305MB +bellsoft/liberica-openjre-debian:25-cds 78b8d1ba9172 505MB +quay.io/prometheus/prometheus:v3.14.0 0230c2ba4c1c 372MB +busybox:1.37 e0b4f7eca906 6.77MB +registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 b8e2bc8f8de9 131MB +paketobuildpacks/builder-noble-java-tiny:latest b54d7bce7d3c 1.26GB +gcr.io/distroless/java-base-debian13:nonroot fe682f609c79 43.1MB +gcr.io/distroless/java25-debian13:nonroot 3ad8b3cc79ab 305MB + +## classic overlay2 store (daemon.json: {"features":{"containerd-snapshotter":false}}) +REPOSITORY:TAG IMAGE ID SIZE +paketobuildpacks/ubuntu-noble-run-tiny:latest 3f2697c8829c 24.9MB +eclipse-temurin:25-jre a33388e97452 352MB +eclipse-temurin:25-jdk ca0436742d32 430MB +eclipse-temurin:25-jre-alpine 0988c057a1e4 226MB +bellsoft/liberica-openjre-debian:25-cds aea7393405da 356MB +quay.io/prometheus/prometheus:v3.14.0 31c1e0aacb3a 261MB +busybox:1.37 db287cb6be81 4.42MB +registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 3bba96c2e4e6 89.3MB +paketobuildpacks/builder-noble-java-tiny:latest d835ec7c8f6e 868MB +gcr.io/distroless/java-base-debian13:nonroot 9bc10980a41f 26.5MB +gcr.io/distroless/java25-debian13:nonroot be8c5fbd580c 226MB diff --git a/docker-images/docs/output/jlink-metrics.txt b/docker-images/docs/output/jlink-metrics.txt new file mode 100644 index 0000000..5a4f2c9 --- /dev/null +++ b/docker-images/docs/output/jlink-metrics.txt @@ -0,0 +1,30 @@ +# jlink modules chosen by jdeps alone: +java.base java.compiler java.datatransfer java.desktop java.instrument java.logging java.management java.naming java.net.http java.prefs java.scripting java.security.jgss java.security.sasl java.sql java.transaction.xa java.xml jdk.jfr jdk.unsupported + +# WARN lines at startup of the jdeps-only image: + i.m.c.i.binder.jvm.JvmGcMetrics : GC notifications will not be available because com.sun.management.GarbageCollectionNotificationInfo is not present + i.m.c.i.binder.jvm.JvmGcMetrics : GC notifications will not be available because no GarbageCollectorMXBean of the JVM provides any. GCs=[G1 Young Generation, G1 Concurrent GC, G1 Old Generation] + +# Metric names: full JRE 61, jdeps-only 46, jdeps + jdk.management 58 +# Present with the full JRE, missing from the jdeps-only image: + jvm_gc_concurrent_phase_time_seconds_count + jvm_gc_concurrent_phase_time_seconds_max + jvm_gc_concurrent_phase_time_seconds_sum + jvm_gc_live_data_size_bytes + jvm_gc_max_data_size_bytes + jvm_gc_memory_allocated_bytes_total + jvm_gc_memory_promoted_bytes_total + jvm_gc_pause_seconds_count + jvm_gc_pause_seconds_max + jvm_gc_pause_seconds_sum + process_cpu_time_ns_total + process_cpu_usage + process_files_max_files + process_files_open_files + system_cpu_usage +# Missing from jdeps + jdk.management: 3 + jvm_gc_concurrent_phase_time_seconds_count + jvm_gc_concurrent_phase_time_seconds_max + jvm_gc_concurrent_phase_time_seconds_sum +# (jvm_gc_concurrent_phase_time_* is registered lazily, after the first G1 concurrent cycle - +# whether it appears within a few seconds of startup varies run to run, in either image.) diff --git a/docker-images/docs/output/pid1-and-signals.txt b/docker-images/docs/output/pid1-and-signals.txt new file mode 100644 index 0000000..6a824ac --- /dev/null +++ b/docker-images/docs/output/pid1-and-signals.txt @@ -0,0 +1,12 @@ +# docker stop sends SIGTERM to PID 1, waits 10 s, then SIGKILL. Exit code 143 = the JVM handled +# SIGTERM (128+15). 137 = it was killed (128+9). + +layered-jre PID 1: java -jar application.jar docker stop: 244 ms exit code: 143 graceful-shutdown log lines: 1 +shell-form PID 1: /bin/sh -c java -jar /app/app.jar docker stop: 10160 ms exit code: 137 graceful-shutdown log lines: 0 +shell-form-alpine PID 1: java -jar /app/app.jar docker stop: 219 ms exit code: 143 graceful-shutdown log lines: 1 +shell-form-wrapper PID 1: /bin/sh -c echo "starting revision $(date +%s)" && java -j docker stop: 10176 ms exit code: 137 graceful-shutdown log lines: 0 +layered-distroless PID 1: (image has no cat; ENTRYPOINT is exec form) docker stop: 256 ms exit code: 143 graceful-shutdown log lines: 1 + +# What /bin/sh is in each base image: +eclipse-temurin:25-jre -> /usr/bin/dash +eclipse-temurin:25-jre-alpine -> /bin/busybox diff --git a/docker-images/docs/output/rebuild-delta.txt b/docker-images/docs/output/rebuild-delta.txt new file mode 100644 index 0000000..030803d --- /dev/null +++ b/docker-images/docs/output/rebuild-delta.txt @@ -0,0 +1,14 @@ +# One-line code change (BuildInfo.REVISION 1 -> 2), every variant rebuilt. +# 'new layers' = layer digests in revision 2 that revision 1 did not have: what a push uploads +# and what a node that already runs revision 1 downloads. + +variant new layers bytes to push image total share +fatjar-jdk 1 of 7 23422K 178M 13.1% +fatjar-jre 1 of 7 23422K 144M 16.2% +layered-jre 1 of 11 6K 144M 0.0% +layered-alpine 1 of 10 6K 98M 0.0% +layered-distroless 1 of 39 6K 97M 0.0% +jlink-distroless 1 of 39 6K 76M 0.0% +aot-cache 2 of 12 15323K 160M 9.6% +buildpacks 2 of 20 64K 125M 0.1% +jib 1 of 10 2K 144M 0.0% diff --git a/docker-images/pom.xml b/docker-images/pom.xml new file mode 100644 index 0000000..96c721b --- /dev/null +++ b/docker-images/pom.xml @@ -0,0 +1,134 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + docker-images + 1.0.0 + docker-images + Dockerizing Spring Boot 4: layered jars, buildpacks, distroless, jlink, AOT cache - measured + + + 25 + 3.5.2 + + sbd/docker-images:buildpacks + + + + + org.springframework.boot + spring-boot-starter-webmvc + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter-validation + + + io.micrometer + micrometer-registry-prometheus + runtime + + + org.springframework.boot + spring-boot-starter-test + test + + + + + app + + + org.springframework.boot + spring-boot-maven-plugin + + + ${buildpacks.image} + + + + + + com.google.cloud.tools + jib-maven-plugin + ${jib.version} + + + eclipse-temurin:25-jre + + + sbd/docker-images:jib + + + + + + + + + + corporate-proxy + + + + org.springframework.boot + spring-boot-maven-plugin + + + host + + ${env.HTTPS_PROXY} + ${env.HTTPS_PROXY} + + + ${project.basedir}/bindings/ca-certificates:/platform/bindings/ca-certificates + + + + + + + + + + no-maven-central + + + + org.springframework.boot + spring-boot-maven-plugin + + + + ${project.basedir}/bindings/dependency-mapping:/platform/bindings/dependency-mapping + + + + + + + + + diff --git a/docker-images/scripts/build-images.sh b/docker-images/scripts/build-images.sh new file mode 100644 index 0000000..432cdf4 --- /dev/null +++ b/docker-images/scripts/build-images.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Build every image variant from target/app.jar. Tag suffix is $1 (default: none). +# ./scripts/build-images.sh -> sbd/docker-images: +# ./scripts/build-images.sh -r2 -> sbd/docker-images:-r2 +# Buildpacks and Jib need extra flags in a network-restricted environment; set +# BUILDPACKS_ARGS / JIB_ARGS (scripts/run-all.sh shows the sandbox values). +set -euo pipefail +source "$(dirname "$0")/env.sh" +SUFFIX="${1:-}" +cd "$MODULE_DIR" +for v in "${VARIANTS[@]}"; do + case "$v" in + buildpacks) + mvn -q -o -DskipTests ${BUILDPACKS_ARGS:-} -Dbuildpacks.image="sbd/docker-images:buildpacks$SUFFIX" \ + spring-boot:build-image > /dev/null ;; + jib) + mvn -q -o -DskipTests ${JIB_ARGS:-} -Djib.to.image="sbd/docker-images:jib$SUFFIX" jib:dockerBuild > /dev/null ;; + *) + docker build -q -f "docker/Dockerfile.$v" -t "sbd/docker-images:$v$SUFFIX" . > /dev/null ;; + esac + echo "built sbd/docker-images:$v$SUFFIX" +done diff --git a/docker-images/scripts/demo-aot-mismatch.sh b/docker-images/scripts/demo-aot-mismatch.sh new file mode 100644 index 0000000..f503757 --- /dev/null +++ b/docker-images/scripts/demo-aot-mismatch.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# The AOT cache trained in image aot-cache, run by its own JVM and by a different JVM build. +# -> docs/output/aot-cache-mismatch.txt +set -uo pipefail +source "$(dirname "$0")/env.sh" +cd "$MODULE_DIR" +docker build -q -f docker/Dockerfile.aot-cache-mismatch -t sbd/docker-images:aot-cache-mismatch . > /dev/null +{ + for t in aot-cache aot-cache-mismatch; do + name="aot-$$"; docker rm -f "$name" > /dev/null 2>&1 + docker run -d --name "$name" "sbd/docker-images:$t" > /dev/null + for _ in $(seq 1 200); do docker logs "$name" 2>&1 | grep -q 'Started ImagesApplication' && break; sleep 0.1; done + echo "## sbd/docker-images:$t" + docker logs "$name" 2>&1 | grep -E '\[aot\]|Starting ImagesApplication|Started ImagesApplication' \ + | sed -E 's/^[0-9T:.-]+Z +INFO 1 --- \[docker-images\] \[ +main\] com.ankurm.images.ImagesApplication +: //' | cut -c1-120 + echo "exit status while running: $(docker inspect -f '{{.State.Status}}' "$name")" + docker rm -f "$name" > /dev/null + echo + done +} | tee "$OUT/aot-cache-mismatch.txt" diff --git a/docker-images/scripts/demo-buildpacks-layers.sh b/docker-images/scripts/demo-buildpacks-layers.sh new file mode 100644 index 0000000..c37002e --- /dev/null +++ b/docker-images/scripts/demo-buildpacks-layers.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Where the buildpacks image's size goes, and why its JRE is bigger than Temurin's. +# -> docs/output/buildpacks-layers.txt +set -uo pipefail +source "$(dirname "$0")/env.sh" +W=$(mktemp -d) +{ + echo "# docker history sbd/docker-images:buildpacks" + docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' sbd/docker-images:buildpacks | cut -c1-110 + echo + cid=$(docker create sbd/docker-images:buildpacks); docker export "$cid" | tar -x -C "$W" layers/paketo-buildpacks_bellsoft-liberica/jre; docker rm "$cid" > /dev/null + cid=$(docker create eclipse-temurin:25-jre); docker export "$cid" | tar -x -C "$W" opt/java/openjdk; docker rm "$cid" > /dev/null + L="$W/layers/paketo-buildpacks_bellsoft-liberica/jre"; T="$W/opt/java/openjdk" + echo "# The two JREs, in MB (du -sm)" + printf '%-44s %6s\n' "BellSoft Liberica JRE (buildpacks) total" "$(du -sm "$L" | cut -f1)" + printf '%-44s %6s\n' " lib/server" "$(du -sm "$L/lib/server" | cut -f1)" + printf '%-44s %6s\n' " lib/client" "$(du -sm "$L/lib/client" | cut -f1)" + printf '%-44s %6s\n' "Temurin JRE (eclipse-temurin:25-jre) total" "$(du -sm "$T" | cut -f1)" + printf '%-44s %6s\n' " lib/server" "$(du -sm "$T/lib/server" | cut -f1)" + printf '%-44s %6s\n' " lib/client" "$( [ -d "$T/lib/client" ] && du -sm "$T/lib/client" | cut -f1 || echo absent)" +} | tee "$OUT/buildpacks-layers.txt" +rm -rf "$W" diff --git a/docker-images/scripts/demo-buildpacks-memory.sh b/docker-images/scripts/demo-buildpacks-memory.sh new file mode 100644 index 0000000..36080b0 --- /dev/null +++ b/docker-images/scripts/demo-buildpacks-memory.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# The buildpacks image under common container memory limits. The Paketo memory calculator runs +# before the JVM and refuses to launch if its fixed regions do not fit. +# -> docs/output/buildpacks-memory.txt +set -uo pipefail +source "$(dirname "$0")/env.sh" +run() { # label, docker args... + local label="$1"; shift + local name="bpm-$$"; docker rm -f "$name" > /dev/null 2>&1 + docker run -d --name "$name" "$@" sbd/docker-images:buildpacks > /dev/null + for _ in $(seq 1 100); do + docker logs "$name" 2>&1 | grep -qE 'Started ImagesApplication|failed to launch' && break; sleep 0.1 + done + echo "## $label" + docker logs "$name" 2>&1 | grep -E 'Calculated JVM Memory|unable to calculate|fixed memory regions|failed to launch|Started ImagesApplication' \ + | sed -E 's/^[0-9T:.-]+Z +INFO 1 --- \[docker-images\] \[ +main\] com.ankurm.images.ImagesApplication +: //; s/\x1b\[[0-9;]*m//g' | cut -c1-230 + echo "state: $(docker inspect -f '{{.State.Status}} (exit {{.State.ExitCode}})' "$name")" + echo + docker rm -f "$name" > /dev/null +} +{ + run "docker run -m 512m" -m 512m + run "docker run -m 768m" -m 768m + run "docker run -m 512m -e BPL_JVM_THREAD_COUNT=50" -m 512m -e BPL_JVM_THREAD_COUNT=50 +} | tee "$OUT/buildpacks-memory.txt" diff --git a/docker-images/scripts/demo-jlink-metrics.sh b/docker-images/scripts/demo-jlink-metrics.sh new file mode 100644 index 0000000..5a39f74 --- /dev/null +++ b/docker-images/scripts/demo-jlink-metrics.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# The jlink image built from jdeps' module list alone runs fine - and exports fewer metrics. +# Diffs /actuator/prometheus metric names against the full-JRE image. -> docs/output/jlink-metrics.txt +set -uo pipefail +source "$(dirname "$0")/env.sh" +names() { + local name="jl-$$-$2" + docker rm -f "$name" > /dev/null 2>&1 + docker run -d --name "$name" -p "$2:8080" "$1" > /dev/null + for _ in $(seq 1 300); do curl -s -o /dev/null "localhost:$2/actuator/health" && break; sleep 0.1; done + curl -s -o /dev/null "localhost:$2/api/items" + curl -s "localhost:$2/actuator/prometheus" | grep -v '^#' | sed 's/[{ ].*//' | sort -u + docker logs "$name" 2>&1 | grep -E ' WARN ' | sed 's/^.* WARN [0-9]* --- \[[^]]*\] \[ *[^]]*\] //' >&2 + docker rm -f "$name" > /dev/null +} +{ + echo "# jlink modules chosen by jdeps alone:" + docker run --rm --entrypoint /opt/jre/bin/java sbd/docker-images:jlink-jdeps-only --list-modules | sed 's/@.*//' | tr '\n' ' '; echo + echo + echo "# WARN lines at startup of the jdeps-only image:" + names sbd/docker-images:jlink-jdeps-only 18101 2>&1 >/tmp/jl-a.txt | sed 's/^/ /' + names sbd/docker-images:layered-distroless 18102 2>/dev/null > /tmp/jl-b.txt + names sbd/docker-images:jlink-distroless 18103 2>/dev/null > /tmp/jl-c.txt + echo + echo "# Metric names: full JRE $(wc -l < /tmp/jl-b.txt), jdeps-only $(wc -l < /tmp/jl-a.txt), jdeps + jdk.management $(wc -l < /tmp/jl-c.txt)" + echo "# Present with the full JRE, missing from the jdeps-only image:" + comm -23 /tmp/jl-b.txt /tmp/jl-a.txt | sed 's/^/ /' + echo "# Missing from jdeps + jdk.management: $(comm -23 /tmp/jl-b.txt /tmp/jl-c.txt | wc -l)" + comm -23 /tmp/jl-b.txt /tmp/jl-c.txt | sed 's/^/ /' + echo "# (jvm_gc_concurrent_phase_time_* is registered lazily, after the first G1 concurrent cycle -" + echo "# whether it appears within a few seconds of startup varies run to run, in either image.)" +} | tee "$OUT/jlink-metrics.txt" diff --git a/docker-images/scripts/demo-signals.sh b/docker-images/scripts/demo-signals.sh new file mode 100644 index 0000000..b1ee51e --- /dev/null +++ b/docker-images/scripts/demo-signals.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Who is PID 1, does SIGTERM reach the JVM, and how long does `docker stop` take? +# -> docs/output/pid1-and-signals.txt +set -uo pipefail +source "$(dirname "$0")/env.sh" +probe() { + local tag="$1" name="sig-$$" + docker rm -f "$name" > /dev/null 2>&1 + docker run -d --name "$name" "sbd/docker-images:$tag" > /dev/null + for _ in $(seq 1 200); do docker logs "$name" 2>&1 | grep -q 'Started ImagesApplication' && break; sleep 0.1; done + local pid1 + # NUL-separated argv; translate in the pipe (a shell variable cannot hold NUL bytes) + pid1=$(docker exec "$name" cat /proc/1/cmdline 2>/dev/null | tr '\0' ' ') + case "$pid1" in ""|*"exec failed"*) pid1="(image has no cat; ENTRYPOINT is exec form)" ;; esac + local t0 t1; t0=$(date +%s%N); docker stop "$name" > /dev/null; t1=$(date +%s%N) + local code; code=$(docker inspect -f '{{.State.ExitCode}}' "$name") + local graceful; graceful=$(docker logs "$name" 2>&1 | grep -c 'Commencing graceful shutdown' || true) + printf '%-22s PID 1: %-58s docker stop: %5d ms exit code: %-4s graceful-shutdown log lines: %s\n' \ + "$tag" "${pid1:0:58}" "$(( (t1 - t0) / 1000000 ))" "$code" "$graceful" + docker rm -f "$name" > /dev/null +} +{ + echo "# docker stop sends SIGTERM to PID 1, waits 10 s, then SIGKILL. Exit code 143 = the JVM handled" + echo "# SIGTERM (128+15). 137 = it was killed (128+9)." + echo + for t in layered-jre shell-form shell-form-alpine shell-form-wrapper layered-distroless; do probe "$t"; done + echo + echo "# What /bin/sh is in each base image:" + echo "eclipse-temurin:25-jre -> $(docker run --rm --entrypoint readlink eclipse-temurin:25-jre -f /bin/sh)" + echo "eclipse-temurin:25-jre-alpine -> $(docker run --rm --entrypoint readlink eclipse-temurin:25-jre-alpine -f /bin/sh)" +} | tee "$OUT/pid1-and-signals.txt" diff --git a/docker-images/scripts/env.sh b/docker-images/scripts/env.sh new file mode 100644 index 0000000..4026bd6 --- /dev/null +++ b/docker-images/scripts/env.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +MODULE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +OUT="$MODULE_DIR/docs/output" +REGISTRY="${REGISTRY:-localhost:5000}" # any registry you can push to; see docs/02-measuring-size.md +REPO="$REGISTRY/sbd-docker-images" +mkdir -p "$OUT" +# Every variant the article measures, in the order the table lists them. +VARIANTS=(fatjar-jdk fatjar-jre layered-jre layered-alpine layered-distroless jlink-distroless aot-cache buildpacks jib) diff --git a/docker-images/scripts/fetch-buildpack-deps.sh b/docker-images/scripts/fetch-buildpack-deps.sh new file mode 100644 index 0000000..494cbe0 --- /dev/null +++ b/docker-images/scripts/fetch-buildpack-deps.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# For -Pno-maven-central only: download the one Maven Central artifact the Spring Boot buildpack +# fetches during a build, and check it against the sha256 the buildpack expects (which is also +# the name of the mapping file next to it). +set -euo pipefail +D="$(cd "$(dirname "$0")/.." && pwd)/bindings/dependency-mapping" +curl -sfL -o "$D/spring-cloud-bindings-2.0.4.jar" \ + https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-bindings/2.0.4/spring-cloud-bindings-2.0.4.jar +echo "32e47c2139d6379836910f0d1cc253a019ac282f3aeea12237069f00046279ad $D/spring-cloud-bindings-2.0.4.jar" | sha256sum -c diff --git a/docker-images/scripts/measure-rebuild.sh b/docker-images/scripts/measure-rebuild.sh new file mode 100644 index 0000000..1c30477 --- /dev/null +++ b/docker-images/scripts/measure-rebuild.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# "A developer changed one line": bump BuildInfo.REVISION, rebuild every variant, and count the +# layers - and bytes - that differ from the previous build. That is what CI pushes and what every +# node pulls on the next rollout. -> docs/output/rebuild-delta.txt +# Run scripts/measure.sh first: it pushes the revision-1 images this compares against. +set -euo pipefail +source "$(dirname "$0")/env.sh" +cd "$MODULE_DIR" +FILE=src/main/java/com/ankurm/images/BuildInfo.java +trap 'sed -i "s/REVISION = \"2\"/REVISION = \"1\"/" "$FILE"; mvn -q -o -DskipTests package' EXIT +sed -i 's/REVISION = "1"/REVISION = "2"/' "$FILE" +mvn -q -o -DskipTests package +./scripts/build-images.sh -r2 > /dev/null + +layers() { crane manifest "$1" | python3 -c 'import json,sys +for l in json.load(sys.stdin)["layers"]: print(l["digest"], l["size"])'; } + +{ + echo "# One-line code change (BuildInfo.REVISION 1 -> 2), every variant rebuilt." + echo "# 'new layers' = layer digests in revision 2 that revision 1 did not have: what a push uploads" + echo "# and what a node that already runs revision 1 downloads." + echo + printf '%-20s %12s %14s %14s %s\n' variant "new layers" "bytes to push" "image total" "share" + for v in "${VARIANTS[@]}"; do + docker tag "sbd/docker-images:$v-r2" "$REPO:$v-r2" && docker push -q "$REPO:$v-r2" > /dev/null + layers "$REPO:$v" | sort > /tmp/r1; layers "$REPO:$v-r2" | sort > /tmp/r2 + new=$(comm -13 <(cut -d' ' -f1 /tmp/r1) <(cut -d' ' -f1 /tmp/r2)) + count=$(echo "$new" | grep -c . || true) + bytes=0; for d in $new; do bytes=$(( bytes + $(grep "^$d " /tmp/r2 | cut -d' ' -f2) )); done + total=$(awk '{s+=$2} END {print s}' /tmp/r2) + printf '%-20s %6s of %-3s %13sK %13sM %5.1f%%\n' "$v" "$count" "$(wc -l < /tmp/r2)" "$(( bytes / 1000 ))" "$(( total / 1000000 ))" \ + "$(python3 -c "print(100*$bytes/$total)")" + done +} | tee "$OUT/rebuild-delta.txt" diff --git a/docker-images/scripts/measure.sh b/docker-images/scripts/measure.sh new file mode 100644 index 0000000..8ee5de5 --- /dev/null +++ b/docker-images/scripts/measure.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Measure every variant: size on disk, size over the wire, layers, user, shell, packages, +# startup to readiness, and memory at idle. -> docs/output/image-matrix.txt +# Needs a registry at $REGISTRY (crane registry serve --address localhost:5000 is enough). +set -uo pipefail +source "$(dirname "$0")/env.sh" +RUNS="${RUNS:-3}" +PORT=18090 + +compressed() { # image in registry -> "bytes layers" + crane manifest "$1" | python3 -c 'import json,sys +m=json.load(sys.stdin); L=m["layers"] +print(sum(l["size"] for l in L)+m["config"]["size"], len(L))' +} + +packages() { # count OS packages from the image filesystem, without running anything in it + local cid; cid=$(docker create "$1" /x 2>/dev/null) + docker export "$cid" > /tmp/img-fs.tar; docker rm "$cid" > /dev/null + local n="" + if tar -tf /tmp/img-fs.tar var/lib/dpkg/status > /dev/null 2>&1; then + n=$(tar -xOf /tmp/img-fs.tar var/lib/dpkg/status | grep -c '^Package:') + n="$n (dpkg)" + elif [ "$(tar -tf /tmp/img-fs.tar | grep -c '^var/lib/dpkg/status.d/.')" -gt 0 ]; then + # distroless and Paketo "tiny" keep one status file per package (plus .md5sums) in status.d/ + n=$(tar -tf /tmp/img-fs.tar | grep -E '^var/lib/dpkg/status.d/[^/]+$' | grep -vc '\.md5sums$') + n="$n (status.d)" + elif tar -tf /tmp/img-fs.tar lib/apk/db/installed > /dev/null 2>&1; then + n=$(tar -xOf /tmp/img-fs.tar lib/apk/db/installed | grep -c '^P:') + n="$n (apk)" + else n="?"; fi + rm -f /tmp/img-fs.tar; echo "$n" +} + +startup() { # image -> "ms_to_ready startedIn rss" + local name="measure-$$" t0 t1 + docker rm -f "$name" > /dev/null 2>&1 + t0=$(date +%s%N) + docker run -d --name "$name" -p "$PORT:8080" "$1" > /dev/null + for _ in $(seq 1 600); do + if [ "$(curl -s -o /dev/null -w '%{http_code}' localhost:$PORT/actuator/health/readiness)" = 200 ]; then break; fi + sleep 0.05 + done + t1=$(date +%s%N) + for _ in 1 2 3 4 5; do curl -s -o /dev/null localhost:$PORT/api/items; done + sleep 2 + local rss started + rss=$(docker stats --no-stream --format '{{.MemUsage}}' "$name" | awk '{print $1}') + started=$(docker logs "$name" 2>&1 | grep -o 'Started ImagesApplication in [0-9.]* seconds' | awk '{print $4}') + docker rm -f "$name" > /dev/null + echo "$(( (t1 - t0) / 1000000 )) ${started:-?} $rss" +} + +{ + echo "# Spring Boot 4.1.1 application (webmvc + actuator + validation + prometheus), fat jar $(du -h "$MODULE_DIR/target/app.jar" | cut -f1)" + echo "# Docker Engine $(docker version --format '{{.Server.Version}}'), classic overlay2 store. $RUNS startup runs each, median shown." + echo "# 'on disk' = docker image inspect .Size (uncompressed). 'pushed' = layer blobs + config in the registry (compressed)." + echo + printf '%-20s %9s %9s %7s %-9s %-6s %-16s %9s %9s %9s %s\n' variant "on disk" pushed layers user shell packages "ready ms" "Started" "RSS" "JVM" + for v in "${VARIANTS[@]}"; do + img="sbd/docker-images:$v" + docker tag "$img" "$REPO:$v" && docker push -q "$REPO:$v" > /dev/null + disk=$(docker image inspect -f '{{.Size}}' "$img") + read -r wire layers < <(compressed "$REPO:$v") + user=$(docker image inspect -f '{{.Config.User}}' "$img"); user="${user:-root}" + if docker run --rm --entrypoint /bin/sh "$img" -c true > /dev/null 2>&1; then shell=yes; else shell=no; fi + pkgs=$(packages "$img") + samples=() + for _ in $(seq 1 "$RUNS"); do samples+=("$(startup "$img")"); done + median=$(printf '%s\n' "${samples[@]}" | sort -n | sed -n "$(( (RUNS + 1) / 2 ))p") + read -r ready started rss <<< "$median" + name="measure-$$"; docker run -d --name "$name" -p "$PORT:8080" "$img" > /dev/null + for _ in $(seq 1 600); do curl -s -o /dev/null localhost:$PORT/api/runtime && break; sleep 0.05; done + jvm=$(curl -s localhost:$PORT/api/runtime | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["java.vendor"].split()[0], r["java.version"])') + docker rm -f "$name" > /dev/null + printf '%-20s %8sM %8sM %7s %-9s %-6s %-16s %9s %9s %9s %s\n' "$v" \ + "$(( disk / 1000000 ))" "$(( wire / 1000000 ))" "$layers" "$user" "$shell" "$pkgs" "$ready" "${started}s" "$rss" "$jvm" + done +} | tee "$OUT/image-matrix.txt" diff --git a/docker-images/scripts/run-all.sh b/docker-images/scripts/run-all.sh new file mode 100644 index 0000000..089fca2 --- /dev/null +++ b/docker-images/scripts/run-all.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# Regenerate every transcript under docs/output/ (about 15 minutes; startup is measured three +# times per image). Stop anything else CPU-hungry first - a parallel job skewed one startup figure +# by two seconds while this was being written. +# +# Needs: Docker, crane (github.com/google/go-containerregistry) for the size-over-the-wire numbers, +# and the base images listed in docs/01-the-variants.md. In a network-restricted build environment +# set BUILDPACKS_ARGS and JIB_ARGS - the values used for the committed output are the defaults below. +set -euo pipefail +cd "$(dirname "$0")/.." +source scripts/env.sh +export BUILDPACKS_ARGS="${BUILDPACKS_ARGS--Pcorporate-proxy,no-maven-central -Dspring-boot.build-image.pullPolicy=IF_NOT_PRESENT}" +export JIB_ARGS="${JIB_ARGS--Djib.from.image=docker://eclipse-temurin:25-jre}" + +crane registry serve --address "$REGISTRY" > /tmp/registry.log 2>&1 & +REG_PID=$! +trap 'kill $REG_PID 2>/dev/null' EXIT +sleep 1 + +mvn -q -DskipTests package +./scripts/build-images.sh +docker build -q -f docker/Dockerfile.jlink-distroless --build-arg EXTRA_MODULES= -t sbd/docker-images:jlink-jdeps-only . > /dev/null +for v in shell-form shell-form-alpine shell-form-wrapper; do + docker build -q -f "docker/Dockerfile.$v" -t "sbd/docker-images:$v" . > /dev/null +done + +./scripts/measure.sh +./scripts/measure-rebuild.sh +./scripts/demo-signals.sh +./scripts/demo-jlink-metrics.sh +./scripts/demo-aot-mismatch.sh +./scripts/demo-buildpacks-memory.sh +./scripts/demo-buildpacks-layers.sh +echo "Regenerated: $(ls docs/output | wc -l) files in docs/output/" diff --git a/docker-images/src/main/java/com/ankurm/images/BuildInfo.java b/docker-images/src/main/java/com/ankurm/images/BuildInfo.java new file mode 100644 index 0000000..22225e8 --- /dev/null +++ b/docker-images/src/main/java/com/ankurm/images/BuildInfo.java @@ -0,0 +1,14 @@ +package com.ankurm.images; + +/** + * The one line scripts/measure-rebuild.sh edits to simulate "a developer changed some code". The + * point is to see which image layers change - and therefore how many bytes a registry push and + * a node pull cost - when nothing but application code moved. + */ +public final class BuildInfo { + + public static final String REVISION = "1"; + + private BuildInfo() { + } +} diff --git a/docker-images/src/main/java/com/ankurm/images/ImagesApplication.java b/docker-images/src/main/java/com/ankurm/images/ImagesApplication.java new file mode 100644 index 0000000..90a3d62 --- /dev/null +++ b/docker-images/src/main/java/com/ankurm/images/ImagesApplication.java @@ -0,0 +1,12 @@ +package com.ankurm.images; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class ImagesApplication { + + public static void main(String[] args) { + SpringApplication.run(ImagesApplication.class, args); + } +} diff --git a/docker-images/src/main/java/com/ankurm/images/InfoController.java b/docker-images/src/main/java/com/ankurm/images/InfoController.java new file mode 100644 index 0000000..344d904 --- /dev/null +++ b/docker-images/src/main/java/com/ankurm/images/InfoController.java @@ -0,0 +1,52 @@ +package com.ankurm.images; + +import java.lang.management.ManagementFactory; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import jakarta.validation.constraints.NotBlank; + +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RestController; + +/** + * A deliberately ordinary service - web, validation, Actuator, Prometheus - so the image sizes + * measured here are what a typical Spring Boot 4 service would get, not a hello-world floor. + * {@code /api/runtime} reports what the JVM inside the container actually sees. + */ +@RestController +public class InfoController { + + public record Item(@NotBlank String name) { + } + + @GetMapping("/api/items") + public List items() { + return List.of(new Item("widget"), new Item("gadget")); + } + + @PostMapping("/api/items") + public Item create(@jakarta.validation.Valid @RequestBody Item item) { + return item; + } + + @GetMapping("/api/runtime") + public Map runtime() { + Runtime rt = Runtime.getRuntime(); + Map out = new LinkedHashMap<>(); + out.put("revision", BuildInfo.REVISION); + out.put("java.version", System.getProperty("java.version")); + out.put("java.vendor", System.getProperty("java.vendor")); + out.put("java.home", System.getProperty("java.home")); + out.put("user.name", System.getProperty("user.name")); + out.put("pid", ProcessHandle.current().pid()); + out.put("availableProcessors", rt.availableProcessors()); + out.put("maxHeapMiB", rt.maxMemory() / (1024 * 1024)); + out.put("inputArguments", ManagementFactory.getRuntimeMXBean().getInputArguments()); + out.put("modules", ModuleLayer.boot().modules().size()); + return out; + } +} diff --git a/docker-images/src/main/resources/application.yaml b/docker-images/src/main/resources/application.yaml new file mode 100644 index 0000000..2b58054 --- /dev/null +++ b/docker-images/src/main/resources/application.yaml @@ -0,0 +1,12 @@ +spring: + application: + name: docker-images +management: + endpoints: + web: + exposure: + include: health,info,prometheus + endpoint: + health: + probes: + enabled: true diff --git a/docker-images/src/test/java/com/ankurm/images/ImagesApplicationTest.java b/docker-images/src/test/java/com/ankurm/images/ImagesApplicationTest.java new file mode 100644 index 0000000..58f3a73 --- /dev/null +++ b/docker-images/src/test/java/com/ankurm/images/ImagesApplicationTest.java @@ -0,0 +1,20 @@ +package com.ankurm.images; + +import org.junit.jupiter.api.Test; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; + +import static org.assertj.core.api.Assertions.assertThat; + +@SpringBootTest +class ImagesApplicationTest { + + @Autowired + InfoController controller; + + @Test + void runtimeReportsTheRevision() { + assertThat(controller.runtime()).containsEntry("revision", BuildInfo.REVISION); + } +}