Java on the GPU with TornadoVM 7: When It Beats the Vector API
How TornadoVM 7.2.0 task graphs and @Parallel loops work, with a measured plain-Java versus Vector API matrix-multiply baseline (up to 43x over the naive loop). Honest about the gap: the TornadoVM code compiles but could not be run without a GPU, so there are no TornadoVM timings.
There are two ways to make a Java loop faster by doing many multiplications at once. One stays on the CPU: the Vector API asks each core to multiply 16 numbers in a single instruction. The other leaves the CPU: TornadoVM translates a Java method into a kernel and runs it on a GPU, where thousands of threads each do a little of the work. Both are “Java, but parallel”. The question this post sets out to answer is the practical one: when does the GPU route win?
Before the structure, the honest part, because it shapes everything below.
What I could and could not measure. The sandbox that built this post has no GPU and no OpenCL runtime (no /etc/OpenCL/vendors, no clinfo) and no TornadoVM SDK. So the TornadoVM program in this post compiles against the 7.2.0 API but was never executed, and I have no TornadoVM timings. What I did measure for real is the CPU side: plain Java and the Vector API on a matrix multiply at three sizes. The “when does the GPU win” section is therefore an argument built from those measurements and from arithmetic, clearly labelled, not a benchmark result. If you have a GPU, running TornadoMatMul.java next to MatMulBench.java is the experiment this post could not do.
Versions. JDK 25 (Temurin 25.0.4.1+1), io.github.beehive-lab:tornado-api7.2.0-jdk22plus (Maven Central lists 7.2.0 in both jdk21 and jdk22plus flavours; the tornado-runtime metadata shows lastUpdated 2026-10-07), the incubating jdk.incubator.vector module run with --add-modules jdk.incubator.vector. Benchmark machine: 2 vCPU Intel Xeon @ 2.10 GHz whose preferred float vector species is 512 bits (16 lanes). There is no companion repository; sources are in the media library, linked beside each block.
What goes wrong first: the API jar alone cannot run anything
The tempting first step is to add the Maven dependency and run. The compile works; the first line that touches TornadoVM fails:
Exception in thread "main" uk.ac.manchester.tornado.api.exceptions.TornadoAPIException: [ERROR] Tornado API Implementation class not specified. Did you remember to add @tornado-argfile?
at uk.ac.manchester.tornado.api.runtime.TornadoAPIProvider.loadScheduleRuntime(TornadoAPIProvider.java:36)
at uk.ac.manchester.tornado.api.TaskGraph.<init>(TaskGraph.java:98)
at TornadoMatMul.main(TornadoMatMul.java:34)
Full capture: tornadovm-run-without-sdk.txt, from running TornadoMatMul.java with only the API jars on the classpath. The message is the whole lesson: tornado-api is the API; the implementation (runtime plus a backend driver for your GPU) is loaded through an argument file that TornadoVM’s own launcher supplies. The installation page I read describes SDK builds installed with SDKMAN! in the form <version>-<jdk-version>-<backend> and lists sdk list tornadovm to see the combinations, and it also describes a Maven route that adds only the API to an existing project. That page did not show a run command, and I could not run the SDK here, so I will not invent one: follow the current installation page for your machine.
The fingerprint.TornadoAPIException ... Tornado API Implementation class not specified at new TaskGraph(...) means “the program started like an ordinary JVM”. Nothing is wrong with your code; start it the way the TornadoVM installation instructs.
The mental model: SIMD lanes versus GPU threads, and the bus between them
A CPU core is fast at one thing at a time, with a trick: a SIMD instruction applies one operation to a whole row of numbers (16 floats on this machine’s 512-bit vectors). That is what the Vector API exposes. A GPU instead has many simple processing units and wants thousands of independent pieces of work. TornadoVM maps loop iterations onto them.
The catch is distance. A GPU has its own memory. Your arrays live in the Java process, so the data must be copied to the device before the kernel runs and the result copied back afterwards.
Everything in the rest of the post is either about the green box (how fast can the kernel go?) or the orange boxes (how much does it cost to get there?). The Vector API route has no orange boxes, which is its whole advantage for small jobs.
A matrix multiply is the textbook GPU workload: every output cell is independent. The TornadoVM version is a plain Java method with two annotated loops, and a task graph that says what to run and what data to move.
public static void matmul(FloatArray a, FloatArray b, FloatArray c, final int n) {
for (@Parallel int i = 0; i < n; i++) {
for (@Parallel int j = 0; j < n; j++) {
float sum = 0f;
for (int k = 0; k < n; k++) {
sum += a.get(i * n + k) * b.get(k * n + j);
}
c.set(i * n + j, sum);
}
}
}
TaskGraph graph = new TaskGraph("mm")
.transferToDevice(DataTransferMode.FIRST_EXECUTION, a, b)
.task("t0", TornadoMatMul::matmul, a, b, c, n)
.transferToHost(DataTransferMode.EVERY_EXECUTION, c);
ImmutableTaskGraph snapshot = graph.snapshot();
try (TornadoExecutionPlan plan = new TornadoExecutionPlan(snapshot)) {
for (int run = 0; run < 5; run++) {
plan.execute();
}
}
Source: TornadoMatMul.java. Compiled against tornado-api 7.2.0-jdk22plus; not executed. Read it piece by piece:
@Parallel on the i and j loops marks them as safe to run as independent threads (no iteration depends on another). The inner k loop stays sequential, accumulating one dot product per cell. The TornadoVM programming-model page says the framework does not auto-parallelise methods; you opt in with this annotation.
FloatArray is TornadoVM’s off-heap array type, used instead of float[], with get and set accessors.
transferToDevice(FIRST_EXECUTION, a, b) copies the inputs once. If you run the graph five times, a and b are not re-sent. transferToHost(EVERY_EXECUTION, c) copies the result back each time. These are the orange boxes in the diagram, and choosing the mode is how you keep them from repeating.
Building the graph computes nothing; execute() does. (The docs page says so; I could not observe it.)
This is the same shape as the matrix-multiply example on the TornadoVM programming-model page. One compile-time detail I did hit: TornadoExecutionPlan implements AutoCloseable and its close() declares a checked TornadoExecutionPlanException, so main has to declare throws Exception (or catch it) when you use try-with-resources.
Going deeper: task graphs
I confirmed the existence and names of TaskGraph, TornadoExecutionPlan (with execute and withDevice), @Parallel, FloatArray and the three DataTransferMode constants (FIRST_EXECUTION, EVERY_EXECUTION, UNDER_DEMAND) with javap on the 7.2.0 jar. The docs page also describes @Reduce and lower-level KernelContext/TileContext APIs, which I did not touch.
The docs page shows the snapshot call spelled snaphot() in one place; the method in the jar I compiled against is snapshot().
Which backends exist in 7.2.0
The brief for this post said “CUDA/OpenCL/SPIR-V”. That does not match what I found, so here is the primary source: the backend enum in the 7.2.0 API jar.
public static final uk.ac.manchester.tornado.api.enums.TornadoVMBackendType OPENCL;
public static final uk.ac.manchester.tornado.api.enums.TornadoVMBackendType METAL;
public static final uk.ac.manchester.tornado.api.enums.TornadoVMBackendType CUDA;
public static final uk.ac.manchester.tornado.api.enums.TornadoVMBackendType JAVA;
public static final uk.ac.manchester.tornado.api.enums.TornadoVMBackendType VIRTUAL;
Output of javap on TornadoVMBackendType, saved as tornadovm-backend-types.txt. The enum lists OpenCL, Metal and CUDA as the device backends, plus JAVA and VIRTUAL. It does not list PTX or SPIR-V. The programming-model page I read says parallel loops map to ND-ranges for “OpenCL, CUDA and PTX” and mentions Metal; the installation page lists OpenCL, CUDA and Metal and says nothing of SPIR-V or PTX. I therefore cannot confirm SPIR-V in 7.2.0 from either source, and I would not repeat that claim without checking the release notes for your version. I also did not find out what the JAVA and VIRTUAL entries do, only that they exist.
The part I could measure: plain Java versus the Vector API
Single thread, float matrices, three implementations of the same multiply (MatMulBench.java): naive (the textbook i-j-k loops, which read matrix B down a column), ikj (the same arithmetic with the loops reordered so everything is read along rows), and vector (the ikj order written with FloatVector and fused multiply-add). Each size is run with 3 warm-ups and 7 timed runs and the median is reported; GFLOP/s is 2n³ divided by the time.
n=512 vector species=Species[float, 16, S_512_BIT] lanes=16
naive median 142.7 ms 1.88 GFLOP/s
ikj median 53.8 ms 4.99 GFLOP/s (2.7x naive)
vector median 12.6 ms 21.25 GFLOP/s (11.3x naive, 4.26x ikj)
max abs difference vs naive: ikj 0.00e+00, vector 4.58e-05
That is the n=512 block of tornadovm-matmul-cpu.txt, which holds all three sizes. The summary across sizes:
n
naive
ikj
Vector API
256
16.8 ms (2.00 GFLOP/s)
6.2 ms (5.39)
1.6 ms (21.55) — 10.8x naive, 4.00x ikj
512
142.7 ms (1.88)
53.8 ms (4.99)
12.6 ms (21.25) — 11.3x naive, 4.26x ikj
1024
7111.4 ms (0.30)
1136.1 ms (1.89)
164.8 ms (13.03) — 43.2x naive, 6.89x ikj
Every number in the table is from the same file. The figure below draws the n=512 row:
Three things to take from the numbers, in decreasing order of surprise.
The naive loop collapses at n=1024 (0.30 GFLOP/s, 7.1 seconds). Reading B down a column makes every access jump a whole row; once the matrices no longer fit in cache, almost every read goes to main memory. That is consistent with the data, but I did not measure cache misses, so treat it as the likely cause. Just reordering the loops (ikj) is 6.3x faster at that size with no library at all.
The Vector API adds another 4x to 7x on top of ikj. Its speed-up over ikj grows from 4.00x at n=256 to 6.89x at n=1024; I did not investigate why.
The Vector API’s own GFLOP/s falls from about 21 to 13 at n=1024. The arithmetic did not get harder; the working set (three 4 MiB matrices at that size) got bigger than what stays close to the core. That is a ceiling the GPU route would also face on the way to the device, which leads to the next section.
Measurement caveats. One thread on a shared 2-vCPU cloud VM, one run of the whole benchmark per size, so read the ratios, not the third digit. The max-difference line shows the Vector API result differs from the naive one by up to 9.16e-05 at n=1024: floating-point addition is not associative, so a different summation order (and fused multiply-add) gives slightly different floats. That is expected, not a bug, but do not compare float results with ==.
Going deeper: the CPU baseline
The Vector API is still incubating and needs --add-modules jdk.incubator.vector at compile time and run time. See the SIMD guide for the current status.
A real application would use a tuned library or blocked (tiled) loops; this benchmark is deliberately simple so the three variants are readable.
When would the GPU win? An argument, not a benchmark
I can compute two things exactly from the matrix sizes. A multiply of two n×n matrices does 2n³ floating-point operations and, for a GPU, has to move three matrices of 4n² bytes each (A and B in, C out) — about 12n² bytes. So the **work per byte moved** is 2n³ / 12n² = n/6 operations per byte.
n
Work (GFLOP)
Data moved (MB)
FLOP per byte
Best CPU time measured
256
0.034
0.79
42.7
1.6 ms
512
0.268
3.15
85.3
12.6 ms
1024
2.147
12.58
170.7
164.8 ms
The first three columns are arithmetic (n³, n² and their ratio); the last is from the table above. The shape of the argument follows. The work grows with n³ but the data with n², so **bigger matrices make the copy cheaper relative to the compute**, which is the regime where a GPU pays off. At n=256, the best CPU time is 1.6 ms; for the GPU route to win it would have to copy 0.79 MB each way and also finish the kernel in the remainder of those 1.6 ms, and the first execution of a graph also includes TornadoVM compiling your Java method into a kernel (as I understand the design; I could not observe it). At n=1024 the CPU needs 164.8 ms for 2.1 GFLOP, and 12.6 MB of copying is small in comparison for any ordinary bus.
So the break-even is: GPU wins when copy-in time + kernel time + copy-out time < CPU time. With the copy sizes above and a bus bandwidth and GPU throughput from your own hardware you can plug numbers in; I deliberately do not, because I have neither. What the structure tells you without them:
Small or one-off problems favour the Vector API (no copies, no kernel compile).
Large problems with high arithmetic per byte (matrix multiply, big stencil or n-body style kernels) favour the GPU, and the advantage grows with size.
Low-arithmetic work (adding two vectors, one pass over data) is limited by memory traffic on both routes, and the copy to the device can dominate; the GPU rarely wins there.
Repeated execution on the same data is where FIRST_EXECUTION helps: upload once, run many times.
What would settle it. Run TornadoMatMul.java under a TornadoVM SDK on your GPU at n=256, 512 and 1024, discard the first execution (which includes compilation), and compare its per-run time with the CPU medians above on your machine. If the numbers contradict the argument here, trust the numbers.
Reference: how the CPU benchmark and the compile check were run
Benchmark: javac --add-modules jdk.incubator.vector MatMulBench.java, then java --add-modules jdk.incubator.vector MatMulBench 512 (and 256, 1024), per the header comment of MatMulBench.java. Random inputs from new Random(42), one thread, 3 warm-ups, 7 timed runs, median reported. The machine reported two CPUs and a 2.10 GHz Xeon, and the JVM chose the 512-bit species shown in the first line of each block.
Compile check: javac -cp "tornado-api-7.2.0-jdk22plus.jar:..." against the API jar and its transitive dependencies resolved by Maven; the first attempt failed with “unreported exception TornadoExecutionPlanException; must be caught or declared to be thrown” from the try-with-resources, fixed by main(...) throws Exception. The program was then started with plain java to capture the failure shown at the top; that is the only time TornadoVM code ran.
Should you even do this? Probably not yet, and probably not for matrix multiply: for dense linear algebra, a tuned native library behind a Java binding usually beats hand-rolled kernels, and if you only need a few times more speed on a CPU, the loop reordering and the Vector API in this post got 11x to 43x over naive Java with no hardware change. Reach for TornadoVM when you have a GPU in production, a compute-heavy kernel with high arithmetic per byte, data you can keep on the device across runs, and a team willing to own an extra runtime (an SDK, drivers, a launcher) in deployment. And hold this post to its limits: the TornadoVM half is compiled, not run, and the break-even is reasoning, not data.
No Comments yet!