Add onnx-djl module: DistilBERT sentiment with ONNX Runtime and DJL, batching and a Spring Boot endpoint with measured latency

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk

onnx-djl/output/01-model-and-tokenizer.txt

@@ -0,0 +1,19 @@
+Model: distilbert-base-uncased-finetuned-sst-2-english, ONNX export, 267 MB
+ONNX Runtime 1.31.0
+
+Inputs of the ONNX graph:
+  input_ids TensorInfo(javaType=INT64,onnxType=ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,shape=[-1, -1],dimNames=[batch_size,sequence_length])
+  attention_mask TensorInfo(javaType=INT64,onnxType=ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,shape=[-1, -1],dimNames=[batch_size,sequence_length])
+Outputs:
+  logits TensorInfo(javaType=FLOAT,onnxType=ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT,shape=[-1, 2],dimNames=[batch_size,""])
+
+Tokenizing "I loved this film.":
+  tokens:         [[CLS], i, loved, this, film, ., [SEP]]
+  input_ids:      [101, 1045, 3866, 2023, 2143, 1012, 102]
+  attention_mask: [1, 1, 1, 1, 1, 1, 1]
+
+Two texts in one batch are padded to the same length:
+  [[CLS], i, loved, this, film, ., [SEP]]
+    mask [1, 1, 1, 1, 1, 1, 1]
+  [[CLS], bad, ., [SEP], [PAD], [PAD], [PAD]]
+    mask [1, 1, 1, 1, 0, 0, 0]

onnx-djl/output/02-classification.txt

@@ -0,0 +1,8 @@
+ONNX Runtime directly, DJL tokenizer, one batch of 6 texts.
+
+  POSITIVE  1.00  I loved this film, the acting was wonderful.
+  NEGATIVE  1.00  The service was slow and the food arrived cold.
+  POSITIVE  1.00  It works.
+  POSITIVE  1.00  Not bad at all.
+  NEGATIVE  1.00  I wanted to love it, but it was a mess.
+  NEGATIVE  0.95  The build finished in four minutes.

onnx-djl/output/03-djl-engine.txt

@@ -0,0 +1,10 @@
+DJL engine: OnnxRuntime 1.21.1
+
+  direct POSITIVE  0.9999 | DJL POSITIVE  0.9999 | difference 0.00e+00
+  direct NEGATIVE  0.9998 | DJL NEGATIVE  0.9998 | difference 0.00e+00
+  direct POSITIVE  0.9999 | DJL POSITIVE  0.9999 | difference 0.00e+00
+  direct POSITIVE  0.9993 | DJL POSITIVE  0.9993 | difference 0.00e+00
+  direct NEGATIVE  0.9996 | DJL NEGATIVE  0.9996 | difference 0.00e+00
+  direct NEGATIVE  0.9491 | DJL NEGATIVE  0.9491 | difference 0.00e+00
+
+Same labels: true, largest score difference: 0.00e+00

onnx-djl/output/04-batching.txt

@@ -0,0 +1,9 @@
+Padding does not change the answers: 32 texts one by one versus one batch of 32.
+  same labels: true, largest score difference 0.00e+00
+
+Time to classify 32 texts, median of 5 runs after a warm-up, by batch size:
+  batch size  1:   230 ms in total,   7.2 ms per text
+  batch size  4:   204 ms in total,   6.4 ms per text
+  batch size  8:   166 ms in total,   5.2 ms per text
+  batch size 16:   142 ms in total,   4.4 ms per text
+  batch size 32:   134 ms in total,   4.2 ms per text

onnx-djl/output/05-engine-order.txt

@@ -0,0 +1,8 @@
+The same two things started in one JVM, in both orders (a separate JVM for each).
+
+direct-first (ONNX Runtime created by my code, then DJL):
+  Caused by: java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.
+
+djl-first (ONNX Runtime created by DJL, then my code):
+  both started
+

onnx-djl/output/06-endpoint.txt

@@ -0,0 +1,16 @@
+POST /sentiment
+{"texts":["I loved this film, the acting was wonderful.","The service was slow and the food arrived cold."]}
+
+HTTP 200
+{
+  "results" : [ {
+    "label" : "POSITIVE",
+    "score" : 0.9998835686297906
+  }, {
+    "label" : "NEGATIVE",
+    "score" : 0.9997554818864473
+  } ]
+}
+
+Empty list: HTTP 400
+65 texts (limit is 64): HTTP 400

onnx-djl/output/07-http-latency.txt

@@ -0,0 +1,8 @@
+Round trip over HTTP on the same machine (client and server in one JVM), median of 20 requests.
+
+   1 text(s) per request:   17.6 ms per request,  17.6 ms per text
+   4 text(s) per request:   31.7 ms per request,   7.9 ms per text
+  16 text(s) per request:   80.0 ms per request,   5.0 ms per text
+  64 text(s) per request:  297.2 ms per request,   4.6 ms per text
+
+100 requests of 1 text: one client after another 1.7 s (59 requests/s), 4 clients at once 1.4 s (70 requests/s)

onnx-djl/output/08-dependencies.txt

@@ -0,0 +1,20 @@
+mvn dependency:tree for the onnx-djl module (first level and the ONNX Runtime entries):
+ai.djl:api:jar:0.38.0:compile
+ai.djl.huggingface:tokenizers:jar:0.38.0:compile
+ai.djl.onnxruntime:onnxruntime-engine:jar:0.38.0:compile
+com.microsoft.onnxruntime:onnxruntime:jar:1.31.0:compile
+org.springframework.boot:spring-boot-starter-webmvc:jar:4.1.1:compile
+org.slf4j:slf4j-simple:jar:2.0.17:test
+org.springframework.boot:spring-boot-starter-test:jar:4.1.1:test
+org.assertj:assertj-core:jar:3.27.7:test
+
+ONNX Runtime and DJL entries anywhere in the tree:
+ai.djl.huggingface:tokenizers:jar:0.38.0:compile
+ai.djl.onnxruntime:onnxruntime-engine:jar:0.38.0:compile
+ai.djl:api:jar:0.38.0:compile
+com.microsoft.onnxruntime:onnxruntime:jar:1.31.0:compile
+
+Verbose tree lines about ONNX Runtime (-Dverbose):
++- ai.djl.onnxruntime:onnxruntime-engine:jar:0.38.0:compile
+|  \- (com.microsoft.onnxruntime:onnxruntime:jar:1.21.1:compile - omitted for conflict with 1.31.0)
++- com.microsoft.onnxruntime:onnxruntime:jar:1.31.0:compile (scope not updated to compile)