Add batch-llm module: 100,000 synthetic tickets classified by a ChatClient in a Spring Batch 6 chunk step with Resilience4j rate limiting and retry, skip-and-record for malformed JSON, restart after kill -9 in a second JVM, and a per-chunk cost ledger. The model is a deterministic simulated ChatModel; prices are configurable placeholders. No provider called.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017ic4b4VQVEJWetTT1dYzyJ
This commit is contained in:
Claude Sonnet 5.5
@@ -0,0 +1,23 @@
|
||||
# What is on the classpath and what the builder offers
|
||||
|
||||
Spring Boot 4.1.2
|
||||
Spring Framework 7.0.10
|
||||
Spring Batch (core) 6.0.6
|
||||
Spring AI (client) 2.0.2
|
||||
Resilience4j 2.4.0
|
||||
Java 25.0.5
|
||||
|
||||
Resilience4j config read from application.yml by resilience4j-spring-boot4:
|
||||
ratelimiter 'llm': limitForPeriod=200 refreshPeriod=PT0.1S timeout=PT1M
|
||||
retry 'llm': maxAttempts=4
|
||||
|
||||
ChunkOrientedStepBuilder methods that matter here (javap of spring-batch-core):
|
||||
faultTolerant []
|
||||
taskExecutor [org.springframework.core.task.AsyncTaskExecutor]
|
||||
retryPolicy [org.springframework.core.retry.RetryPolicy]
|
||||
retry [[Ljava.lang.Class;]
|
||||
retryLimit [long]
|
||||
skip [[Ljava.lang.Class;]
|
||||
skipLimit [long]
|
||||
skipListener [org.springframework.batch.core.listener.SkipListener]
|
||||
processorNonTransactional []
|
||||
@@ -0,0 +1,15 @@
|
||||
# The simulated provider: same input, same outcome
|
||||
|
||||
Two separate provider instances, same seed, tickets 1..20,000, one call each: identical outcome lists = true
|
||||
|
||||
Outcome of the first call per ticket (the plan: 1.5% 429, 0.5% 503, 0.3% never valid, 1.2% invalid on the first reply only):
|
||||
429 304 1.52%
|
||||
503 93 0.47%
|
||||
bad JSON: category not in schema 76 0.38%
|
||||
bad JSON: missing field 77 0.39%
|
||||
bad JSON: not JSON 137 0.69%
|
||||
valid JSON 17,361 86.81%
|
||||
valid JSON (in a markdown fence, stripped) 1,952 9.76%
|
||||
|
||||
Quota check: quota 100 calls/second, 300 calls fired back to back -> 201 rejected with 429 (quota rejections plus any injected 429)
|
||||
Every number above is a property of the TEST PLAN, not a measurement of any provider.
|
||||
@@ -0,0 +1,11 @@
|
||||
# 4,000 tickets against a simulated quota of 800 calls per second
|
||||
|
||||
Both runs: 64 virtual threads, Resilience4j retry max 4 attempts, 25 ms x2 backoff, the same fault plan.
|
||||
Time is indicative only (shared 2-vCPU sandbox, other processes running).
|
||||
|
||||
scenario ok bad JSON exhausted calls quota 429s retries seconds
|
||||
no limiter 3066 13 921 7,596 4,396 3,543 3.5
|
||||
limiter 600/s 3986 14 0 4,146 0 85 6.9
|
||||
|
||||
'exhausted' = tickets whose 4 attempts all met a 429/503: with a step around this call, the step fails.
|
||||
'calls' counts every attempt the provider received, rejected or not.
|
||||
@@ -0,0 +1,33 @@
|
||||
# 3,000 tickets through the real job
|
||||
|
||||
job status COMPLETED
|
||||
step: read=3000 write=2987 processSkip=13 commits=30 rollbacks=0
|
||||
TICKET_RESULT rows 2987
|
||||
SKIPPED_TICKET rows 13
|
||||
classified + skipped 3000 (tickets: 3000)
|
||||
|
||||
Skipped tickets and why (first one of each kind):
|
||||
ticket 46 category not in schema raw reply: {"category":"Account & Payments","priority":1,"confidence":0.74}
|
||||
ticket 234 not JSON raw reply: Sure! Here is the classification you asked for: {"category":"BILLING","priority":4,"confidence":0.7}
|
||||
ticket 491 missing field raw reply: {"category":"OTHER","priority":2}
|
||||
|
||||
First 3 ledger rows (one per committed chunk):
|
||||
tickets written skipped attempts 429s 503s bad promptTok complTok cost
|
||||
1-100 99 1 104 1 1 3 10831 1469 0.007619000
|
||||
101-200 100 0 100 0 0 0 10574 1453 0.007466500
|
||||
201-300 99 1 105 2 1 3 10736 1495 0.007610500
|
||||
|
||||
Ledger totals vs the provider's own meter (they must agree when nothing was lost):
|
||||
ledger provider
|
||||
calls received 3,104 3,104
|
||||
429 rejections 39 39
|
||||
503 rejections 14 14
|
||||
billed calls 3,051 3,051
|
||||
prompt tokens 324,790 324,790
|
||||
completion tokens 44,107 44,107
|
||||
|
||||
Cost = prompt tokens x input price + completion tokens x output price, per million tokens.
|
||||
table 'EXAMPLE-NOT-A-REAL-PRICE' (input 0.50, output 1.50): 0.228555500 <- the ledger's total
|
||||
same tokens under table 'OTHER-EXAMPLE' (input 2.00, output 6.00): 0.914222000
|
||||
Both tables are invented inputs. Tokens are the measurement; money is tokens x a table you supply.
|
||||
Tokens spent on replies that were then skipped: bad replies billed = 64 of 3051 billed calls.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Which layer retries what: counts taken from the provider's meter
|
||||
|
||||
(a) A transient DB failure on the write of chunk 2 (tickets 101-200), 300 tickets, Spring Batch write retry = 2
|
||||
no failure status=COMPLETED billed calls=304 rollbacks=0
|
||||
failure + Spring Batch retry status=COMPLETED billed calls=304 rollbacks=0
|
||||
-> the write was retried inside the same transaction; the model was not asked again.
|
||||
|
||||
(b) The provider answers 429 to everything, 1 ticket. Calls the provider received for that ticket:
|
||||
Resilience4j retry only (max 4 attempts) status=FAILED attempts=4
|
||||
+ Spring Batch retry of the same exception (2) status=FAILED attempts=12
|
||||
-> 4 x (1 + 2) = 12. Two retry layers on the same failure multiply, and each attempt also takes a limiter permit.
|
||||
|
||||
(c) A poison row at WRITE time (ticket 250): chunk 3 (201-300) rolls back and is scanned one item at a time.
|
||||
processorNonTransactional (cached) status=COMPLETED billed calls=304 (no-failure baseline: 304) rollbacks=2 ledger rows=100 skipped=3
|
||||
default (processor runs again per item) status=COMPLETED billed calls=403 (extra over baseline: 99) rollbacks=2 ledger rows=100 skipped=3
|
||||
-> without processorNonTransactional(), every item of the failed chunk is classified a second time and paid for twice.
|
||||
@@ -0,0 +1,13 @@
|
||||
# A model regression, the skip limit, and a restart with a fixed model
|
||||
|
||||
Execution 1: simulated model returns invalid JSON for 50% of tickets, no re-ask, skip limit 50, 2,000 tickets.
|
||||
status / reason FAILED / org.springframework.batch.core.step.FatalStepExecutionException: Unable to process chunk
|
||||
tickets read 200 of 2000
|
||||
calls the provider billed before the step stopped: 200 (a full run needs about 2,000)
|
||||
committed results 52 skipped rows 50
|
||||
|
||||
Execution 2: same job parameters, same database, model fixed, skip limit 1000.
|
||||
status COMPLETED
|
||||
tickets read this time 1900 (execution 1 had committed 100, so the reader resumed after them)
|
||||
committed results 1942 skipped rows 58 classified + skipped = 2000
|
||||
tickets present in BOTH tables: 0 (stale skip rows from execution 1 are deleted when the ticket is classified)
|
||||
@@ -0,0 +1,20 @@
|
||||
# One uninterrupted run, 100000 tickets (java -jar, fresh H2 file, fresh provider meter)
|
||||
|
||||
START run.id=clean tickets=100000 chunk=100
|
||||
JOB FINISHED: executionId=1 status=COMPLETED seconds=55.2
|
||||
---- REPORT ----
|
||||
tickets total=100000 classified=99691 skipped=309 unaccounted=0
|
||||
step executions:
|
||||
{ID=1, JOB=1, STATUS=COMPLETED, READS=100000, WRITES=99691, SKIPS=309, COMMITS=1000, ROLLBACKS=0}
|
||||
ledger (committed chunks only): {ROWS_=1000, WRITTEN=99691, SKIPPED=309, ATTEMPTS=103519, R429=1531, U503=458, RETRIES=1989, BILLED=101530, BAD=1839, PT=10809111, CT=1467192, COST=7.605343500}
|
||||
price table 'EXAMPLE-NOT-A-REAL-PRICE': input 0.50 / output 1.50 per million tokens (configurable placeholder)
|
||||
provider's own meter (survives kill -9):
|
||||
received 103,519
|
||||
rejected_429 1,531
|
||||
window_429 0
|
||||
unavailable_503 458
|
||||
billed 101,530
|
||||
prompt_tokens 10,809,111
|
||||
completion_tokens 1,467,192
|
||||
billed calls the ledger does not know about (work lost in a crash or rollback): 0
|
||||
results checksum (sha-256 over ordered rows): 98bdd29feb6d6f8d
|
||||
@@ -0,0 +1,30 @@
|
||||
# Crash and restart, 100000 tickets: JVM 1 killed with kill -9 after 400 committed chunks, JVM 2 resumes
|
||||
|
||||
## JVM 1 (progress lines, then SIGKILL)
|
||||
PROGRESS committed-chunks-this-jvm=250 last-ticket=25000
|
||||
PROGRESS committed-chunks-this-jvm=300 last-ticket=30000
|
||||
PROGRESS committed-chunks-this-jvm=350 last-ticket=35000
|
||||
PROGRESS committed-chunks-this-jvm=400 last-ticket=40000
|
||||
(process killed by SIGKILL; no shutdown hook ran)
|
||||
|
||||
## JVM 2
|
||||
RECOVER: execution 1 is STARTED but its JVM is gone; marking it FAILED so the instance can restart
|
||||
START run.id=crash-demo tickets=100000 chunk=100
|
||||
JOB FINISHED: executionId=33 status=COMPLETED seconds=34.2
|
||||
---- REPORT ----
|
||||
tickets total=100000 classified=99691 skipped=309 unaccounted=0
|
||||
step executions:
|
||||
{ID=1, JOB=1, STATUS=FAILED, READS=39600, WRITES=39493, SKIPS=107, COMMITS=396, ROLLBACKS=0}
|
||||
{ID=33, JOB=33, STATUS=COMPLETED, READS=60400, WRITES=60198, SKIPS=202, COMMITS=604, ROLLBACKS=0}
|
||||
ledger (committed chunks only): {ROWS_=1000, WRITTEN=99691, SKIPPED=309, ATTEMPTS=103519, R429=1531, U503=458, RETRIES=1989, BILLED=101530, BAD=1839, PT=10809111, CT=1467192, COST=7.605343500}
|
||||
price table 'EXAMPLE-NOT-A-REAL-PRICE': input 0.50 / output 1.50 per million tokens (configurable placeholder)
|
||||
provider's own meter (survives kill -9):
|
||||
received 104,350
|
||||
rejected_429 1,543
|
||||
window_429 0
|
||||
unavailable_503 462
|
||||
billed 102,302
|
||||
prompt_tokens 10,891,290
|
||||
completion_tokens 1,478,281
|
||||
billed calls the ledger does not know about (work lost in a crash or rollback): 772
|
||||
results checksum (sha-256 over ordered rows): 98bdd29feb6d6f8d
|
||||
@@ -0,0 +1,15 @@
|
||||
# Clean run vs crashed-and-resumed run
|
||||
|
||||
## target/clean/run.log
|
||||
tickets total=100000 classified=99691 skipped=309 unaccounted=0
|
||||
ledger (committed chunks only): {ROWS_=1000, WRITTEN=99691, SKIPPED=309, ATTEMPTS=103519, R429=1531, U503=458, RETRIES=1989, BILLED=101530, BAD=1839, PT=10809111, CT=1467192, COST=7.605343500}
|
||||
billed calls the ledger does not know about (work lost in a crash or rollback): 0
|
||||
results checksum (sha-256 over ordered rows): 98bdd29feb6d6f8d
|
||||
|
||||
## target/crash/jvm2.log
|
||||
tickets total=100000 classified=99691 skipped=309 unaccounted=0
|
||||
ledger (committed chunks only): {ROWS_=1000, WRITTEN=99691, SKIPPED=309, ATTEMPTS=103519, R429=1531, U503=458, RETRIES=1989, BILLED=101530, BAD=1839, PT=10809111, CT=1467192, COST=7.605343500}
|
||||
billed calls the ledger does not know about (work lost in a crash or rollback): 772
|
||||
results checksum (sha-256 over ordered rows): 98bdd29feb6d6f8d
|
||||
|
||||
checksums identical: 98bdd29feb6d6f8d
|
||||
Reference in New Issue
Block a user