Add advisors module: custom logging, PII redaction and token-budget advisors
Tests pin down chain ordering (including ties), BaseAdvisor stream behaviour, redaction order versus memory and logging, the tool loop, and how a refusal surfaces on calls, streams and over HTTP. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Ja4jkzrbQ4LQZBNrb5mkZE
This commit is contained in:
Claude
@@ -0,0 +1,8 @@
|
||||
# Advisor chain: what decides who runs first
|
||||
|
||||
registered as C300, A100, B200 (the number is HIGHEST_PRECEDENCE + n)
|
||||
call: A100> B200> C300> model C300< B200< A100<
|
||||
stream: A100> B200> C300> model C300< B200< A100<
|
||||
|
||||
plus one advisor added on the request with .advisors(...) at n=150:
|
||||
call: A100> R150> B200> C300> model C300< B200< R150< A100<
|
||||
@@ -0,0 +1,6 @@
|
||||
# Two advisors with the same order number
|
||||
|
||||
registered X, Y -> Y> X> model X< Y<
|
||||
registered Y, X -> X> Y> model Y< X<
|
||||
|
||||
For equal numbers the advisor registered LAST runs first (outermost).
|
||||
@@ -0,0 +1,16 @@
|
||||
# What is actually in the chain
|
||||
|
||||
one custom advisor at HIGHEST_PRECEDENCE + 100; the chain a call runs through:
|
||||
Dump HIGHEST_PRECEDENCE + 100
|
||||
Tool Calling Advisor HIGHEST_PRECEDENCE + 300
|
||||
call LOWEST_PRECEDENCE
|
||||
|
||||
and the chain a stream runs through:
|
||||
Dump HIGHEST_PRECEDENCE + 100
|
||||
Tool Calling Advisor HIGHEST_PRECEDENCE + 300
|
||||
stream LOWEST_PRECEDENCE
|
||||
|
||||
the same, with ToolCallingAdvisor.builder().build() added by hand:
|
||||
Dump HIGHEST_PRECEDENCE + 100
|
||||
Tool Calling Advisor HIGHEST_PRECEDENCE + 300
|
||||
call LOWEST_PRECEDENCE
|
||||
@@ -0,0 +1,11 @@
|
||||
# BaseAdvisor on a call and on a stream
|
||||
|
||||
call: before x1, after x1
|
||||
before ran on thread: the caller's thread
|
||||
|
||||
stream: the model streamed 5 chunks: one |two |thre|e fo|ur
|
||||
before x1, after x1
|
||||
after saw only: "ur"
|
||||
before ran on thread: a boundedElastic worker, not the caller's thread
|
||||
|
||||
stream whose last chunk has no finish reason: before x1, after x0
|
||||
@@ -0,0 +1,13 @@
|
||||
# LoggingAdvisor: a call and a stream
|
||||
|
||||
call, content logging off (the default):
|
||||
[LoggingAdvisor] request messages=2 roles=SU
|
||||
[LoggingAdvisor] response chars=14 tokens=12+3 took 5 ms
|
||||
|
||||
stream:
|
||||
[LoggingAdvisor] request messages=2 roles=SU
|
||||
[LoggingAdvisor] complete 3 chunks, 14 chars tokens=12+3 took 5 ms
|
||||
|
||||
call, content logging on:
|
||||
[LoggingAdvisor] request messages=1 roles=U last="Where is my refund?"
|
||||
[LoggingAdvisor] response chars=14 tokens=5+3 text="Refund issued." took 5 ms
|
||||
@@ -0,0 +1,5 @@
|
||||
# LoggingAdvisor when the model call fails
|
||||
|
||||
caller got: IllegalStateException: provider returned 503
|
||||
[LoggingAdvisor] request messages=1 roles=U
|
||||
[LoggingAdvisor] failed IllegalStateException after 5 ms
|
||||
@@ -0,0 +1,8 @@
|
||||
# PiiRedactionAdvisor on a call
|
||||
|
||||
caller sends: Hi, I'm Priya. Email [email protected] or call +91 98765 43210. Card 4111 1111 1111 1111, order 1234 5678 9012 3456, and again [email protected].
|
||||
model was sent: Hi, I'm Priya. Email <EMAIL_1> or call <PHONE_1>. Card <CARD_1>, order 1234 5678 9012 3456, and again <EMAIL_1>.
|
||||
caller receives: You said: Hi, I'm Priya. Email [email protected] or call +91 98765 43210. Card 4111 1111 1111 1111, order 1234 5678 9012 3456, and again [email protected].
|
||||
|
||||
with restore switched off, the caller receives:
|
||||
You said: Hi, I'm Priya. Email <EMAIL_1> or call <PHONE_1>. Card <CARD_1>, order 1234 5678 9012 3456, and again <EMAIL_1>.
|
||||
@@ -0,0 +1,14 @@
|
||||
# What pattern-based redaction misses
|
||||
|
||||
in: My name is Priya Sharma and I live at 14 Hill Road, Bandra, Mumbai 400050.
|
||||
out: My name is Priya Sharma and I live at 14 Hill Road, Bandra, Mumbai 400050.
|
||||
|
||||
in: Passport N1234567, PAN ABCDE1234F.
|
||||
out: Passport N1234567, PAN ABCDE1234F.
|
||||
|
||||
in: Write to priya (at) example (dot) com
|
||||
out: Write to priya (at) example (dot) com
|
||||
|
||||
in: Card 4111-1111-1111-1112 and order 1234 5678 9012 3456
|
||||
out: Card 4111-1111-1111-1112 and order 1234 5678 9012 3456
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Restoring placeholders in a stream
|
||||
|
||||
the model streams 5-character chunks: Sure,| I wi|ll wr|ite t|o <EM|AIL_1|> now|.
|
||||
|
||||
restore each chunk on its own:
|
||||
chunks: Sure,| I wi|ll wr|ite t|o <EM|AIL_1|> now|.
|
||||
joined: Sure, I will write to <EMAIL_1> now.
|
||||
|
||||
PiiRedactionAdvisor (holds back from an unfinished "<"):
|
||||
chunks: Sure,| I wi|ll wr|ite t|o |[email protected] now|.
|
||||
joined: Sure, I will write to [email protected] now.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Redaction order versus logging and memory
|
||||
|
||||
memory advisor is fixed at HIGHEST_PRECEDENCE + 200
|
||||
user says: My email is [email protected]
|
||||
|
||||
A redaction +100, logging +400 (redaction outside both):
|
||||
log line saw: "My email is <EMAIL_1>"
|
||||
memory stored: "My email is <EMAIL_1>"
|
||||
model was sent: "My email is <EMAIL_1>"
|
||||
|
||||
B redaction +100, logging +50 (logging outside redaction):
|
||||
log line saw: "My email is [email protected]"
|
||||
memory stored: "My email is <EMAIL_1>"
|
||||
model was sent: "My email is <EMAIL_1>"
|
||||
|
||||
C redaction +300 (inside the memory advisor), logging +400:
|
||||
log line saw: "My email is <EMAIL_1>"
|
||||
memory stored: "My email is [email protected]"
|
||||
model was sent: "My email is <EMAIL_1>"
|
||||
@@ -0,0 +1,18 @@
|
||||
# Placeholders across two turns with memory
|
||||
|
||||
turn 1: My email is [email protected]
|
||||
turn 2: Also cc [email protected]
|
||||
|
||||
numbering restarts on every request:
|
||||
model was sent on turn 2: U:My email is <EMAIL_1> | A:You said: My email is <EMAIL_1> | U:Also cc <EMAIL_1>
|
||||
caller receives: You said: Also cc [email protected]
|
||||
|
||||
numbering kept per conversation (the default):
|
||||
model was sent on turn 2: U:My email is <EMAIL_1> | A:You said: My email is <EMAIL_1> | U:Also cc <EMAIL_2>
|
||||
caller receives: You said: Also cc [email protected]
|
||||
|
||||
what the memory stores (placeholders, never the addresses):
|
||||
USER My email is <EMAIL_1>
|
||||
ASSISTANT You said: My email is <EMAIL_1>
|
||||
USER Also cc <EMAIL_2>
|
||||
ASSISTANT You said: Also cc <EMAIL_2>
|
||||
@@ -0,0 +1,15 @@
|
||||
# TokenBudgetAdvisor: 40 tokens per request, 60 per user
|
||||
|
||||
alice asks short questions; "spent" is the usage the model reported:
|
||||
call 1: answered, spent=13, model calls=1
|
||||
call 2: answered, spent=26, model calls=2
|
||||
call 3: answered, spent=39, model calls=3
|
||||
call 4: answered, spent=52, model calls=4
|
||||
call 5: answered, spent=65, model calls=5
|
||||
call 6: refused (user alice has used 65 of 60 tokens), spent=65, model calls=5
|
||||
|
||||
bob pastes a stack trace of 103 estimated tokens:
|
||||
refused: prompt is about 103 tokens, the limit per request is 40
|
||||
model calls: 5 (was 5), bob's spent: 0
|
||||
|
||||
bob then asks a short question: answered, bob spent=13, alice spent=65
|
||||
@@ -0,0 +1,9 @@
|
||||
# Token accounting on a stream
|
||||
|
||||
same question, same answer:
|
||||
call, usage from the response: 9 tokens
|
||||
stream, usage on the last chunk: 9 tokens
|
||||
stream, no usage reported (estimated): 9 tokens
|
||||
|
||||
A stream that reports no usage is billed by the provider all the same. The estimate above
|
||||
matches only because this scripted model and the advisor use the same tokenizer.
|
||||
@@ -0,0 +1,5 @@
|
||||
# A refusal on the stream path
|
||||
|
||||
subscriber got: TokenBudgetExceededException
|
||||
message: prompt is about 13 tokens, the limit per request is 5
|
||||
model calls: 0
|
||||
@@ -0,0 +1,13 @@
|
||||
# Where an advisor sits relative to the tool loop
|
||||
|
||||
one question, one tool call, so the model is called twice (ToolCallingAdvisor is at +300)
|
||||
|
||||
logging advisor at +250 (outside the tool loop):
|
||||
[LoggingAdvisor] request messages=1 roles=U
|
||||
[LoggingAdvisor] response chars=20 took 5 ms
|
||||
|
||||
logging advisor at +400 (inside the tool loop):
|
||||
[LoggingAdvisor] request messages=1 roles=U
|
||||
[LoggingAdvisor] response chars=0 took 5 ms
|
||||
[LoggingAdvisor] request messages=3 roles=UAT
|
||||
[LoggingAdvisor] response chars=20 took 5 ms
|
||||
@@ -0,0 +1,11 @@
|
||||
# Where a refusal surfaces
|
||||
|
||||
an advisor that throws IllegalStateException("refused"), model calls counted afterwards
|
||||
|
||||
plain advisor, call failed while running it with IllegalStateException: refused
|
||||
plain advisor, stream, throws eagerly failed while running it with IllegalStateException: refused
|
||||
plain advisor, stream, error inside Flux.defer failed while running it with IllegalStateException: refused
|
||||
BaseAdvisor.before, call failed while running it with IllegalStateException: refused
|
||||
BaseAdvisor.before, stream failed while running it with IllegalStateException: Stream processing failed (cause: IllegalStateException: refused)
|
||||
|
||||
model calls: 0
|
||||
@@ -0,0 +1,7 @@
|
||||
# Advisor context: request down, response up
|
||||
|
||||
outer: original request has tenant=null, the copy it forwards has tenant=acme
|
||||
inner: request context has tenant=acme
|
||||
outer: response context has verdict=clean, tenant=acme
|
||||
|
||||
req.context().put(...) on the incoming request: allowed
|
||||
@@ -0,0 +1,6 @@
|
||||
# Testing an advisor with a stub chain
|
||||
|
||||
what the caller sent: Mail [email protected] please
|
||||
what reached the rest: Mail <EMAIL_1> please
|
||||
what the caller got back: Noted: Mail [email protected] please
|
||||
caller's request unchanged: true
|
||||
@@ -0,0 +1,9 @@
|
||||
# Over HTTP: 200, then 429
|
||||
|
||||
limits for this run: 60 tokens per request, 30 per user
|
||||
dana request 1 -> HTTP 200, model calls so far: 1
|
||||
dana request 2 -> HTTP 200, model calls so far: 2
|
||||
dana request 3 -> HTTP 429, model calls so far: 2
|
||||
dana request 4 -> HTTP 429, model calls so far: 2
|
||||
|
||||
erin pastes a long stack trace -> HTTP 429, model calls: 2 (was 2)
|
||||
@@ -0,0 +1,8 @@
|
||||
# Built-in advisor orders in 2.0.1
|
||||
|
||||
SimpleLoggerAdvisor getOrder() = 0
|
||||
SafeGuardAdvisor getOrder() = 0
|
||||
MessageChatMemoryAdvisor getOrder() = -2147483448 (HIGHEST_PRECEDENCE + 200)
|
||||
Tool Calling Advisor getOrder() = -2147483348 (HIGHEST_PRECEDENCE + 300)
|
||||
|
||||
Advisor.DEFAULT_CHAT_MEMORY_PRECEDENCE_ORDER = HIGHEST_PRECEDENCE + 200
|
||||
@@ -0,0 +1,7 @@
|
||||
# Token budget and the tool loop
|
||||
|
||||
the model reports 100+10 tokens for round 1 (asks for the tool) and 130+20 for round 2 (answers)
|
||||
so the provider would bill 260 tokens for this one question
|
||||
|
||||
budget advisor at +250 (outside the tool loop): recorded 260
|
||||
budget advisor at +400 (inside the tool loop): recorded 260
|
||||
Reference in New Issue
Block a user