asmhatreandClaude Sonnet 5.5
c97bcf9adf
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017ic4b4VQVEJWetTT1dYzyJ
57 lines
4.3 KiB
Plaintext
57 lines
4.3 KiB
Plaintext
engine label: ollama
|
|
java: 25.0.5+8-LTS
|
|
first request (includes model load): ttft 4946 ms, 48 completion tokens
|
|
|
|
throughput, 5 prompts x 3 repeats, temperature 0, max_tokens 48
|
|
p rep ttft_ms decode_tps wall_tps prompt compl
|
|
1 1 247 17.59 16.40 41 48
|
|
1 2 66 18.43 18.32 41 48
|
|
1 3 63 18.02 17.93 41 48
|
|
2 1 289 18.99 17.35 45 48
|
|
2 2 92 18.93 18.58 45 48
|
|
2 3 65 18.56 18.43 45 48
|
|
3 1 322 18.36 16.63 43 48
|
|
3 2 58 20.94 20.79 43 48
|
|
3 3 84 17.65 17.46 43 48
|
|
4 1 287 19.89 18.07 46 48
|
|
4 2 68 19.30 19.15 46 48
|
|
4 3 70 18.57 18.44 46 48
|
|
5 1 282 17.87 16.46 45 48
|
|
5 2 90 18.76 18.44 45 48
|
|
5 3 68 18.74 18.55 45 48
|
|
|
|
median ttft_ms 84 | median decode_tps 18.57 | median wall_tps 18.32 | n=15
|
|
|
|
tool calling (one tool), 10 runs, temperature 0.7 so runs differ
|
|
question: What is the temperature in Oslo in Celsius? Use the tools.
|
|
expected: currentTemperature(Oslo), answer contains 17
|
|
run 1 ok calls=[currentTemperature(Oslo)] answer=The current temperature in Oslo is 17 degrees Celsius.
|
|
run 2 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo, Norway, is 32 degrees Celsius.
|
|
run 3 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo is 68 degrees Fahrenheit.
|
|
run 4 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius.
|
|
run 5 wrong or missing call calls=[celsiusToFahrenheit(10)] answer=The temperature in Oslo is currently 50 degrees Fahrenheit.
|
|
run 6 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo, at 20 degrees Celsius, is 68 degree
|
|
run 7 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo in Celsius is 32 degrees.
|
|
run 8 wrong or missing call calls=[celsiusToFahrenheit(4)] answer=The temperature in Oslo (Oslo) is currently 39 degrees Celsi
|
|
run 9 wrong or missing call calls=[celsiusToFahrenheit(6)] answer=The temperature in Oslo is 43 degrees Fahrenheit.
|
|
run 10 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius.
|
|
tool-call success (one tool): 1/10 (no call 0, wrong/missing call 9, wrong answer 0, errors 0)
|
|
|
|
tool calling (two tools in sequence), 10 runs, temperature 0.7 so runs differ
|
|
question: What is the temperature in Pune in Fahrenheit? Use the tools.
|
|
expected: currentTemperature(Pune), celsiusToFahrenheit(31), answer contains 88
|
|
run 1 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit.
|
|
run 2 wrong or missing call calls=[celsiusToFahrenheit(32)] answer=The temperature in Pune is currently 90 degrees Fahrenheit.
|
|
run 3 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune is 79 degrees Fahrenheit.
|
|
run 4 wrong or missing call calls=[celsiusToFahrenheit(25)] answer=The temperature in Pune is 77 degrees Fahrenheit.
|
|
run 5 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune (Pune) is 79 degrees Fahrenheit. It'
|
|
run 6 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit.
|
|
run 7 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit.
|
|
run 8 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit.
|
|
run 9 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Pune is 32 degrees Fahrenheit.
|
|
run 10 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit.
|
|
tool-call success (two tools in sequence): 0/10 (no call 0, wrong/missing call 10, wrong answer 0, errors 0)
|
|
|
|
memory (idle container memory before: 12.15MiB)
|
|
container memory during run: peak 510 MiB, median 509 MiB, 55 samples
|