asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add audio module: meeting minutes with Spring AI speech-to-text, structured output and text-to-speech

Browse Source 
OpenAiAudioTranscriptionModel -> ChatClient.entity(Minutes.class) -> OpenAiAudioSpeechModel,
WAV chunking at a silence under the 25 MB upload limit, SDK retry behaviour, the 4096-character
speech limit and a cost model. Audio endpoints are a local mock and the chat model a stub; no real
model was called. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CifR2BE7PBvRy5hT8TS9oz
This commit is contained in:
asmhatre
2026-10-26 02:00:28 +00:00
co-authored by Claude Sonnet 5.5
parent ff58ff0189
commit 279be7eb9f
36 changed files with 1576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audio/.gitignore
+1
View File
@@ -0,0 +1 @@
target/
audio/README.md
+37
View File
@@ -0,0 +1,37 @@
# audio
A meeting-minutes generator on Spring AI 2.0.2: a recording goes through `OpenAiAudioTranscriptionModel`, the transcript becomes a typed `Minutes` record with `ChatClient.entity(...)`, and the action items are read back with `OpenAiAudioSpeechModel`. Files over the upload limit are split in the middle of a silence; a cost model with formulas prices a meeting.
**No real model was called.** The two audio endpoints (`/v1/audio/transcriptions`, `/v1/audio/speech`) are a local **mock** (`src/test/.../support/FakeAudioServer.java`) that the real Spring AI classes and the OpenAI Java SDK talk to over HTTP, so the requests in `output/` are what those classes really send. The "recording" is 30 synthetic tones, one per scripted sentence, and the mock's "transcription" decodes the pitch back into the sentence. The chat model is a scripted **stub**. Nothing here measures the accuracy, latency or price of any real model; the cost table applies OpenAI's published prices (read on 2026-10-26) to stated assumptions.
| | |
|---|---|
| Spring Boot | 4.1.2 |
| Spring AI | 2.0.2 (`spring-ai-openai`, `spring-ai-client-chat`) |
| Java | 25 |
| Run | `scripts/run-all.sh` (about 45 s) |
## Source
| File | What it is |
|---|---|
| [`MinutesService`](src/main/java/com/ankurm/audio/MinutesService.java) | transcribe (chunked) -> `summarise` -> `readAloud` (split at 4096 characters) |
| [`SilenceChunker`](src/main/java/com/ankurm/audio/SilenceChunker.java) | splits a WAV under a byte budget, cutting in the middle of the longest silence it finds; also the blind baseline |
| [`Minutes`](src/main/java/com/ankurm/audio/Minutes.java) | the record the model's reply is parsed into |
| [`NamedBytes`](src/main/java/com/ankurm/audio/NamedBytes.java) | a `ByteArrayResource` with a file name (the extension is how the upload is typed) |
| [`CostModel`](src/main/java/com/ankurm/audio/CostModel.java) | the three cost formulas and the prices they were run with |
| [`Wav`](src/main/java/com/ankurm/audio/Wav.java) | the 60 lines of RIFF/WAVE the chunker needs |
## Captured output (`output/`)
| File | Shows |
|---|---|
| `01-api-shape.txt` | class names and defaults in the 2.0.2 jars |
| `02-transcribe-small.txt`, `02b-filename.txt` | the multipart request; file name and extension |
| `03-large-file.txt` | 413 on a 30 MB file; two silence-aligned chunks recover every line |
| `04-blind-split.txt` | cutting every N bytes lands inside a sentence |
| `05-structured-output.txt`, `05b-bad-replies.txt` | the schema appended to the prompt; three replies that fail to parse |
| `06-speech.txt`, `06b-speech-split.txt` | the speech request; the 4096-character wall; splitting |
| `07-retries.txt` | SDK retries re-upload the whole file |
| `08-cost.txt` | cost per meeting length; file size per encoding |
| `09-end-to-end.txt` | the three steps wired together |
audio/output/01-api-shape.txt
+15
View File
@@ -0,0 +1,15 @@
# Spring AI 2.0.2 audio classes and defaults (read from the jars)
speech-to-text interface : org.springframework.ai.audio.transcription.TranscriptionModel
speech-to-text OpenAI : org.springframework.ai.openai.OpenAiAudioTranscriptionModel
text-to-speech interface : org.springframework.ai.audio.tts.TextToSpeechModel
text-to-speech OpenAI : org.springframework.ai.openai.OpenAiAudioSpeechModel
transcription DEFAULT_TRANSCRIPTION_MODEL = whisper-1
transcription DEFAULT_RESPONSE_FORMAT = text
speech DEFAULT_SPEECH_MODEL = gpt-4o-mini-tts
speech DEFAULT_VOICE = alloy
speech DEFAULT_RESPONSE_FORMAT = mp3
speech DEFAULT_SPEED = 1.0
both DEFAULT_TIMEOUT = PT1M
both DEFAULT_MAX_RETRIES = 3
audio/output/02-transcribe-small.txt
+18
View File
@@ -0,0 +1,18 @@
# One 60-second clip (1920044 bytes WAV) through OpenAiAudioTranscriptionModel -- MOCK server
model option: whisper-1
request : POST /v1/audio/transcriptions (multipart, 1920524 bytes)
fields : {model=whisper-1, response_format=text}
file part : name=standup.wav type=application/octet-stream bytes=1920044
transcribe(Resource) -> "Okay, let's get started with the Falcon weekly sync. First item is the release date for version two point four."
with language=en and prompt, fields now: {model=whisper-1, language=en, prompt=Falcon weekly sync, response_format=text}
response metadata class: OpenAiAudioTranscriptionResponseMetadata
model option: gpt-transcribe
request : POST /v1/audio/transcriptions (multipart, 1920529 bytes)
fields : {model=gpt-transcribe, response_format=text}
file part : name=standup.wav type=application/octet-stream bytes=1920044
transcribe(Resource) -> "Okay, let's get started with the Falcon weekly sync. First item is the release date for version two point four."
with language=en and prompt, fields now: {model=gpt-transcribe, language=en, prompt=Falcon weekly sync, response_format=text}
response metadata class: OpenAiAudioTranscriptionResponseMetadata
audio/output/02b-filename.txt
+5
View File
@@ -0,0 +1,5 @@
# The upload's file name decides whether it is accepted -- MOCK server
file name standup.wav -> ok: 52 chars
file name standup.flac -> BadRequestException: 400: mock: Invalid file format. Supported formats: [mp3, mp4, mpeg, mpga, m4a, wav, webm]
file name standup -> BadRequestException: 400: mock: Invalid file format. Supported formats: [mp3, mp4, mpeg, mpga, m4a, wav, webm]
audio/output/03-large-file.txt
+14
View File
@@ -0,0 +1,14 @@
# A 16-minute recording, 30,278,444 bytes -- MOCK server
1) the whole file in one request (mock limit 26,214,400 bytes):
exception: com.openai.errors.UnexpectedStatusCodeException
message : 413: mock: Maximum content size limit (26214400) exceeded (30278929 bytes read)
2) MinutesService.transcribe: budget 24,000,000 bytes per chunk, silence of at least 2.0 s, RMS below 500
chunk 0: 0.0 s -> 733.5 s 23,472,022 bytes cut: mid-silence
chunk 1: 733.5 s -> 946.2 s 6,806,466 bytes cut: end of file
requests the mock saw: 2
- file=part-00-meeting.wav bytes=23,472,022 prompt=(none)
- file=part-01-meeting.wav bytes=6,806,466 prompt="...the on call rotation changes next month."
script lines recovered: 30 of 30, in order: true
[inaudible] markers: 0
audio/output/04-blind-split.txt
+8
View File
@@ -0,0 +1,8 @@
# Cutting every 24,000,000 bytes, wherever that lands -- MOCK server
chunk 0: 0.0 s -> 750.0 s 24,000,000 bytes boundary falls INSIDE sentence 23 ("Everyone should check the new rotation sheet before the end of this week.")
chunk 1: 750.0 s -> 946.2 s 6,278,488 bytes boundary falls at end of file
script lines recovered: 29 of 30; [inaudible] markers: 2
(mock rule: a tone cut to under 60% of its length is reported as [inaudible]; a real model would not fail this
cleanly -- OpenAI's guide only says splitting mid-sentence can remove context and reduce accuracy.)
audio/output/05-structured-output.txt
+58
View File
@@ -0,0 +1,58 @@
# entity(Minutes.class) against a STUB ChatModel
messages sent to the model:
[SYSTEM] 222 chars
[USER] 2799 chars
tail of the USER message (the format instructions Spring AI appended):
Your response should be in JSON format.
Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
Do not include markdown code blocks in your response.
Remove the ```json markdown from the output.
Here is the JSON Schema instance your output must adhere to:
```{
"$schema" : "https://json-schema.org/draft/2020-12/schema",
"type" : "object",
"properties" : {
"actionItems" : {
"type" : "array",
"items" : {
"type" : "object",
"properties" : {
"due" : {
"type" : "string"
},
"owner" : {
"type" : "string"
},
"task" : {
"type" : "string"
}
},
"required" : [ "due", "owner", "task" ],
"additionalProperties" : false
}
},
"decisions" : {
"type" : "array",
"items" : {
"type" : "string"
}
},
"title" : {
"type" : "string"
}
},
"required" : [ "actionItems", "decisions", "title" ],
"additionalProperties" : false
}```
parsed: Falcon weekly sync
decisions : 2
action item : Rahul | Fix the login timeout bug | due: before the release on the fourteenth of November
action item : Priya | Send the revised budget to finance | due: Friday
action item : Sana | Open tickets for both export endpoint issues | due: today
spoken form:
Falcon weekly sync. 2 decisions. Ship version 2.4 on the fourteenth of November. The data pipeline migrates to the new cluster next sprint. 3 action items. Rahul: Fix the login timeout bug, due before the release on the fourteenth of November. Priya: Send the revised budget to finance, due Friday. Sana: Open tickets for both export endpoint issues, due today.
audio/output/05b-bad-replies.txt
+5
View File
@@ -0,0 +1,5 @@
# Three replies a model can give instead of clean JSON (STUB model)
wrapped in a markdown fence with a sentence before it -> StreamReadException: Unrecognized token 'Here': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false...
truncated: closing brace missing -> UnexpectedEndOfInputException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: REDACTED (`StreamReadFeatu...
prose instead of JSON -> StreamReadException: Unrecognized token 'I': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') ...
audio/output/06-speech.txt
+11
View File
@@ -0,0 +1,11 @@
# OpenAiAudioSpeechModel -- MOCK server
speech.call(String) with every default:
request json : {"input":"Three action items.","model":"gpt-4o-mini-tts","voice":"alloy","response_format":"mp3","speed":1.0}
response : 20310 bytes
with TextToSpeechPrompt and options:
request json : {"input":"Rahul: fix the login timeout bug.","model":"gpt-4o-mini-tts","voice":"coral","instructions":"Calm, like a project manager reading notes.","response_format":"wav","speed":1.1}
response : 35244 bytes, starts with RIFF
input of 5000 characters -> BadRequestException: 400: mock: the maximum length of input is 4096 characters, got 5000
audio/output/06b-speech-split.txt
+6
View File
@@ -0,0 +1,6 @@
# Splitting a long read-back at sentence ends
text: 11051 characters, 120 sentences
part 0: 4038 characters, ends with " take space."
part 1: 4047 characters, ends with " take space."
part 2: 2964 characters, ends with " take space."
audio/output/07-retries.txt
+6
View File
@@ -0,0 +1,6 @@
# A 120-second clip (3840044 bytes) against a flaky endpoint -- MOCK server
maxRetries=0, server fails the first 2 requests with 429 -> RateLimitException requests: 1, bytes uploaded: 3,840,523
maxRetries=0, server fails the first 2 requests with 400 -> BadRequestException requests: 1, bytes uploaded: 3,840,523
maxRetries=3, server fails the first 2 requests with 429 -> ok requests: 3, bytes uploaded: 11,521,569
maxRetries=3, server fails the first 2 requests with 400 -> BadRequestException requests: 1, bytes uploaded: 3,840,523
audio/output/08-cost.txt
+14
View File
@@ -0,0 +1,14 @@
# Cost of one meeting, USD (prices of 2026-10-26; word and token counts are ASSUMPTIONS)
assumptions: 150 words/minute spoken, 0.75 words/token, summary output 600 tokens, read-back 120 words
prices : gpt-transcribe $0.0045/min | whisper-1 $0.006/min | gpt-4o-mini $0.15 in / $0.60 out per 1M | tts-1 $15 per 1M chars
minutes words gpt-transc whisper-1 summary tts-1 total(gt)
15 2,250 0.0675 0.0900 0.0008 0.0108 0.0791
30 4,500 0.1350 0.1800 0.0013 0.0108 0.1471
60 9,000 0.2700 0.3600 0.0022 0.0108 0.2830
120 18,000 0.5400 0.7200 0.0040 0.0108 0.5548
transcription is billed on audio duration, not file size: a 60-minute recording costs $0.2700 whatever its encoding.
size of 60 minutes: 16 kHz mono 16-bit WAV = 115.2 MB | 64 kbit/s mp3 = 28.8 MB | 32 kbit/s mp3 = 14.4 MB
minutes that fit in 25,000,000 bytes: WAV 13.0 | 64 kbit/s 52 | 32 kbit/s 104
audio/output/09-end-to-end.txt
+7
View File
@@ -0,0 +1,7 @@
# Recording -> transcript -> Minutes -> audio (MOCK audio server, STUB chat model)
1) transcribe: 30,278,444 bytes -> 2 request(s) -> 1701 characters
2) summarise : transcript of 292 words reached the chat model as the user message: true
2 action items: [ActionItem[owner=Rahul, task=Fix the login timeout bug, due=before the release], ActionItem[owner=Priya, task=Send the revised budget to finance, due=Friday]]
3) read back : 1 clip(s), 221910 bytes, spoken text: Falcon weekly sync. 1 decision. Ship version 2.4 on the fourteenth of November. 2 action items. Rahul: Fix the login timeout bug, due before the release. Priya: Send the revised budget to finance, due Friday.
speech request: {"input":"Falcon weekly sync. 1 decision. Ship version 2.4 on the fourteenth of November. 2 action items. Rahul: Fix the login timeout bug, due before the release. Priya: Send the revised budget to finance, due Friday.","model":"gpt-4o-mini-tts","voice":"coral","response_format":"wav","speed":1.0}
audio/pom.xml
+59
View File
@@ -0,0 +1,59 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.2</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>audio</artifactId>
<version>1.0.0</version>
<name>audio</name>
<description>Meeting-minutes generator on Spring AI audio: OpenAiAudioTranscriptionModel, a structured-output ChatClient for action items, OpenAiAudioSpeechModel for read-back, WAV chunking for the upload limit and a cost model. Run against a local mock server.</description>
<properties>
<java.version>25</java.version>
<spring-ai.version>2.0.2</spring-ai.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bom</artifactId>
<version>${spring-ai.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-client-chat</artifactId></dependency>
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-openai</artifactId></dependency>
<dependency><groupId>tools.jackson.core</groupId><artifactId>jackson-databind</artifactId></dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8</argLine>
</configuration>
</plugin>
</plugins>
</build>
</project>
audio/scripts/run-all.sh
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
# Regenerates every file under output/ from the test suite. No API key, no Docker, no network
# beyond Maven Central: the audio endpoints are a local mock and the chat model is a stub.
set -euo pipefail
cd "$(dirname "$0")/.."
rm -rf target output
mvn -q -B test 2>&1 | grep -E "Tests run:|BUILD|FAIL|ERROR" || true
ls output
audio/src/main/java/com/ankurm/audio/CostModel.java
+50
View File
@@ -0,0 +1,50 @@
package com.ankurm.audio;
/**
* The cost of one meeting as three line items. Prices are inputs, not constants of nature: the
* defaults in {@link Prices#OPENAI_2026_10_26} were read off OpenAI's pricing and model pages on
* 2026-10-26 and will drift.
*
* <pre>
* transcription = minutes x pricePerMinute (audio length, not file size)
* summary = tokensIn x in/1e6 + tokensOut x out/1e6
* read-back = characters x pricePerMillionChars/1e6 (tts-1 family, billed per character)
* | speechTokens x audioOut/1e6 + textTokens x textIn/1e6 (gpt-4o-mini-tts, billed per token)
* </pre>
*/
public final class CostModel {
/** Dollar prices. {@code ttsPerMillionChars} applies to tts-1 / tts-1-hd; the *Token* fields to gpt-4o-mini-tts. */
public record Prices(double transcribePerMinute, double chatInPerMillion, double chatOutPerMillion,
double ttsPerMillionChars, double miniTtsTextInPerMillion, double miniTtsAudioOutPerMillion) {
/** gpt-transcribe, gpt-4o-mini (chat), tts-1, gpt-4o-mini-tts as listed on 2026-10-26. */
public static final Prices OPENAI_2026_10_26 = new Prices(0.0045, 0.15, 0.60, 15.0, 0.60, 12.0);
/** whisper-1 for the transcription line, everything else as above. */
public static final Prices WHISPER_1 = new Prices(0.006, 0.15, 0.60, 15.0, 0.60, 12.0);
}
/** Rule-of-thumb token estimate: about 0.75 English words per token. An assumption, not a measurement. */
public static final double WORDS_PER_TOKEN = 0.75;
/** Spoken English runs at roughly 150 words per minute. Also an assumption. */
public static final double WORDS_PER_MINUTE = 150;
private CostModel() {}
public static double transcription(double minutes, Prices p) {
return minutes * p.transcribePerMinute();
}
public static double summary(double transcriptWords, double outputTokens, Prices p) {
double tokensIn = transcriptWords / WORDS_PER_TOKEN;
return tokensIn * p.chatInPerMillion() / 1e6 + outputTokens * p.chatOutPerMillion() / 1e6;
}
public static double readBackPerChar(int characters, Prices p) {
return characters * p.ttsPerMillionChars() / 1e6;
}
public static double readBackPerToken(double textTokens, double audioTokens, Prices p) {
return textTokens * p.miniTtsTextInPerMillion() / 1e6 + audioTokens * p.miniTtsAudioOutPerMillion() / 1e6;
}
}
audio/src/main/java/com/ankurm/audio/Minutes.java
+29
View File
@@ -0,0 +1,29 @@
package com.ankurm.audio;
import java.util.List;
/**
* What we want back from the model. Spring AI turns this record into a JSON Schema, appends it to
* the prompt, and parses the reply into it ({@code ChatClient...entity(Minutes.class)}).
*/
public record Minutes(String title, List<String> decisions, List<ActionItem> actionItems) {
/** {@code due} is free text ("Friday", "14 November") because a meeting rarely says an ISO date. */
public record ActionItem(String owner, String task, String due) {}
/** The text we hand to text-to-speech: short, spoken-friendly, no markup. */
public String toSpokenText() {
StringBuilder sb = new StringBuilder(title).append(". ");
if (!decisions.isEmpty()) {
sb.append(decisions.size()).append(decisions.size() == 1 ? " decision. " : " decisions. ");
decisions.forEach(d -> sb.append(d.endsWith(".") ? d : d + ".").append(' '));
}
sb.append(actionItems.size()).append(actionItems.size() == 1 ? " action item. " : " action items. ");
for (ActionItem a : actionItems) {
sb.append(a.owner()).append(": ").append(a.task());
if (a.due() != null && !a.due().isBlank()) sb.append(", due ").append(a.due());
sb.append(". ");
}
return sb.toString().trim();
}
}
audio/src/main/java/com/ankurm/audio/MinutesService.java
+101
View File
@@ -0,0 +1,101 @@
package com.ankurm.audio;
import java.util.ArrayList;
import java.util.List;
import org.springframework.ai.audio.transcription.AudioTranscriptionPrompt;
import org.springframework.ai.audio.transcription.TranscriptionModel;
import org.springframework.ai.audio.tts.TextToSpeechModel;
import org.springframework.ai.audio.tts.TextToSpeechPrompt;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.openai.OpenAiAudioSpeechOptions;
import org.springframework.ai.openai.OpenAiAudioTranscriptionOptions;
/**
* Audio in, minutes out, audio back: transcribe (chunking if the file is too big), summarise into a
* typed {@link Minutes}, read it aloud. Each step takes the Spring AI interface, not the OpenAI
* class, so a different provider's TranscriptionModel / TextToSpeechModel drops in.
*/
public class MinutesService {
/** OpenAI's documented ceiling is "25 MB" without saying MB or MiB, so stay under 25,000,000 bytes. */
public static final long UPLOAD_BUDGET_BYTES = 24_000_000L;
/** The speech endpoint's documented maximum input length, in characters. */
public static final int TTS_MAX_CHARS = 4096;
private static final String SYSTEM = """
You write meeting minutes from a raw transcript.
Only include action items that someone explicitly agreed to do.
Use the speaker's own words for the task. If no due date was said, use an empty string.
Do not invent owners.""";
private final TranscriptionModel transcription;
private final ChatClient chat;
private final TextToSpeechModel speech;
private final SilenceChunker chunker;
private final String transcriptionModelName;
private final String speechModelName;
public MinutesService(TranscriptionModel transcription, ChatClient chat, TextToSpeechModel speech,
String transcriptionModelName, String speechModelName) {
this.transcription = transcription;
this.chat = chat;
this.speech = speech;
this.transcriptionModelName = transcriptionModelName;
this.speechModelName = speechModelName;
this.chunker = new SilenceChunker(UPLOAD_BUDGET_BYTES, 2.0, 500);
}
/** Records what happened, for the transcripts and the cost model. */
public record Transcription(String text, List<SilenceChunker.Chunk> chunks, List<String> pieces) {}
/** Transcribe a WAV of any size. Chunks run in order because each carries the tail of the one before as context. */
public Transcription transcribe(byte[] wav, String filename) {
List<SilenceChunker.Chunk> chunks = chunker.split(wav);
List<String> pieces = new ArrayList<>();
for (SilenceChunker.Chunk c : chunks) {
String previousTail = pieces.isEmpty() ? null : tail(pieces.getLast(), 200);
OpenAiAudioTranscriptionOptions.Builder opts = OpenAiAudioTranscriptionOptions.builder().model(transcriptionModelName);
if (previousTail != null) opts.prompt(previousTail);
String name = chunks.size() == 1 ? filename : "part-%02d-%s".formatted(c.index(), filename);
pieces.add(transcription.call(new AudioTranscriptionPrompt(new NamedBytes(c.wav(), name), opts.build()))
.getResult().getOutput());
}
return new Transcription(String.join(" ", pieces), chunks, pieces);
}
public Minutes summarise(String transcript) {
// transcript goes in as the user message itself, not through a {placeholder}: a transcript with
// braces in it would otherwise be read as template variables.
return chat.prompt().system(SYSTEM).user(transcript).call().entity(Minutes.class);
}
/** One audio clip per piece of at most {@link #TTS_MAX_CHARS} characters, cut on sentence ends. */
public List<byte[]> readAloud(Minutes minutes, String voice) {
List<byte[]> clips = new ArrayList<>();
for (String part : splitForSpeech(minutes.toSpokenText(), TTS_MAX_CHARS)) {
OpenAiAudioSpeechOptions opts = OpenAiAudioSpeechOptions.builder()
.model(speechModelName).voice(voice).responseFormat("wav").build();
clips.add(speech.call(new TextToSpeechPrompt(part, opts)).getResult().getOutput());
}
return clips;
}
static List<String> splitForSpeech(String text, int max) {
List<String> parts = new ArrayList<>();
StringBuilder cur = new StringBuilder();
for (String sentence : text.split("(?<=[.!?])\\s+")) {
if (cur.length() + sentence.length() + 1 > max && !cur.isEmpty()) {
parts.add(cur.toString().trim());
cur.setLength(0);
}
cur.append(sentence).append(' ');
}
if (!cur.isEmpty()) parts.add(cur.toString().trim());
return parts;
}
private static String tail(String s, int n) {
return s.length() <= n ? s : s.substring(s.length() - n);
}
}
audio/src/main/java/com/ankurm/audio/NamedBytes.java
+23
View File
@@ -0,0 +1,23 @@
package com.ankurm.audio;
import org.springframework.core.io.ByteArrayResource;
/**
* A {@link ByteArrayResource} that reports a file name. The name matters: the transcription
* endpoint decides the audio format from the extension on the multipart upload, and a bare
* ByteArrayResource has none.
*/
public class NamedBytes extends ByteArrayResource {
private final String filename;
public NamedBytes(byte[] bytes, String filename) {
super(bytes);
this.filename = filename;
}
@Override
public String getFilename() {
return filename;
}
}
audio/src/main/java/com/ankurm/audio/SilenceChunker.java
+98
View File
@@ -0,0 +1,98 @@
package com.ankurm.audio;
import java.util.ArrayList;
import java.util.List;
/**
* Splits a WAV file into pieces that each fit under an upload limit, cutting in the middle of the
* longest silence it can find instead of at an arbitrary byte. The OpenAI speech-to-text guide
* says to avoid "splitting in the middle of a sentence, which can remove context and reduce
* accuracy"; this is that advice as code.
*/
public final class SilenceChunker {
/** One piece: where it starts in the original recording, and its own valid WAV bytes. */
public record Chunk(int index, double startSeconds, double endSeconds, byte[] wav, String cutKind) {
public int bytes() { return wav.length; }
}
private static final int WINDOW_MS = 100;
private final long maxBytes;
private final double minSilenceSeconds;
private final int silenceRms;
/**
* @param maxBytes hard ceiling for each chunk, header included
* @param minSilenceSeconds shortest quiet stretch worth cutting in
* @param silenceRms windows whose RMS is below this count as quiet (16-bit scale)
*/
public SilenceChunker(long maxBytes, double minSilenceSeconds, int silenceRms) {
this.maxBytes = maxBytes;
this.minSilenceSeconds = minSilenceSeconds;
this.silenceRms = silenceRms;
}
public List<Chunk> split(byte[] wav) {
Wav.Info info = Wav.parse(wav);
if (wav.length <= maxBytes) {
return List.of(new Chunk(0, 0, info.seconds(), wav, "none (fits)"));
}
int maxFrames = (int) ((maxBytes - 44) / info.bytesPerFrame());
int win = info.sampleRate() * WINDOW_MS / 1000;
int total = info.frames();
List<Chunk> out = new ArrayList<>();
int start = 0;
while (start < total) {
int limit = Math.min(total, start + maxFrames);
if (limit == total) {
out.add(chunk(out.size(), wav, info, start, total, "end of file"));
break;
}
int cut = quietCut(wav, info, win, start + maxFrames / 2, limit);
String kind = "mid-silence";
if (cut < 0) { cut = limit; kind = "HARD CUT (no silence found)"; }
out.add(chunk(out.size(), wav, info, start, cut, kind));
start = cut;
}
return out;
}
/** Naive baseline: every {@code maxBytes}, wherever that lands. */
public List<Chunk> splitBlindly(byte[] wav) {
Wav.Info info = Wav.parse(wav);
int maxFrames = (int) ((maxBytes - 44) / info.bytesPerFrame());
List<Chunk> out = new ArrayList<>();
for (int s = 0; s < info.frames(); s += maxFrames) {
out.add(chunk(out.size(), wav, info, s, Math.min(info.frames(), s + maxFrames), "fixed size"));
}
return out;
}
private Chunk chunk(int i, byte[] wav, Wav.Info info, int from, int to, String kind) {
return new Chunk(i, (double) from / info.sampleRate(), (double) to / info.sampleRate(),
Wav.slice(wav, info, from, to), kind);
}
/** Middle frame of the LAST quiet run (at least minSilenceSeconds long) inside [from, limit), or -1. */
private int quietCut(byte[] wav, Wav.Info info, int win, int from, int limit) {
int minWindows = (int) Math.ceil(minSilenceSeconds * 1000 / WINDOW_MS);
int runStart = -1, bestMid = -1;
for (int f = from; f + win <= limit; f += win) {
boolean quiet = rms(wav, info, f, win) < silenceRms;
if (quiet && runStart < 0) runStart = f;
if ((!quiet || f + 2 * win > limit) && runStart >= 0) {
int end = quiet ? f + win : f;
if ((end - runStart) / win >= minWindows) bestMid = runStart + (end - runStart) / 2;
runStart = -1;
}
}
return bestMid;
}
private static double rms(byte[] wav, Wav.Info info, int from, int n) {
double sum = 0;
for (int i = 0; i < n; i++) { double s = Wav.sample(wav, info, from + i); sum += s * s; }
return Math.sqrt(sum / n);
}
}
audio/src/main/java/com/ankurm/audio/Wav.java
+68
View File
@@ -0,0 +1,68 @@
package com.ankurm.audio;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Arrays;
/**
* Just enough RIFF/WAVE to read and write 16-bit PCM. Production code would hand this to ffmpeg
* or a library; it is here so the chunking logic is visible and testable with no native binary.
*/
public final class Wav {
/** Format and position of the PCM payload inside a WAV file. */
public record Info(int sampleRate, int channels, int bitsPerSample, int dataOffset, int dataLength) {
public int bytesPerFrame() { return channels * bitsPerSample / 8; }
public int frames() { return dataLength / bytesPerFrame(); }
public double seconds() { return (double) frames() / sampleRate; }
}
private Wav() {}
public static Info parse(byte[] wav) {
ByteBuffer b = ByteBuffer.wrap(wav).order(ByteOrder.LITTLE_ENDIAN);
if (wav.length < 44 || b.getInt(0) != 0x46464952 /* RIFF */ || b.getInt(8) != 0x45564157 /* WAVE */) {
throw new IllegalArgumentException("not a RIFF/WAVE file");
}
int pos = 12, rate = 0, channels = 0, bits = 0;
while (pos + 8 <= wav.length) {
int id = b.getInt(pos);
int len = b.getInt(pos + 4);
if (id == 0x20746d66 /* "fmt " */) {
channels = b.getShort(pos + 10);
rate = b.getInt(pos + 12);
bits = b.getShort(pos + 22);
} else if (id == 0x61746164 /* "data" */) {
int available = Math.min(len < 0 ? Integer.MAX_VALUE : len, wav.length - (pos + 8));
if (bits != 16) throw new IllegalArgumentException("only 16-bit PCM is supported, got " + bits);
return new Info(rate, channels, bits, pos + 8, available);
}
pos += 8 + len + (len & 1);
}
throw new IllegalArgumentException("no data chunk");
}
/** Wraps raw PCM in a canonical 44-byte header. */
public static byte[] wrap(byte[] pcm, int offset, int length, int sampleRate, int channels, int bits) {
ByteBuffer b = ByteBuffer.allocate(44 + length).order(ByteOrder.LITTLE_ENDIAN);
b.putInt(0x46464952).putInt(36 + length).putInt(0x45564157);
b.putInt(0x20746d66).putInt(16).putShort((short) 1).putShort((short) channels).putInt(sampleRate)
.putInt(sampleRate * channels * bits / 8).putShort((short) (channels * bits / 8)).putShort((short) bits);
b.putInt(0x61746164).putInt(length);
b.put(pcm, offset, length);
return b.array();
}
/** A new, valid WAV file holding frames [fromFrame, toFrame) of {@code wav}. */
public static byte[] slice(byte[] wav, Info info, int fromFrame, int toFrame) {
int from = info.dataOffset() + fromFrame * info.bytesPerFrame();
int len = (toFrame - fromFrame) * info.bytesPerFrame();
return wrap(Arrays.copyOfRange(wav, from, from + len), 0, len, info.sampleRate(), info.channels(), info.bitsPerSample());
}
/** Sample {@code frame} of channel 0 as a signed 16-bit value. */
public static short sample(byte[] wav, Info info, int frame) {
int p = info.dataOffset() + frame * info.bytesPerFrame();
return (short) ((wav[p] & 0xff) | (wav[p + 1] << 8));
}
}
audio/src/test/java/com/ankurm/audio/ApiShapeTest.java
+37
View File
@@ -0,0 +1,37 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.audio.transcription.TranscriptionModel;
import org.springframework.ai.audio.tts.TextToSpeechModel;
import org.springframework.ai.openai.AbstractOpenAiOptions;
import org.springframework.ai.openai.OpenAiAudioSpeechModel;
import org.springframework.ai.openai.OpenAiAudioSpeechOptions;
import org.springframework.ai.openai.OpenAiAudioTranscriptionModel;
import org.springframework.ai.openai.OpenAiAudioTranscriptionOptions;
/** What the shipped 2.0.2 classes are called and what their defaults are. Nothing is sent anywhere. */
class ApiShapeTest {
@Test
void namesAndDefaults() {
try (Transcript t = new Transcript("01-api-shape.txt", "Spring AI 2.0.2 audio classes and defaults (read from the jars)")) {
t.line("speech-to-text interface : %s", TranscriptionModel.class.getName());
t.line("speech-to-text OpenAI : %s", OpenAiAudioTranscriptionModel.class.getName());
t.line("text-to-speech interface : %s", TextToSpeechModel.class.getName());
t.line("text-to-speech OpenAI : %s", OpenAiAudioSpeechModel.class.getName());
t.blank();
t.line("transcription DEFAULT_TRANSCRIPTION_MODEL = %s", OpenAiAudioTranscriptionOptions.DEFAULT_TRANSCRIPTION_MODEL);
t.line("transcription DEFAULT_RESPONSE_FORMAT = %s", OpenAiAudioTranscriptionOptions.DEFAULT_RESPONSE_FORMAT);
t.line("speech DEFAULT_SPEECH_MODEL = %s", OpenAiAudioSpeechOptions.DEFAULT_SPEECH_MODEL);
t.line("speech DEFAULT_VOICE = %s", OpenAiAudioSpeechOptions.DEFAULT_VOICE);
t.line("speech DEFAULT_RESPONSE_FORMAT = %s", OpenAiAudioSpeechOptions.DEFAULT_RESPONSE_FORMAT);
t.line("speech DEFAULT_SPEED = %s", OpenAiAudioSpeechOptions.DEFAULT_SPEED);
t.line("both DEFAULT_TIMEOUT = %s", AbstractOpenAiOptions.DEFAULT_TIMEOUT);
t.line("both DEFAULT_MAX_RETRIES = %d", AbstractOpenAiOptions.DEFAULT_MAX_RETRIES);
}
assertThat(OpenAiAudioTranscriptionOptions.DEFAULT_TRANSCRIPTION_MODEL).isNotBlank();
}
}
audio/src/test/java/com/ankurm/audio/CostTest.java
+37
View File
@@ -0,0 +1,37 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.CostModel.Prices;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
/** The cost formulas applied to meetings of different length. Prices are inputs read from OpenAI's pages on 2026-10-26. */
class CostTest {
@Test
void perMeeting() {
try (Transcript t = new Transcript("08-cost.txt", "Cost of one meeting, USD (prices of 2026-10-26; word and token counts are ASSUMPTIONS)")) {
t.line("assumptions: %.0f words/minute spoken, %.2f words/token, summary output 600 tokens, read-back 120 words", CostModel.WORDS_PER_MINUTE, CostModel.WORDS_PER_TOKEN);
t.line("prices : gpt-transcribe $0.0045/min | whisper-1 $0.006/min | gpt-4o-mini $0.15 in / $0.60 out per 1M | tts-1 $15 per 1M chars");
t.blank();
t.line("%8s %10s %10s %10s %10s %10s %10s", "minutes", "words", "gpt-transc", "whisper-1", "summary", "tts-1", "total(gt)");
for (int minutes : new int[] { 15, 30, 60, 120 }) {
double words = minutes * CostModel.WORDS_PER_MINUTE;
double a = CostModel.transcription(minutes, Prices.OPENAI_2026_10_26);
double w = CostModel.transcription(minutes, Prices.WHISPER_1);
double s = CostModel.summary(words, 600, Prices.OPENAI_2026_10_26);
double r = CostModel.readBackPerChar((int) (120 * 6.0), Prices.OPENAI_2026_10_26);
t.line("%8d %,10.0f %10.4f %10.4f %10.4f %10.4f %10.4f", minutes, words, a, w, s, r, a + s + r);
}
t.blank();
double perHour = CostModel.transcription(60, Prices.OPENAI_2026_10_26);
double wavBytes = 16_000 * 2 * 60 * 60, mp3At64 = 64_000 / 8.0 * 60 * 60, mp3At32 = 32_000 / 8.0 * 60 * 60;
t.line("transcription is billed on audio duration, not file size: a 60-minute recording costs $%.4f whatever its encoding.", perHour);
t.line("size of 60 minutes: 16 kHz mono 16-bit WAV = %.1f MB | 64 kbit/s mp3 = %.1f MB | 32 kbit/s mp3 = %.1f MB", wavBytes / 1e6, mp3At64 / 1e6, mp3At32 / 1e6);
t.line("minutes that fit in 25,000,000 bytes: WAV %.1f | 64 kbit/s %.0f | 32 kbit/s %.0f", 25_000_000 / (16_000 * 2.0) / 60,
25_000_000 / (64_000 / 8.0) / 60, 25_000_000 / (32_000 / 8.0) / 60);
assertThat(perHour).isEqualTo(0.27, org.assertj.core.data.Offset.offset(1e-9));
}
}
}
audio/src/test/java/com/ankurm/audio/LargeFileTest.java
+89
View File
@@ -0,0 +1,89 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
import com.ankurm.audio.support.FakeAudioServer;
import com.ankurm.audio.support.MeetingFixture;
import com.ankurm.audio.support.Models;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
/** A 30 MB recording: refused whole, accepted in silence-aligned pieces, damaged by a blind split. MOCK server. */
class LargeFileTest {
private static final byte[] WAV = MeetingFixture.wav();
@Test
void tooBigThenChunked() throws Exception {
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("03-large-file.txt", "A %.0f-minute recording, %,d bytes -- MOCK server".formatted(
MeetingFixture.totalSeconds() / 60, WAV.length))) {
var model = Models.transcription(server.baseUrl(), "gpt-transcribe", 0);
t.line("1) the whole file in one request (mock limit %,d bytes):", FakeAudioServer.UPLOAD_LIMIT);
Throwable thrown = null;
try {
model.transcribe(new NamedBytes(WAV, "meeting.wav"));
} catch (RuntimeException e) {
thrown = e;
}
assertThat(thrown).isNotNull();
t.line(" exception: %s", thrown.getClass().getName());
t.line(" message : %s", thrown.getMessage());
t.blank();
server.clear();
var service = new MinutesService(model, ChatClient.create(new com.ankurm.audio.support.ScriptedChatModel("{}")), null,
"gpt-transcribe", "gpt-4o-mini-tts");
MinutesService.Transcription tr = service.transcribe(WAV, "meeting.wav");
t.line("2) MinutesService.transcribe: budget %,d bytes per chunk, silence of at least 2.0 s, RMS below 500", MinutesService.UPLOAD_BUDGET_BYTES);
for (var c : tr.chunks()) {
t.line(" chunk %d: %7.1f s -> %7.1f s %,11d bytes cut: %s", c.index(), c.startSeconds(), c.endSeconds(), c.bytes(), c.cutKind());
}
t.line(" requests the mock saw: %d", server.seen().size());
for (var s : server.seen()) {
t.line(" - file=%s bytes=%,d prompt=%s", s.filename(), s.fileBytes(),
s.fields().containsKey("prompt") ? "\"..." + s.fields().get("prompt").substring(Math.max(0, s.fields().get("prompt").length() - 40)) + "\"" : "(none)");
}
List<String> expected = MeetingFixture.SCRIPT;
String joined = tr.text();
long found = expected.stream().filter(joined::contains).count();
t.line(" script lines recovered: %d of %d, in order: %s", found, expected.size(),
joined.equals(String.join(" ", expected)));
t.line(" [inaudible] markers: %d", joined.split("\\[inaudible]", -1).length - 1);
assertThat(joined).isEqualTo(String.join(" ", expected));
assertThat(tr.chunks()).hasSizeGreaterThan(1).allSatisfy(c -> assertThat(c.bytes()).isLessThanOrEqualTo((int) MinutesService.UPLOAD_BUDGET_BYTES));
}
}
@Test
void blindSplitCutsASentence() throws Exception {
var chunker = new SilenceChunker(MinutesService.UPLOAD_BUDGET_BYTES, 2.0, 500);
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("04-blind-split.txt", "Cutting every 24,000,000 bytes, wherever that lands -- MOCK server")) {
var model = Models.transcription(server.baseUrl(), "gpt-transcribe", 0);
var chunks = chunker.splitBlindly(WAV);
StringBuilder all = new StringBuilder();
for (var c : chunks) {
int inside = MeetingFixture.sentenceAt(c.endSeconds());
t.line("chunk %d: %7.1f s -> %7.1f s %,d bytes boundary falls %s", c.index(), c.startSeconds(), c.endSeconds(), c.bytes(),
c.index() == chunks.size() - 1 ? "at end of file"
: inside >= 0 ? "INSIDE sentence " + inside + " (\"" + MeetingFixture.SCRIPT.get(inside) + "\")" : "in silence");
all.append(model.transcribe(new NamedBytes(c.wav(), "part-" + c.index() + ".wav"))).append(' ');
}
String text = all.toString().trim();
long found = MeetingFixture.SCRIPT.stream().filter(text::contains).count();
t.blank();
t.line("script lines recovered: %d of %d; [inaudible] markers: %d", found, MeetingFixture.SCRIPT.size(),
text.split("\\[inaudible]", -1).length - 1);
t.line("(mock rule: a tone cut to under 60% of its length is reported as [inaudible]; a real model would not fail this");
t.line(" cleanly -- OpenAI's guide only says splitting mid-sentence can remove context and reduce accuracy.)");
assertThat(found).isLessThan(MeetingFixture.SCRIPT.size());
assertThat(text).contains("[inaudible]");
}
}
}
audio/src/test/java/com/ankurm/audio/MeetingEndToEndTest.java
+43
View File
@@ -0,0 +1,43 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.support.FakeAudioServer;
import com.ankurm.audio.support.MeetingFixture;
import com.ankurm.audio.support.Models;
import com.ankurm.audio.support.ScriptedChatModel;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
/** Audio in, minutes out, audio back, with all three real/stub parts wired through MinutesService. */
class MeetingEndToEndTest {
@Test
void wholePipeline() throws Exception {
String reply = """
{"title":"Falcon weekly sync","decisions":["Ship version 2.4 on the fourteenth of November"],
"actionItems":[{"owner":"Rahul","task":"Fix the login timeout bug","due":"before the release"},
{"owner":"Priya","task":"Send the revised budget to finance","due":"Friday"}]}""";
ScriptedChatModel chat = new ScriptedChatModel(reply);
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("09-end-to-end.txt", "Recording -> transcript -> Minutes -> audio (MOCK audio server, STUB chat model)")) {
var service = new MinutesService(Models.transcription(server.baseUrl(), "gpt-transcribe", 0), ChatClient.create(chat),
Models.speech(server.baseUrl(), "gpt-4o-mini-tts"), "gpt-transcribe", "gpt-4o-mini-tts");
byte[] wav = MeetingFixture.wav();
var tr = service.transcribe(wav, "falcon-sync.wav");
t.line("1) transcribe: %,d bytes -> %d request(s) -> %d characters", wav.length, tr.chunks().size(), tr.text().length());
Minutes minutes = service.summarise(tr.text());
t.line("2) summarise : transcript of %d words reached the chat model as the user message: %s",
tr.text().split("\\s+").length,
chat.prompts().getFirst().getInstructions().stream().anyMatch(m -> m.getText().contains(MeetingFixture.SCRIPT.getFirst())));
t.line(" %d action items: %s", minutes.actionItems().size(), minutes.actionItems());
var clips = service.readAloud(minutes, "coral");
t.line("3) read back : %d clip(s), %d bytes, spoken text: %s", clips.size(), clips.getFirst().length, minutes.toSpokenText());
t.line(" speech request: %s", server.seen().getLast().json());
assertThat(clips).hasSize(1);
assertThat(Wav.parse(clips.getFirst()).seconds()).isGreaterThan(1.0);
}
}
}
audio/src/test/java/com/ankurm/audio/RetryTest.java
+39
View File
@@ -0,0 +1,39 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.support.FakeAudioServer;
import com.ankurm.audio.support.MeetingFixture;
import com.ankurm.audio.support.Models;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
/** What the OpenAI SDK underneath Spring AI does with a failing endpoint. MOCK server. */
class RetryTest {
@Test
void retriesAreTheSdksNotSpringsAndEachOneReuploadsTheFile() throws Exception {
byte[] whole = MeetingFixture.wav();
byte[] clip = Wav.slice(whole, Wav.parse(whole), 0, 120 * MeetingFixture.SAMPLE_RATE);
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("07-retries.txt", "A 120-second clip (" + clip.length + " bytes) against a flaky endpoint -- MOCK server")) {
for (int maxRetries : new int[] { 0, 3 }) {
for (int status : new int[] { 429, 400 }) {
server.clear();
server.failNext(2, status);
var model = Models.transcription(server.baseUrl(), "gpt-transcribe", maxRetries);
String outcome;
try {
model.transcribe(new NamedBytes(clip, "x.wav"));
outcome = "ok";
} catch (RuntimeException e) {
outcome = e.getClass().getSimpleName();
}
long bytes = server.seen().stream().mapToLong(FakeAudioServer.Seen::bodyBytes).sum();
t.line("maxRetries=%d, server fails the first 2 requests with %d -> %-26s requests: %d, bytes uploaded: %,d",
maxRetries, status, outcome, server.seen().size(), bytes);
}
}
}
}
}
audio/src/test/java/com/ankurm/audio/SpeechTest.java
+68
View File
@@ -0,0 +1,68 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import java.util.List;
import com.ankurm.audio.support.FakeAudioServer;
import com.ankurm.audio.support.Models;
import com.ankurm.audio.support.ScriptedChatModel;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.audio.tts.TextToSpeechPrompt;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.openai.OpenAiAudioSpeechOptions;
/** Text to speech: the wire format, the 4096-character wall, and splitting around it. MOCK server. */
class SpeechTest {
@Test
void wireAndLimit() throws Exception {
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("06-speech.txt", "OpenAiAudioSpeechModel -- MOCK server")) {
var speech = Models.speech(server.baseUrl(), "gpt-4o-mini-tts");
byte[] bytes = speech.call("Three action items.");
t.line("speech.call(String) with every default:");
t.line(" request json : %s", server.seen().getLast().json());
t.line(" response : %d bytes", bytes.length);
t.blank();
var opts = OpenAiAudioSpeechOptions.builder().model("gpt-4o-mini-tts").voice("coral").responseFormat("wav").speed(1.1)
.instructions("Calm, like a project manager reading notes.").build();
byte[] wav = speech.call(new TextToSpeechPrompt("Rahul: fix the login timeout bug.", opts)).getResult().getOutput();
t.line("with TextToSpeechPrompt and options:");
t.line(" request json : %s", server.seen().getLast().json());
t.line(" response : %d bytes, starts with %s", wav.length, new String(wav, 0, 4));
t.blank();
String tooLong = "word ".repeat(1000); // 5000 characters
String outcome;
try {
speech.call(new TextToSpeechPrompt(tooLong, opts));
outcome = "accepted";
} catch (RuntimeException e) {
outcome = e.getClass().getSimpleName() + ": " + e.getMessage();
}
t.line("input of %d characters -> %s", tooLong.length(), outcome);
assertThat(outcome).contains("4096");
}
}
@Test
void splittingOnSentenceEnds() throws Exception {
StringBuilder sb = new StringBuilder();
for (int i = 1; i <= 120; i++) sb.append("Action item number ").append(i).append(" is assigned to somebody with a long enough description to take space.").append(' ');
String text = sb.toString().trim();
List<String> parts = MinutesService.splitForSpeech(text, MinutesService.TTS_MAX_CHARS);
try (Transcript t = new Transcript("06b-speech-split.txt", "Splitting a long read-back at sentence ends")) {
t.line("text: %d characters, %d sentences", text.length(), 120);
for (int i = 0; i < parts.size(); i++) {
String p = parts.get(i);
t.line("part %d: %4d characters, ends with \"%s\"", i, p.length(), p.substring(p.length() - 12));
}
}
assertThat(parts).allSatisfy(p -> assertThat(p.length()).isLessThanOrEqualTo(MinutesService.TTS_MAX_CHARS));
assertThat(String.join(" ", parts)).isEqualTo(text);
}
}
audio/src/test/java/com/ankurm/audio/SummariseTest.java
+83
View File
@@ -0,0 +1,83 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.support.MeetingFixture;
import com.ankurm.audio.support.ScriptedChatModel;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.Message;
/** Structured output for action items. The chat model is a STUB that replays canned JSON; it is not a language model. */
class SummariseTest {
private static final String GOOD = """
{"title":"Falcon weekly sync",
"decisions":["Ship version 2.4 on the fourteenth of November","The data pipeline migrates to the new cluster next sprint"],
"actionItems":[
{"owner":"Rahul","task":"Fix the login timeout bug","due":"before the release on the fourteenth of November"},
{"owner":"Priya","task":"Send the revised budget to finance","due":"Friday"},
{"owner":"Sana","task":"Open tickets for both export endpoint issues","due":"today"}]}""";
@Test
void promptAndParsedRecord() {
ScriptedChatModel model = new ScriptedChatModel(GOOD);
var service = new MinutesService(null, ChatClient.create(model), null, "gpt-transcribe", "gpt-4o-mini-tts");
String transcript = String.join(" ", MeetingFixture.SCRIPT);
Minutes m = service.summarise(transcript);
try (Transcript t = new Transcript("05-structured-output.txt", "entity(Minutes.class) against a STUB ChatModel")) {
t.line("messages sent to the model:");
for (Message msg : model.prompts().getFirst().getInstructions()) {
String text = msg.getText();
t.line(" [%s] %d chars", msg.getMessageType(), text.length());
}
String user = model.prompts().getFirst().getInstructions().stream()
.filter(x -> x.getMessageType().name().equals("USER")).findFirst().orElseThrow().getText();
int at = user.indexOf("Your response should be in JSON format.");
t.line("");
t.line("tail of the USER message (the format instructions Spring AI appended):");
t.line("%s", user.substring(at));
t.blank();
t.line("parsed: %s", m.title());
t.line(" decisions : %d", m.decisions().size());
m.actionItems().forEach(a -> t.line(" action item : %-6s | %s | due: %s", a.owner(), a.task(), a.due()));
t.blank();
t.line("spoken form:");
t.line(" %s", m.toSpokenText());
}
assertThat(m.actionItems()).hasSize(3);
assertThat(m.actionItems().getFirst().owner()).isEqualTo("Rahul");
}
@Test
void whatABadReplyLooksLike() {
try (Transcript t = new Transcript("05b-bad-replies.txt", "Three replies a model can give instead of clean JSON (STUB model)")) {
String[] replies = {
"Here are the minutes:\n```json\n" + GOOD + "\n```",
"{\"title\":\"Falcon weekly sync\",\"decisions\":[],\"actionItems\":[{\"owner\":\"Rahul\",\"task\":\"Fix it\",\"due\":null}]",
"I could not find any action items in this transcript."
};
String[] labels = { "wrapped in a markdown fence with a sentence before it", "truncated: closing brace missing", "prose instead of JSON" };
for (int i = 0; i < replies.length; i++) {
var service = new MinutesService(null, ChatClient.create(new ScriptedChatModel(replies[i])), null, "x", "y");
String outcome;
try {
Minutes m = service.summarise("anything");
outcome = "parsed: " + m.actionItems().size() + " action item(s), first due=" + m.actionItems().getFirst().due();
} catch (RuntimeException e) {
outcome = e.getClass().getSimpleName() + ": " + first(e.getMessage());
}
t.line("%-52s -> %s", labels[i], outcome);
}
}
}
private static String first(String s) {
if (s == null) return "(no message)";
String one = s.replace('\n', ' ');
return one.length() > 110 ? one.substring(0, 110) + "..." : one;
}
}
audio/src/test/java/com/ankurm/audio/TranscribeTest.java
+68
View File
@@ -0,0 +1,68 @@
package com.ankurm.audio;
import static org.assertj.core.api.Assertions.assertThat;
import com.ankurm.audio.support.FakeAudioServer;
import com.ankurm.audio.support.MeetingFixture;
import com.ankurm.audio.support.Models;
import com.ankurm.audio.support.Transcript;
import org.junit.jupiter.api.Test;
import org.springframework.ai.audio.transcription.AudioTranscriptionPrompt;
import org.springframework.ai.audio.transcription.AudioTranscriptionResponse;
import org.springframework.ai.openai.OpenAiAudioTranscriptionOptions;
/** The smallest working call, and exactly what the real client puts on the wire. MOCK server. */
class TranscribeTest {
@Test
void smallClip() throws Exception {
byte[] whole = MeetingFixture.wav();
Wav.Info info = Wav.parse(whole);
byte[] clip = Wav.slice(whole, info, 0, 60 * MeetingFixture.SAMPLE_RATE); // first 60 s: two sentences
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("02-transcribe-small.txt",
"One 60-second clip (" + clip.length + " bytes WAV) through OpenAiAudioTranscriptionModel -- MOCK server")) {
for (String model : new String[] { "whisper-1", "gpt-transcribe" }) {
server.clear();
var m = Models.transcription(server.baseUrl(), model);
// The convenience method: text in, text out.
String text = m.transcribe(new NamedBytes(clip, "standup.wav"));
FakeAudioServer.Seen s = server.seen().getFirst();
t.line("model option: %s", model);
t.line(" request : POST %s (multipart, %d bytes)", s.path(), s.bodyBytes());
t.line(" fields : %s", s.fields());
t.line(" file part : name=%s type=%s bytes=%d", s.filename(), s.fileContentType(), s.fileBytes());
t.line(" transcribe(Resource) -> \"%s\"", text);
// The full form: options per call, and a response object.
AudioTranscriptionResponse r = m.call(new AudioTranscriptionPrompt(new NamedBytes(clip, "standup.wav"),
OpenAiAudioTranscriptionOptions.builder().model(model).language("en").prompt("Falcon weekly sync").build()));
FakeAudioServer.Seen s2 = server.seen().getLast();
t.line(" with language=en and prompt, fields now: %s", s2.fields());
t.line(" response metadata class: %s", r.getMetadata().getClass().getSimpleName());
t.blank();
assertThat(text).startsWith(MeetingFixture.SCRIPT.get(0)).contains(MeetingFixture.SCRIPT.get(1));
}
}
}
@Test
void filenameExtensionMatters() throws Exception {
byte[] whole = MeetingFixture.wav();
byte[] clip = Wav.slice(whole, Wav.parse(whole), 0, 20 * MeetingFixture.SAMPLE_RATE);
try (FakeAudioServer server = new FakeAudioServer();
Transcript t = new Transcript("02b-filename.txt", "The upload's file name decides whether it is accepted -- MOCK server")) {
var m = Models.transcription(server.baseUrl(), "gpt-transcribe", 0);
for (String name : new String[] { "standup.wav", "standup.flac", "standup" }) {
String outcome;
try {
outcome = "ok: " + m.transcribe(new NamedBytes(clip, name)).length() + " chars";
} catch (RuntimeException e) {
outcome = e.getClass().getSimpleName() + ": " + e.getMessage();
}
t.line("file name %-13s -> %s", name, outcome);
}
}
}
}
audio/src/test/java/com/ankurm/audio/support/FakeAudioServer.java
+252
View File
@@ -0,0 +1,252 @@
package com.ankurm.audio.support;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import com.ankurm.audio.Wav;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
/**
* A MOCK of two OpenAI endpoints, /v1/audio/transcriptions and /v1/audio/speech, on a local port.
* The real Spring AI model classes (and the OpenAI Java SDK underneath them) talk to it over HTTP.
*
* <p>What is real: the requests. Everything the client sends is recorded here as it arrived.
* <p>What is NOT real: the "transcription". There is no speech recogniser. The server decodes the
* pitch of each tone in the synthetic recording ({@link MeetingFixture}) back into the scripted
* sentence, and answers "[inaudible]" for a tone cut short (the mock's rule, standing in for the
* accuracy loss the OpenAI guide describes when you split mid-sentence). The "speech" is a beep
* whose length follows the text length. Nothing here says anything about accuracy, latency or
* cost of a real model.
*
* <p>Mock rules taken from OpenAI's documentation: uploads over 25 MB are refused (413; this mock
* uses 25 x 1024 x 1024 bytes), only the documented input extensions are accepted, and speech
* input over 4096 characters is refused (400).
*/
public final class FakeAudioServer implements AutoCloseable {
public static final long UPLOAD_LIMIT = 25L * 1024 * 1024;
private static final List<String> EXTENSIONS = List.of("mp3", "mp4", "mpeg", "mpga", "m4a", "wav", "webm");
private static final JsonMapper JSON = JsonMapper.builder().build();
/** One recorded request. */
public record Seen(String path, String contentType, long bodyBytes, Map<String, String> fields,
String filename, String fileContentType, int fileBytes, String json) {}
private final HttpServer server;
private final List<Seen> seen = new CopyOnWriteArrayList<>();
private final AtomicInteger failuresLeft = new AtomicInteger();
private volatile int failureStatus;
public FakeAudioServer() throws IOException {
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/v1/audio/transcriptions", this::transcriptions);
server.createContext("/v1/audio/speech", this::speech);
server.start();
}
/** The base URL to give to Spring AI's {@code baseUrl(...)} option (the SDK appends /audio/...). */
public String baseUrl() {
return "http://127.0.0.1:" + server.getAddress().getPort() + "/v1";
}
public List<Seen> seen() { return seen; }
public void clear() { seen.clear(); }
/** The next {@code times} requests fail with {@code status} before normal service resumes. */
public void failNext(int times, int status) {
failureStatus = status;
failuresLeft.set(times);
}
@Override
public void close() { server.stop(0); }
// ---------------------------------------------------------------- transcriptions
private void transcriptions(HttpExchange ex) throws IOException {
byte[] body = ex.getRequestBody().readAllBytes();
String ct = ex.getRequestHeaders().getFirst("Content-Type");
Multipart mp = Multipart.parse(body, ct);
seen.add(new Seen("/v1/audio/transcriptions", ct, body.length, mp.fields, mp.filename, mp.fileType,
mp.file == null ? 0 : mp.file.length, null));
if (failuresLeft.getAndUpdate(n -> Math.max(0, n - 1)) > 0) {
reply(ex, failureStatus, "application/json", error("mock: simulated failure " + failureStatus, "server_error"));
return;
}
if (body.length > UPLOAD_LIMIT) {
reply(ex, 413, "application/json", error("mock: Maximum content size limit (" + UPLOAD_LIMIT
+ ") exceeded (" + body.length + " bytes read)", "invalid_request_error"));
return;
}
String ext = mp.filename == null || !mp.filename.contains(".") ? ""
: mp.filename.substring(mp.filename.lastIndexOf('.') + 1).toLowerCase();
if (!EXTENSIONS.contains(ext)) {
reply(ex, 400, "application/json", error("mock: Invalid file format. Supported formats: " + EXTENSIONS,
"invalid_request_error"));
return;
}
String text = decode(mp.file);
String model = mp.fields.getOrDefault("model", "");
String format = mp.fields.getOrDefault("response_format", "json");
if (format.equals("text")) {
reply(ex, 200, "text/plain", text);
} else if (model.equals("gpt-transcribe")) {
reply(ex, 200, "application/json", JSON.writeValueAsString(
Map.of("text", text, "languages", List.of(Map.of("code", "en")))));
} else {
reply(ex, 200, "application/json", JSON.writeValueAsString(Map.of("text", text)));
}
}
/** Pitch decoder: tones back into script lines. See the class comment: this is the mock's "recogniser". */
static String decode(byte[] wav) {
Wav.Info info = Wav.parse(wav);
int win = info.sampleRate() / 20; // 50 ms
List<String> out = new ArrayList<>();
int runStart = -1;
for (int f = 0; f + win <= info.frames() + win; f += win) {
boolean active = f + win <= info.frames() && rms(wav, info, f, win) > 500;
if (active && runStart < 0) runStart = f;
if (!active && runStart >= 0) {
out.add(sentence(wav, info, runStart, f));
runStart = -1;
}
}
return String.join(" ", out);
}
private static String sentence(byte[] wav, Wav.Info info, int from, int to) {
int crossings = 0, last = 0, first = -1, lastAt = -1; // hysteresis: only count swings between +1000 and -1000
for (int i = from; i < to; i++) {
short v = Wav.sample(wav, info, i);
int sign = v > 1000 ? 1 : v < -1000 ? -1 : 0;
if (sign != 0) {
if (last != 0 && sign != last) { crossings++; if (first < 0) first = i; lastAt = i; }
last = sign;
}
}
double seconds = (double) (to - from) / info.sampleRate();
// frequency from the span between the first and last crossing, so window edges do not matter
double hz = crossings < 3 ? 0 : (crossings - 1) / 2.0 / ((double) (lastAt - first) / info.sampleRate());
int idx = (int) Math.round((hz - 300) / 20);
if (idx < 0 || idx >= MeetingFixture.SCRIPT.size()) return "[inaudible]";
if (seconds < 0.6 * MeetingFixture.duration(idx)) return "[inaudible]";
return MeetingFixture.SCRIPT.get(idx);
}
private static double rms(byte[] wav, Wav.Info info, int from, int n) {
double sum = 0;
for (int i = 0; i < n; i++) { double s = Wav.sample(wav, info, from + i); sum += s * s; }
return Math.sqrt(sum / n);
}
// ---------------------------------------------------------------- speech
private void speech(HttpExchange ex) throws IOException {
byte[] body = ex.getRequestBody().readAllBytes();
String json = new String(body, StandardCharsets.UTF_8);
seen.add(new Seen("/v1/audio/speech", ex.getRequestHeaders().getFirst("Content-Type"), body.length,
Map.of(), null, null, 0, json));
JsonNode req = JSON.readTree(json);
String input = req.path("input").asText("");
if (input.length() > 4096) {
reply(ex, 400, "application/json", error("mock: the maximum length of input is 4096 characters, got "
+ input.length(), "invalid_request_error"));
return;
}
double seconds = Math.max(0.5, input.length() / 15.0); // ~15 characters per second of speech
int rate = 8000, frames = (int) (seconds * rate);
byte[] pcm = new byte[frames * 2];
for (int f = 0; f < frames; f++) {
short v = (short) (3000 * Math.sin(2 * Math.PI * 440 * f / rate));
pcm[2 * f] = (byte) v;
pcm[2 * f + 1] = (byte) (v >> 8);
}
byte[] wav = Wav.wrap(pcm, 0, pcm.length, rate, 1, 16);
String fmt = req.path("response_format").asText("mp3");
reply(ex, 200, fmt.equals("wav") ? "audio/wav" : "audio/mpeg", wav);
}
// ---------------------------------------------------------------- plumbing
private static String error(String message, String type) {
return JSON.writeValueAsString(Map.of("error", Map.of("message", message, "type", type)));
}
private static void reply(HttpExchange ex, int status, String type, String body) throws IOException {
reply(ex, status, type, body.getBytes(StandardCharsets.UTF_8));
}
private static void reply(HttpExchange ex, int status, String type, byte[] body) throws IOException {
ex.getResponseHeaders().add("Content-Type", type);
ex.sendResponseHeaders(status, body.length);
ex.getResponseBody().write(body);
ex.close();
}
/** Just enough multipart/form-data. */
private static final class Multipart {
final Map<String, String> fields = new LinkedHashMap<>();
String filename, fileType;
byte[] file;
static Multipart parse(byte[] body, String contentType) {
Multipart m = new Multipart();
if (contentType == null || !contentType.contains("boundary=")) return m;
String boundary = "--" + contentType.substring(contentType.indexOf("boundary=") + 9).replace("\"", "");
byte[] delim = boundary.getBytes(StandardCharsets.ISO_8859_1);
int pos = indexOf(body, delim, 0);
while (pos >= 0) {
int start = pos + delim.length;
if (start + 2 > body.length || (body[start] == '-' && body[start + 1] == '-')) break;
start += 2; // CRLF after the boundary
int next = indexOf(body, delim, start);
if (next < 0) break;
int headEnd = indexOf(body, "\r\n\r\n".getBytes(StandardCharsets.ISO_8859_1), start);
String head = new String(body, start, headEnd - start, StandardCharsets.ISO_8859_1);
int dataStart = headEnd + 4, dataEnd = next - 2; // CRLF before the next boundary
String name = attr(head, "name");
if (head.contains("filename=")) {
m.filename = attr(head, "filename");
m.fileType = head.lines().filter(l -> l.toLowerCase().startsWith("content-type:"))
.map(l -> l.substring(13).trim()).findFirst().orElse(null);
ByteArrayOutputStream bo = new ByteArrayOutputStream();
bo.write(body, dataStart, dataEnd - dataStart);
m.file = bo.toByteArray();
} else {
m.fields.put(name, new String(body, dataStart, dataEnd - dataStart, StandardCharsets.UTF_8));
}
pos = next;
}
return m;
}
private static String attr(String head, String key) {
int i = head.indexOf(key + "=\"");
if (i < 0) return null;
int s = i + key.length() + 2;
return head.substring(s, head.indexOf('"', s));
}
private static int indexOf(byte[] hay, byte[] needle, int from) {
outer:
for (int i = from; i <= hay.length - needle.length; i++) {
for (int j = 0; j < needle.length; j++) if (hay[i + j] != needle[j]) continue outer;
return i;
}
return -1;
}
}
}
audio/src/test/java/com/ankurm/audio/support/MeetingFixture.java
+108
View File
@@ -0,0 +1,108 @@
package com.ankurm.audio.support;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.List;
import com.ankurm.audio.Wav;
/**
* A SYNTHETIC "meeting": 30 scripted sentences, each rendered as a pure tone whose pitch encodes
* the sentence number (300 Hz + 20 Hz x n), separated by ~25 seconds of low noise. It is not
* speech. The mock server decodes the pitch back into the script line, which lets the real Spring
* AI client and the real chunking code run end to end with no key and no model, and lets the
* tests check that no sentence was lost or cut in half.
*/
public final class MeetingFixture {
public static final int SAMPLE_RATE = 16_000;
public static final double FIRST_START = 11.0;
public static final double STEP = 32.0;
public static final double TAIL = 5.0;
public static final List<String> SCRIPT = List.of(
"Okay, let's get started with the Falcon weekly sync.",
"First item is the release date for version two point four.",
"Rahul says the login timeout bug is the only blocker left.",
"Rahul will fix the login timeout bug before the release on the fourteenth of November.",
"Can we confirm QA has a full day on the thirteenth.",
"Meera confirms QA has the whole day on the thirteenth.",
"Decision: we ship version two point four on the fourteenth of November.",
"Next is the budget for the cloud account.",
"Priya noted that spend is twelve percent over plan this quarter.",
"Priya will send the revised budget to finance by Friday.",
"Does anyone object to moving the data pipeline to the new cluster.",
"No objections, the pipeline moves to the new cluster next sprint.",
"Decision: the data pipeline migrates to the new cluster next sprint.",
"Let's talk about the offsite in December.",
"Meera will book the venue for the offsite and share options by Wednesday.",
"Arjun asked whether we still need the staging database.",
"We keep staging until the migration is finished.",
"Arjun will write up the staging database retirement plan after the migration.",
"Any updates on the security review.",
"The review found two medium issues, both in the export endpoint.",
"Sana will open tickets for both export endpoint issues today.",
"Decision: the export endpoint fixes are not release blockers.",
"Quick note that the on call rotation changes next month.",
"Everyone should check the new rotation sheet before the end of this week.",
"Any other business.",
"Rahul wants a second reviewer on the login fix.",
"Sana will review the login fix once Rahul opens the pull request.",
"Great, that is everything on the list.",
"Thanks everyone, the notes will go out this afternoon.",
"Meeting closed.");
private MeetingFixture() {}
/** Seconds of tone for sentence {@code i}: proportional to its word count. */
private static final double[] DURATIONS = SCRIPT.stream()
.mapToDouble(line -> 1.5 + 0.35 * line.split("\\s+").length).toArray();
public static double duration(int i) {
return DURATIONS[i];
}
public static double start(int i) {
return FIRST_START + STEP * i;
}
public static double pitch(int i) {
return 300 + 20 * i;
}
public static double totalSeconds() {
return start(SCRIPT.size() - 1) + duration(SCRIPT.size() - 1) + TAIL;
}
/** The whole recording as a 16 kHz mono 16-bit WAV (about 30 MB). */
public static byte[] wav() {
return CACHE;
}
private static final byte[] CACHE = build();
private static byte[] build() {
int frames = (int) (totalSeconds() * SAMPLE_RATE);
ByteBuffer pcm = ByteBuffer.allocate(frames * 2).order(ByteOrder.LITTLE_ENDIAN);
long seed = 42;
for (int f = 0; f < frames; f++) {
seed = seed * 6364136223846793005L + 1442695040888963407L;
double v = ((seed >>> 33) % 81) - 40; // +-40 of room noise
double t = (double) f / SAMPLE_RATE;
int i = (int) Math.floor((t - FIRST_START) / STEP);
if (i >= 0 && i < SCRIPT.size() && t < start(i) + duration(i)) {
v += 8000 * Math.sin(2 * Math.PI * pitch(i) * (t - start(i)));
}
pcm.putShort((short) Math.round(v));
}
return Wav.wrap(pcm.array(), 0, pcm.capacity(), SAMPLE_RATE, 1, 16);
}
/** Which sentence does this exact slice of time fall inside, or -1 for silence. */
public static int sentenceAt(double seconds) {
for (int i = 0; i < SCRIPT.size(); i++) {
if (seconds >= start(i) && seconds < start(i) + duration(i)) return i;
}
return -1;
}
}
audio/src/test/java/com/ankurm/audio/support/Models.java
+28
View File
@@ -0,0 +1,28 @@
package com.ankurm.audio.support;
import org.springframework.ai.openai.OpenAiAudioSpeechModel;
import org.springframework.ai.openai.OpenAiAudioSpeechOptions;
import org.springframework.ai.openai.OpenAiAudioTranscriptionModel;
import org.springframework.ai.openai.OpenAiAudioTranscriptionOptions;
/** The two real Spring AI OpenAI audio models, pointed at the local mock. */
public final class Models {
private Models() {}
public static OpenAiAudioTranscriptionModel transcription(String baseUrl, String model) {
return transcription(baseUrl, model, null);
}
public static OpenAiAudioTranscriptionModel transcription(String baseUrl, String model, Integer maxRetries) {
OpenAiAudioTranscriptionOptions.Builder o = OpenAiAudioTranscriptionOptions.builder()
.baseUrl(baseUrl).apiKey("test-key-not-real").model(model);
if (maxRetries != null) o.maxRetries(maxRetries);
return OpenAiAudioTranscriptionModel.builder().options(o.build()).build();
}
public static OpenAiAudioSpeechModel speech(String baseUrl, String model) {
return OpenAiAudioSpeechModel.builder().options(OpenAiAudioSpeechOptions.builder()
.baseUrl(baseUrl).apiKey("test-key-not-real").model(model).build()).build();
}
}
audio/src/test/java/com/ankurm/audio/support/ScriptedChatModel.java
+36
View File
@@ -0,0 +1,36 @@
package com.ankurm.audio.support;
import java.util.ArrayList;
import java.util.List;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;
/**
* A ChatModel that replays canned replies and remembers every prompt it received. STUB: it is not
* a language model and understands nothing in the prompt.
*/
public final class ScriptedChatModel implements ChatModel {
private final List<String> replies;
private final List<Prompt> seen = new ArrayList<>();
private int next;
public ScriptedChatModel(String... replies) {
this.replies = List.of(replies);
}
@Override
public ChatResponse call(Prompt prompt) {
seen.add(prompt);
String reply = replies.get(Math.min(next++, replies.size() - 1));
return new ChatResponse(List.of(new Generation(new AssistantMessage(reply))));
}
public List<Prompt> prompts() {
return seen;
}
}
audio/src/test/java/com/ankurm/audio/support/Transcript.java
+46
View File
@@ -0,0 +1,46 @@
package com.ankurm.audio.support;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.nio.file.Files;
import java.nio.file.Path;
/**
* Writes a numbered transcript under {@code output/} (module root, not {@code docs/}) and echoes
* it to the console. Every console block quoted in the article comes out of one of these files.
*/
public final class Transcript implements AutoCloseable {
private final Path path;
private final StringWriter buffer = new StringWriter();
private final PrintWriter out = new PrintWriter(buffer);
public Transcript(String fileName, String title) {
this.path = Path.of("output", fileName);
out.println("# " + title);
out.println();
}
public Transcript line(String format, Object... args) {
out.println(args.length == 0 ? format : String.format(format, args));
return this;
}
public Transcript blank() {
out.println();
return this;
}
@Override
public void close() {
out.flush();
try {
Files.createDirectories(path.getParent());
Files.writeString(path, buffer.toString());
} catch (IOException e) {
throw new IllegalStateException("could not write " + path, e);
}
System.out.print(buffer);
}
}