Add audio module: meeting minutes with Spring AI speech-to-text, structured output and text-to-speech

OpenAiAudioTranscriptionModel -> ChatClient.entity(Minutes.class) -> OpenAiAudioSpeechModel, WAV chunking at a silence under the 25 MB upload limit, SDK retry behaviour, the 4096-character speech limit and a cost model. Audio endpoints are a local mock and the chat model a stub; no real model was called. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01CifR2BE7PBvRy5hT8TS9oz
2026-10-26 02:00:28 +00:00
parent ff58ff0189
commit 279be7eb9f
36 changed files with 1576 additions and 0 deletions
@@ -0,0 +1,15 @@
+# Spring AI 2.0.2 audio classes and defaults (read from the jars)
+
+speech-to-text interface : org.springframework.ai.audio.transcription.TranscriptionModel
+speech-to-text OpenAI    : org.springframework.ai.openai.OpenAiAudioTranscriptionModel
+text-to-speech interface : org.springframework.ai.audio.tts.TextToSpeechModel
+text-to-speech OpenAI    : org.springframework.ai.openai.OpenAiAudioSpeechModel
+
+transcription DEFAULT_TRANSCRIPTION_MODEL = whisper-1
+transcription DEFAULT_RESPONSE_FORMAT     = text
+speech        DEFAULT_SPEECH_MODEL        = gpt-4o-mini-tts
+speech        DEFAULT_VOICE               = alloy
+speech        DEFAULT_RESPONSE_FORMAT     = mp3
+speech        DEFAULT_SPEED               = 1.0
+both          DEFAULT_TIMEOUT             = PT1M
+both          DEFAULT_MAX_RETRIES         = 3
@@ -0,0 +1,18 @@
+# One 60-second clip (1920044 bytes WAV) through OpenAiAudioTranscriptionModel -- MOCK server
+
+model option: whisper-1
+  request   : POST /v1/audio/transcriptions  (multipart, 1920524 bytes)
+  fields    : {model=whisper-1, response_format=text}
+  file part : name=standup.wav type=application/octet-stream bytes=1920044
+  transcribe(Resource) -> "Okay, let's get started with the Falcon weekly sync. First item is the release date for version two point four."
+  with language=en and prompt, fields now: {model=whisper-1, language=en, prompt=Falcon weekly sync, response_format=text}
+  response metadata class: OpenAiAudioTranscriptionResponseMetadata
+
+model option: gpt-transcribe
+  request   : POST /v1/audio/transcriptions  (multipart, 1920529 bytes)
+  fields    : {model=gpt-transcribe, response_format=text}
+  file part : name=standup.wav type=application/octet-stream bytes=1920044
+  transcribe(Resource) -> "Okay, let's get started with the Falcon weekly sync. First item is the release date for version two point four."
+  with language=en and prompt, fields now: {model=gpt-transcribe, language=en, prompt=Falcon weekly sync, response_format=text}
+  response metadata class: OpenAiAudioTranscriptionResponseMetadata
+
@@ -0,0 +1,5 @@
+# The upload's file name decides whether it is accepted -- MOCK server
+
+file name standup.wav   -> ok: 52 chars
+file name standup.flac  -> BadRequestException: 400: mock: Invalid file format. Supported formats: [mp3, mp4, mpeg, mpga, m4a, wav, webm]
+file name standup       -> BadRequestException: 400: mock: Invalid file format. Supported formats: [mp3, mp4, mpeg, mpga, m4a, wav, webm]
@@ -0,0 +1,14 @@
+# A 16-minute recording, 30,278,444 bytes -- MOCK server
+
+1) the whole file in one request (mock limit 26,214,400 bytes):
+   exception: com.openai.errors.UnexpectedStatusCodeException
+   message  : 413: mock: Maximum content size limit (26214400) exceeded (30278929 bytes read)
+
+2) MinutesService.transcribe: budget 24,000,000 bytes per chunk, silence of at least 2.0 s, RMS below 500
+   chunk 0:     0.0 s ->   733.5 s   23,472,022 bytes  cut: mid-silence
+   chunk 1:   733.5 s ->   946.2 s    6,806,466 bytes  cut: end of file
+   requests the mock saw: 2
+   - file=part-00-meeting.wav bytes=23,472,022 prompt=(none)
+   - file=part-01-meeting.wav bytes=6,806,466 prompt="...the on call rotation changes next month."
+   script lines recovered: 30 of 30, in order: true
+   [inaudible] markers: 0
@@ -0,0 +1,8 @@
+# Cutting every 24,000,000 bytes, wherever that lands -- MOCK server
+
+chunk 0:     0.0 s ->   750.0 s  24,000,000 bytes  boundary falls INSIDE sentence 23 ("Everyone should check the new rotation sheet before the end of this week.")
+chunk 1:   750.0 s ->   946.2 s  6,278,488 bytes  boundary falls at end of file
+
+script lines recovered: 29 of 30; [inaudible] markers: 2
+(mock rule: a tone cut to under 60% of its length is reported as [inaudible]; a real model would not fail this
+ cleanly -- OpenAI's guide only says splitting mid-sentence can remove context and reduce accuracy.)
@@ -0,0 +1,58 @@
+# entity(Minutes.class) against a STUB ChatModel
+
+messages sent to the model:
+  [SYSTEM] 222 chars
+  [USER] 2799 chars
+
+tail of the USER message (the format instructions Spring AI appended):
+Your response should be in JSON format.
+Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
+Do not include markdown code blocks in your response.
+Remove the ```json markdown from the output.
+Here is the JSON Schema instance your output must adhere to:
+```{
+  "$schema" : "https://json-schema.org/draft/2020-12/schema",
+  "type" : "object",
+  "properties" : {
+    "actionItems" : {
+      "type" : "array",
+      "items" : {
+        "type" : "object",
+        "properties" : {
+          "due" : {
+            "type" : "string"
+          },
+          "owner" : {
+            "type" : "string"
+          },
+          "task" : {
+            "type" : "string"
+          }
+        },
+        "required" : [ "due", "owner", "task" ],
+        "additionalProperties" : false
+      }
+    },
+    "decisions" : {
+      "type" : "array",
+      "items" : {
+        "type" : "string"
+      }
+    },
+    "title" : {
+      "type" : "string"
+    }
+  },
+  "required" : [ "actionItems", "decisions", "title" ],
+  "additionalProperties" : false
+}```
+
+
+parsed: Falcon weekly sync
+  decisions   : 2
+  action item : Rahul  | Fix the login timeout bug | due: before the release on the fourteenth of November
+  action item : Priya  | Send the revised budget to finance | due: Friday
+  action item : Sana   | Open tickets for both export endpoint issues | due: today
+
+spoken form:
+  Falcon weekly sync. 2 decisions. Ship version 2.4 on the fourteenth of November. The data pipeline migrates to the new cluster next sprint. 3 action items. Rahul: Fix the login timeout bug, due before the release on the fourteenth of November. Priya: Send the revised budget to finance, due Friday. Sana: Open tickets for both export endpoint issues, due today.
@@ -0,0 +1,5 @@
+# Three replies a model can give instead of clean JSON (STUB model)
+
+wrapped in a markdown fence with a sentence before it -> StreamReadException: Unrecognized token 'Here': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false...
+truncated: closing brace missing                     -> UnexpectedEndOfInputException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: REDACTED (`StreamReadFeatu...
+prose instead of JSON                                -> StreamReadException: Unrecognized token 'I': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') ...
@@ -0,0 +1,11 @@
+# OpenAiAudioSpeechModel -- MOCK server
+
+speech.call(String) with every default:
+  request json : {"input":"Three action items.","model":"gpt-4o-mini-tts","voice":"alloy","response_format":"mp3","speed":1.0}
+  response     : 20310 bytes
+
+with TextToSpeechPrompt and options:
+  request json : {"input":"Rahul: fix the login timeout bug.","model":"gpt-4o-mini-tts","voice":"coral","instructions":"Calm, like a project manager reading notes.","response_format":"wav","speed":1.1}
+  response     : 35244 bytes, starts with RIFF
+
+input of 5000 characters -> BadRequestException: 400: mock: the maximum length of input is 4096 characters, got 5000
@@ -0,0 +1,6 @@
+# Splitting a long read-back at sentence ends
+
+text: 11051 characters, 120 sentences
+part 0: 4038 characters, ends with " take space."
+part 1: 4047 characters, ends with " take space."
+part 2: 2964 characters, ends with " take space."
@@ -0,0 +1,6 @@
+# A 120-second clip (3840044 bytes) against a flaky endpoint -- MOCK server
+
+maxRetries=0, server fails the first 2 requests with 429 -> RateLimitException         requests: 1, bytes uploaded: 3,840,523
+maxRetries=0, server fails the first 2 requests with 400 -> BadRequestException        requests: 1, bytes uploaded: 3,840,523
+maxRetries=3, server fails the first 2 requests with 429 -> ok                         requests: 3, bytes uploaded: 11,521,569
+maxRetries=3, server fails the first 2 requests with 400 -> BadRequestException        requests: 1, bytes uploaded: 3,840,523
@@ -0,0 +1,14 @@
+# Cost of one meeting, USD (prices of 2026-10-26; word and token counts are ASSUMPTIONS)
+
+assumptions: 150 words/minute spoken, 0.75 words/token, summary output 600 tokens, read-back 120 words
+prices      : gpt-transcribe $0.0045/min | whisper-1 $0.006/min | gpt-4o-mini $0.15 in / $0.60 out per 1M | tts-1 $15 per 1M chars
+
+ minutes       words  gpt-transc   whisper-1     summary       tts-1   total(gt)
+      15       2,250      0.0675      0.0900      0.0008      0.0108      0.0791
+      30       4,500      0.1350      0.1800      0.0013      0.0108      0.1471
+      60       9,000      0.2700      0.3600      0.0022      0.0108      0.2830
+     120      18,000      0.5400      0.7200      0.0040      0.0108      0.5548
+
+transcription is billed on audio duration, not file size: a 60-minute recording costs $0.2700 whatever its encoding.
+size of 60 minutes: 16 kHz mono 16-bit WAV = 115.2 MB | 64 kbit/s mp3 = 28.8 MB | 32 kbit/s mp3 = 14.4 MB
+minutes that fit in 25,000,000 bytes: WAV 13.0 | 64 kbit/s 52 | 32 kbit/s 104
@@ -0,0 +1,7 @@
+# Recording -> transcript -> Minutes -> audio (MOCK audio server, STUB chat model)
+
+1) transcribe: 30,278,444 bytes -> 2 request(s) -> 1701 characters
+2) summarise : transcript of 292 words reached the chat model as the user message: true
+   2 action items: [ActionItem[owner=Rahul, task=Fix the login timeout bug, due=before the release], ActionItem[owner=Priya, task=Send the revised budget to finance, due=Friday]]
+3) read back : 1 clip(s), 221910 bytes, spoken text: Falcon weekly sync. 1 decision. Ship version 2.4 on the fourteenth of November. 2 action items. Rahul: Fix the login timeout bug, due before the release. Priya: Send the revised budget to finance, due Friday.
+   speech request: {"input":"Falcon weekly sync. 1 decision. Ship version 2.4 on the fourteenth of November. 2 action items. Rahul: Fix the login timeout bug, due before the release. Priya: Send the revised budget to finance, due Friday.","model":"gpt-4o-mini-tts","voice":"coral","response_format":"wav","speed":1.0}