AudioToTextProvider: preserve segments and speaker labels in transcript output (verbose_json is already requested)

Describe the feature you'd like to request

The transcribe() method in lib/Service/OpenAiAPIService.php (line ~812 in v4.5.1) sends response_format = 'verbose_json' on every audio transcription request. The comment above the request explains why: "Verbose needed for extraction of audio duration". After using the last segment's end field for quota accounting, the entire segments[] array is discarded and only $response['text'] is returned to callers (line ~838).

Every transcription call therefore pays for:

• Larger response body (segments[] with per-segment start / end / text and, on newer STT providers, speaker labels)
• Extra JSON parsing work

...but the output surface only exposes what a plain response_format = 'json' request would have returned. The extra data is fetched, parsed, and thrown away.

For Talk call recording transcripts specifically, this loses meaningful value:

• No timestamps in Recording <ts> - transcript.md — impossible to jump to "when was this discussed" in a longer meeting
• No speaker attribution — the transcript reads as a flat text blob even when the STT provider returned per-segment speaker labels
• No paragraph structure — one long string without natural breaks makes the transcript harder to skim and harder for the summariser to shape

Framed as a code smell: if the segments data isn't worth exposing to callers, request json and save bandwidth. If it is worth having — which the current implementation seems to concede by requesting verbose_json — please make it available in the output.

Environment where observed:

• Nextcloud 33.0.6 (Hub 26 Winter)
• integration_openai 4.5.1
• Talk 23.0.8
• STT backend: LocalAI-compatible endpoint (LiteLLM proxy + local Whisper on Intel NPU with speaker diarization) — url config points at the LiteLLM endpoint. Same shape applies to the OpenAI cloud Whisper API — its verbose_json response also returns segments[].start / segments[].end, and the OpenAI API is progressively adding richer diarization info to newer models.

Describe the solution you'd like

Two edits in OpenAiAPIService::transcribe().

First, in the existing quota-accounting block a few lines above the insertion point, swap the destructive array_pop($response['segments']) for the non-destructive end($response['segments']). array_pop mutates the array in place, so if left as-is the formatting loop below never sees the final segment (it has already been removed by the time the loop runs). end() returns the same last element but leaves the array intact.

Then, after the (now non-destructive) quota accounting, format the segments into the returned string when they're present. Fall back to $response['text'] when they're not (defensive against non-shim providers or future API changes):

// After the existing quota accounting
if (isset($response['segments']) && is_array($response['segments']) && !empty($response['segments'])) {
    $lines = [];
    $currentSpeaker = null;
    $currentLine = '';
    $lastTsAt = -30.0;  // ensures first ts is always emitted
    foreach ($response['segments'] as $segment) {
        $segText = trim($segment['text'] ?? '');
        if ($segText === '') { continue; }
        $speaker = $segment['speaker'] ?? '';
        $start = floatval($segment['start'] ?? 0);
        $totalSeconds = intval($start);
        $ts = sprintf('%02d:%02d:%02d',
            intdiv($totalSeconds, 3600),
            intdiv($totalSeconds % 3600, 60),
            $totalSeconds % 60);
        $newTurn = ($speaker !== $currentSpeaker) || ($start - $lastTsAt >= 30.0);
        if ($newTurn) {
            if ($currentLine !== '') { $lines[] = $currentLine; }
            $speakerPrefix = $speaker !== '' ? "SPEAKER_{$speaker}: " : '';
            $currentLine = "[{$ts}] {$speakerPrefix}{$segText}";
            $currentSpeaker = $speaker;
            $lastTsAt = $start;
        } else {
            $currentLine .= ' ' . $segText;
        }
    }
    if ($currentLine !== '') { $lines[] = $currentLine; }
    if (!empty($lines)) {
        return implode("\n", $lines);
    }
}

return $response['text'];  // Fallback

Example output for a real 2-speaker Swedish Talk recording (22 minutes, real conversation):

[00:00:00] SPEAKER_A: Tanken var så här...
[00:00:30] SPEAKER_A: Om vi har ett country field som är generiskt, som inte ska vara filtrerat...
[00:01:12] SPEAKER_B: Ja, men exakt.
[00:01:15] SPEAKER_A: Men tanken är så här, jag vill välja en leverantör...

Formatting choices in this sketch — any of them negotiable:

• [HH:MM:SS] prefix — whole seconds are sufficient for meeting-minute reading. Users needing subtitle-grade precision would still use response_format = 'srt' | 'vtt' directly.
• SPEAKER_A: labels only when speaker field is present — fully backward compatible when it isn't (older OpenAI models, non-diarizing providers).
• Consecutive same-speaker segments joined with spaces — avoids one-line-per-Whisper-chunk noise; segments already come pre-punctuated.
• Fresh [HH:MM:SS] every 30s within a long same-speaker turn — lets users navigate long monologues.

Why this isn't asking for admin configuration. Deliberately distinct from #405 (SummaryProvider prompt customization, declined as out of scope): this request asks for consistent output improvement for all users, not admin-level toggles. No .env variables, no UI settings, no per-deployment risk. Same output shape for Talk call recordings, Assistant "Transcribe audio", and every other STT path. Same behaviour whether the backend is LocalAI or cloud OpenAI. If a specific caller ever needs the plain text back, explode("\n", $text) and stripping [HH:MM:SS] SPEAKER_X: prefixes trivially recovers it — this is strictly additive to the current output.

Also affects the Assistant "Transcribe audio" task, not just Talk recordings. Users who paste audio into the Nextcloud Assistant's transcription tool will also get the timestamped, speaker-attributed output. Usually a plus — makes ad-hoc transcriptions much more useful.

Describe alternatives you've considered

1. Hard-patching OpenAiAPIService.php locally on every Nextcloud instance. What I'm doing now. Fragile: patch gets clobbered on every integration_openai app update, has to be manually re-applied. Not maintainable across a fleet.

2. Wrapping the STT endpoint on the shim side to inline timestamps and speaker labels into the plain text field. Technically works, but abuses the plain-text semantics of response_format = 'json' and forces the LocalAI-side toolchain to make Nextcloud-specific formatting decisions. Doesn't help users of the cloud OpenAI API.

3. Adding a new response format parameter and letting callers request AudioToTextProvider-style output. Overkill — it doubles the API surface for one output shape. Simpler to make the improved output the default and provide the escape hatch in the (unlikely) case a caller needs the flat text.

Related:

• #403 — RefreshModels cron hits api.openai.com because per-service URL fallback skips the global url (filed same day, still open)
• #405 — SummaryProvider hardcodes summary prompt (filed same day, declined as out of scope)

Start in lib/Service/OpenAiAPIService.php at OpenAiAPIService::transcribe(), especially the existing quota-accounting block and the response return around line 838. Preserve the segments data after quota accounting and format available timestamps and speaker labels, while retaining the text fallback; done means transcription callers receive the richer output for supported responses without segments failing.