api7 / api7/aisix

Streaming-path parity for gpt-4o-audio: surface `delta.audio` (follow-up to #517)

Open
#518 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug P1 real chain test
Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Context / follow-up to #517

#517 surfaced gpt-4o-audio's message.audio on the non-stream /v1/chat/completions response. The streaming path still drops audio:

  • OpenAiStreamDelta (crates/aisix-provider-openai/src/wire.rs) does not model audio.
  • ChatDelta has no extra passthrough.
  • RenderedDelta (crates/aisix-proxy/src/render.rs) has no audio/extra.

OpenAI streams gpt-4o-audio output via choices[].delta.audio ({ id?, data?, transcript?, expires_at? } incremental chunks). With the streaming structs not modeling it, a customer streaming a gpt-4o-audio response gets no audio deltas.

This is the same non-stream-vs-stream split as reasoning (#466 non-stream → #502 streaming parity).

Ask

Surface delta.audio on the streaming path: model it on OpenAiStreamDelta, carry it through ChatDelta (likely via an extra flatten mirroring the message-level approach), and emit it on RenderedDelta so streamed gpt-4o-audio audio reaches the client.

Verification

  • Mock-parity: extend the mock to stream delta.audio chunks and assert the gateway forwards them.
  • Real-chain: a streaming variant of the AISIX-Cloud PR-K (#677) gpt-4o-audio spec.

Refs

  • non-stream fix: #517
  • product issue: api7/AISIX-Cloud#684

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OpenAiStreamDelta in crates/aisix-provider-openai/src/wire.rs, then trace ChatDelta and RenderedDelta in crates/aisix-proxy/src/render.rs alongside the non-stream fix in #517. Extend the mock-parity coverage and the streaming AISIX-Cloud spec so incremental delta.audio chunks are forwarded to the client.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.