apache / apache/shenyu

[BUG] AiProxy switches providers after partial SSE output and mixes stream content

Open
#7,021 2 comments 0 reactions 0 assignees View on GitHub
AI feature: plugin priority: high type: bug
Dominant language
Java
Stars
8.8k
Forks
3.1k
Avg merge
7d 1h
Merged PRs (30d)
85

Description

## Description

`AiProxyExecutorService.executeDirectStream` applies retry to the main provider Flux and then unconditionally switches to the fallback provider through `onErrorResume`.

If the main provider emits one or more chunks before failing, those chunks have already been written to the client SSE response. The fallback provider then starts a new completion from the beginning and appends its chunks to the already committed response.

This issue is distinct from #6647:

- #6647 tracks replaying the main upstream after a mid-stream retry.
- This issue tracks switching to a different fallback provider after partial output, producing one SSE response containing content from multiple completions/providers.

Related PR: #6341, reviewed at head `8fa93d7446e052f149d3740bbe7ff2052c375e23`.

## Affected code

- `shenyu-plugin-ai-proxy/.../AiProxyExecutorService.java`: `retryWhen(...).onErrorResume(...fallback...)` wraps the whole streaming Flux.
- `shenyu-plugin-ai-proxy/.../AiProxyPlugin.java`: the Flux is written directly to the committed SSE response.

## Reproduction

An equivalent Reactor sequence:

```text
main provider: emit "main-partial" -> error
retry: emit "main-partial" -> error
fallback: emit "fallback-completion"
```

produces:

```text
main-partial, main-partial, fallback-completion
```

Even after #6647 removes the retry replay, an unconditional fallback after the first emitted chunk would still produce:

```text
main-partial, fallback-completion
```

## Impact

- Clients receive a syntactically valid SSE stream with semantically corrupted, mixed completion content.
- Tool calls or structured output can be duplicated or combined across providers.
- Multiple providers may bill for the same client request.
- The response is already committed, so the gateway cannot safely replace earlier chunks.

## Expected behavior

Provider retry and fallback are allowed only before the first response chunk is emitted. After any chunk is sent, an upstream failure must terminate the current stream without starting another provider completion.

## Acceptance criteria

- [ ] Track whether the main stream has emitted a chunk.
- [ ] Allow retry/fallback only before the first emission.
- [ ] After partial emission, propagate/terminate the stream without invoking fallback.
- [ ] Add a regression test where the main provider emits a chunk and then errors while fallback is configured.
- [ ] Assert no duplicated prefix and no fallback chunk is appended after partial output.
- [ ] Retain the existing error-before-first-chunk fallback behavior.
- [ ] Coordinate the retry portion with #6647 rather than implementing two competing fixes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at AiProxyExecutorService.executeDirectStream and trace how the Flux reaches the committed SSE response in AiProxyPlugin. Review the retry and fallback handling alongside issue #6647, then add the regression case described in the acceptance criteria. Done means fallback still works before the first chunk, but a post-chunk failure ends the stream without replayed or fallback content.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.