jd-opensource / jd-opensource/JoyAI-VL-Interaction
Streaming adapter can accumulate failed frames past context and image limits
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 190
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The WebInfer streaming adapter appends each incoming frame to `current_chunk` before calling the main model. If that call fails (for example because the prompt exceeds the model context window), the appended turn remains in session state. Retrying then appends the frame again, so the prompt keeps growing until the backend's multimodal item limit is also exceeded.
Observed failure sequence with an 8,192-token backend context and a 32-image vLLM limit:
1. The prompt first fails with `Input length (...) exceeds ... 8192`.
2. Each retry adds more images to the same session.
3. The backend eventually returns `At most 32 image(s) may be provided in one prompt`.
4. The adapter exposes the backend 400 as a 502 response.
The current `CHUNK` setting bounds summary cadence, but async-summary carry-over can still leave enough unsummarized turns for the main-model prompt to exceed a backend-specific context or image limit.
## Expected behavior
- Deployments should be able to configure a recent-turn/recent-image window for main-model prompts without discarding full session history used by summaries and long-term memory.
- A failed main-model call should not permanently append the failed turn, so a retry starts from the same prior state.
- Existing deployments should retain their current behavior unless the optional limits are enabled.
## Proposed fix
Add opt-in `MAIN_MAX_PROMPT_TURNS` and `MAIN_MAX_PROMPT_IMAGES` settings (both defaulting to `0`, unlimited), and snapshot/restore append-only turn state around main-model calls.
I have a tested implementation ready and will link the pull request here.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the WebInfer streaming adapter's current_chunk append and main-model call, then inspect how CHUNK and session state are maintained. Check the proposed settings and append-only state around retries; done means optional turn/image limits work without changing defaults and a failed call leaves the prior state intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100