how to interrupt generating task and release gpu computing resources (not include kv cache) ?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
when using an HTTP request to call a streaming generation interface for content generation, if the user interrupts (for example, in a streaming conversation, the user speaks and interrupts the robot's speech), it is necessary to terminate the backend generation task from continuing (the remaining tokens will no longer be used) and release GPU computing resources (the KV cache needs to be retained).
How can I implement this requirement? If I just exit the backend's async for loop in openai-server.py midway, will the GPU computing resources be released?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading openai-server.py and tracing the streaming generation path around its async for loop. Identify how an interrupted HTTP request is represented and where GPU work and KV-cache lifetime are managed. Done means documenting or implementing a clear cancellation path that stops remaining generation while preserving the KV cache, with the relevant behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100