multi-slot KV like llama.cpp
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 152
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 11
Description
Hi,
Even with the patch ( https://github.com/FastFlowLM/FastFlowLM/issues/486 ) we only have 1 slot avalable for concurrency. Is there any plan to introduce multi-slot for FLM ? This will permit to have multi-task with different prompt memories, using the huge prefill speed that you provide.
Thank you for your hard work. Happy to see that you joined AMD?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #486 and tracing how the current single KV slot limits concurrent tasks in the C++ codebase. Define how multiple slots should preserve separate prompt memories for concurrent requests, then validate that distinct tasks can run concurrently with the expected prefill behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100