ByteDance-Seed / ByteDance-Seed/FlexPrefill
Require Support For DeepSeek-MLA and batchsize > 1
- Dominant language
- Python
- Stars
- 172
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
FlexPrefill is an amazing work for longtxt tasks
Thanks for the authors, we've make it into sglang and save 32% time for RULER and NIAH benchmarks, Qwen3-235B-FP8 on 4 H20.
But now we have 3 problems, looking for help
* Support DeepSeek-MLA
We've adjust the dim to meet the MLA's [B * N, H, 192], [B * N, H, 192], [B * N, H, 128] in sglang's `forward_extend`, but the RULER accuracy seems drop so hard, (the niah_multikey_1 drops from 1.0 to 0.2 ... )
* Support batchsize > 1
In longtxt scenes, usually the batch_size is not too large, but usually goes to 2 or 4, so the batch_size > 1 is requiring support.
* Compatibality with sglang's --chunked-prefix-cache feature
Another thing is about the sglang's --chunked-prefix-cache feature, it requires Attention kernel to return a `lse` to support cache for long sequence, while FlexPrefill kernel does NOT return lse. Does FlexPrefill can work with this feature ?
Can you support for these 3 problems ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.