ByteDance-Seed / ByteDance-Seed/FlexPrefill

Require Support For DeepSeek-MLA and batchsize > 1

Open
#18 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
172
Forks
11
PR merge metrics
No merged PRs in 30d

Description

FlexPrefill is an amazing work for longtxt tasks

Thanks for the authors, we've make it into sglang and save 32% time for RULER and NIAH benchmarks, Qwen3-235B-FP8 on 4 H20.

But now we have 3 problems, looking for help

* Support DeepSeek-MLA
We've adjust the dim to meet the MLA's [B * N, H, 192], [B * N, H, 192], [B * N, H, 128] in sglang's `forward_extend`, but the RULER accuracy seems drop so hard, (the niah_multikey_1 drops from 1.0 to 0.2 ... )

* Support batchsize > 1
In longtxt scenes, usually the batch_size is not too large, but usually goes to 2 or 4, so the batch_size > 1 is requiring support.

* Compatibality with sglang's --chunked-prefix-cache feature
Another thing is about the sglang's --chunked-prefix-cache feature, it requires Attention kernel to return a `lse` to support cache for long sequence, while FlexPrefill kernel does NOT return lse. Does FlexPrefill can work with this feature ?

Can you support for these 3 problems ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.