[Docs] Guidance on setting `num_tokens_per_iter` and `max_prefill_iters` to optimal values
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### 📚 The doc issue
I noticed that by default the prefill completely stalls token generation for existing requests. So if a machine gets a couple of requests with big prompts (that are not prefix-cached), then it can cause a long lag/pause for other users that were streaming from that machine.
Looking into this, I found the following options which seem like they may help solve this problem:
```
num_tokens_per_iter(int): the number of tokens processed in each forward pass. Working with `max_prefill_iters` enables "Dynamic SplitFuse"-like scheduling
max_prefill_iters(int): the max number of forward pass during prefill stage
```
Following the reference to "Dynamic SplitFuse" I found this page:
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-fastgen/README.md
But I wasn't able to find an explanation that corresponds to `num_tokens_per_iter` and `max_prefill_iters`. Could an explanation be added somewhere?
Contributor guide
Assessment
This issue has not been assessed yet.