InternLM / InternLM/lmdeploy

[Docs] Guidance on setting `num_tokens_per_iter` and `max_prefill_iters` to optimal values

Open
#1,740 7 comments 0 reactions 1 assignee Claimed by @lzhangzz View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### 📚 The doc issue

I noticed that by default the prefill completely stalls token generation for existing requests. So if a machine gets a couple of requests with big prompts (that are not prefix-cached), then it can cause a long lag/pause for other users that were streaming from that machine.

Looking into this, I found the following options which seem like they may help solve this problem:
```
num_tokens_per_iter(int): the number of tokens processed in each forward pass. Working with `max_prefill_iters` enables "Dynamic SplitFuse"-like scheduling
max_prefill_iters(int): the max number of forward pass during prefill stage
```

Following the reference to "Dynamic SplitFuse" I found this page:

https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-fastgen/README.md

But I wasn't able to find an explanation that corresponds to `num_tokens_per_iter` and `max_prefill_iters`. Could an explanation be added somewhere?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.