prompt will always be truncated
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
when call "/v1/chat/completions", there will call the function
[check_length](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L437) to compute `max_new_tokens` use [min(max_tokens, context_len - token_num)](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L169) where [token_num](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/base_model_worker.py#L152) is len(tokinzer(pormot).input_ids),
but when compute `max_src_len` use `max_src_len = context_len - max_new_tokens - 1` in [inference.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/inference.py#L97), this will lead to truncate the prompt every time.
such as context_len=4096, token_num=len(tokenizer(promot).input_ids)=8, max_new_tokens = 4096-8= 4088
then max_src_len = context_len - max_new_tokens - 1 = 4096-4088-1=7
when use `input_ids = input_ids[-max_src_len:]` to truncate the prompt, this will drop the first token
all links:
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L437
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/base_model_worker.py#L152
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/inference.py#L97
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L169
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked check_length and max_new_tokens logic in fastchat/serve/openai_api_server.py and base_model_worker.py, then inspect the max_src_len calculation in fastchat/serve/inference.py. Reproduce the reported context_len=4096 and token_num=8 example and trace the resulting input_ids slice. Done means the prompt is not unnecessarily truncated while the completion token limit remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100