[Feature Request] Support for Huggingface Chat Templates
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Now that many newer Huggingface models come with a chat template in their tokenizer, FastChat should use it as the primary way to build conversations, falling back to conversation.py when a template isn't available.
This would reduce the burden to add conversation templates every time a new popular model is released. For example, Llama 3 could be supported right on release this way.
Relevant links:
https://huggingface.co/docs/transformers/v4.34.0/en/chat_templating
https://github.com/vllm-project/vllm/issues/1361
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with conversation.py and the Hugging Face chat templating documentation linked in the issue. Trace how FastChat currently builds conversations, then determine where tokenizer chat templates can take priority while preserving the existing fallback. Done means supported models use their tokenizer template and models without one continue using conversation.py.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100