deepset-ai / deepset-ai/haystack
OpenAIGenerator uses chat_completions endpoint. Error with model that has no chat_template in config
@vblagoje is already working on this.
Since Sep 23, 2024.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Describe the bug
I'm using the OpenAIGenerator to access a vLLM endpoint on runpod. When using a base model like Mistral v0.3 that has not been instruction tuned and so does not have a chat template in it's config for the tokenizer, I get an error returned from the api endpoint. Digging into this I see that the OpenAIGenerator uses the chat_completion/ endpoint for the OpenAIGenerator and not the completion/ endpoint. This means I've been unintentionally using a chat template with other models up to this point.
Error message
"Cannot use apply_chat_template() because tokenizer.chat_template is not set and no template argument was passed!"
Expected behavior
I expected for the completions/ api endpoint to be used and the hugging face model to not try to use apply_chat_template()
Additional context
I tried to use the client.completions method directly as a work around.
completion = generator.client.completions.create(model=generator.model, prompt="And then, something unexpected happened.", **generator.generation_kwargs)
The process on the server crashes with a 'NoneType' object has no attribute 'headers'.
System:
- Haystack version (commit or version number): haystack-ai==2.2.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.