deepset-ai / deepset-ai/haystack
Investigate image support with Transformers
Open
Nobody has claimed this yet.
P3
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
We would like to use some models via Transformers that support multimodal user messages.
- We want to support (some) Image-Text-to-Text models
- The current component (
HuggingFaceLocalChatGenerator) might not easy/practical to extend, and it might make sense to develop a dedicated component - I would not give this investigation high-priority: for multimodal open models, it's better to first focus on Ollama that provides more standardization and does not require GPU; I would also expect users who have GPU to run vLLM (currently in Haystack it can be done via OpenAI with some limitations)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the HuggingFaceLocalChatGenerator component and the Transformers Image-Text-to-Text model requirements. Compare extending the current component with developing a dedicated component, while considering the stated Ollama and vLLM alternatives. Done means the supported scope and recommended implementation direction are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100