AnswerDotAI / AnswerDotAI/byaldi

Multimodal way to retrieve documents

Open
#81 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
851
Forks
91
PR merge metrics
No merged PRs in 30d

Description

Hello,
Thank you for the amazing library. I referred the RAG implementation notebooks from hugging face using this library and loved it. I am just wondering if there is a way to not just have a text prompt but feed in an image as well to retrieve multimodal documents or images.

To be specific the user query contains text as well as an image input to colpali that can retrieve top k similar docs which then can be used by the vlm including the prompt and the image for generating an output.

I am not sure byaldi supports that integration or even colpali does. Or is there any other way. Hope this is the right space to raise this issue or discussion!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.