deepset-ai / deepset-ai/haystack
Allowing full metadata filtering/search functionality for document stores like Qdrant
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I need to do metadata filtering for my RAG (w embedding retriever and re-ranker) pipeline. Haystack's current filtering is too limited for my use case.
Describe the solution you'd like
A clear and concise description of what you want to happen.
I would like to use Qdrant doc store since it's metadata filtering has the functionality I need, but Haystack's filtering is limited and does not allow me to use the Qdrant functionality I need. Is there any way that all the metadata filtering functionality of the document stores like Qdrant, Milvus, etc. can be exposed? Based on my limited review of the code, I think the limitation comes from the fact that the client is created within (not outside) the document store.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
I know that I can use all of Qdrant's metadata filtering functionality in Langchain. (Again, I think this is because the client can be created by the user along with some other info, and then the vector store is created from the client and embeddings -- one can find details/examples on Langchain). I would really like to use Haystack since based on my experience it has a better design (and is probably faster than Langchain?), and so far I've really liked what I see and have used from Haystack.
Instead of Qdrant, think I could use OpenSearch with a custom query, as described in Metadata to Customize Retrieval section of https://docs.cloud.deepset.ai/docs/use-metadata-in-your-search-system. However, I do not want to use AWS or any cloud-based service since I am trying to build a RAG app that is totally local. (I guess OpenSearch can be run locally, but the docker container uses up too much precious memory (about 2GB)) on my 16GB Windows machine).
Additional context
Add any other context or screenshots about the feature request here.
Thank you for your time and attention to my request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Haystack's document-store filtering API and its Qdrant and other document-store integrations, focusing on how metadata filters and clients are currently exposed. Compare the requested capabilities with Qdrant and Milvus filtering, then define what broader filtering support would cover and how completion would be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- database, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100