deepset-ai / deepset-ai/haystack

Allowing full metadata filtering/search functionality for document stores like Qdrant

Open
#9,138 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P3
Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I need to do metadata filtering for my RAG (w embedding retriever and re-ranker) pipeline. Haystack's current filtering is too limited for my use case.

Describe the solution you'd like
A clear and concise description of what you want to happen.

I would like to use Qdrant doc store since it's metadata filtering has the functionality I need, but Haystack's filtering is limited and does not allow me to use the Qdrant functionality I need. Is there any way that all the metadata filtering functionality of the document stores like Qdrant, Milvus, etc. can be exposed? Based on my limited review of the code, I think the limitation comes from the fact that the client is created within (not outside) the document store.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

I know that I can use all of Qdrant's metadata filtering functionality in Langchain. (Again, I think this is because the client can be created by the user along with some other info, and then the vector store is created from the client and embeddings -- one can find details/examples on Langchain). I would really like to use Haystack since based on my experience it has a better design (and is probably faster than Langchain?), and so far I've really liked what I see and have used from Haystack.

Instead of Qdrant, think I could use OpenSearch with a custom query, as described in Metadata to Customize Retrieval section of https://docs.cloud.deepset.ai/docs/use-metadata-in-your-search-system. However, I do not want to use AWS or any cloud-based service since I am trying to build a RAG app that is totally local. (I guess OpenSearch can be run locally, but the docker container uses up too much precious memory (about 2GB)) on my 16GB Windows machine).

Additional context
Add any other context or screenshots about the feature request here.

Thank you for your time and attention to my request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Haystack's document-store filtering API and its Qdrant and other document-store integrations, focusing on how metadata filters and clients are currently exposed. Compare the requested capabilities with Qdrant and Milvus filtering, then define what broader filtering support would cover and how completion would be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
database, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.