deepset-ai / deepset-ai/haystack-core-integrations
Support zero-downtime vectorization on PgvectorDocumentStore
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 332
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 80
Description
Is your feature request related to a problem? Please describe.
In our current use case, we need to always recreate the table when running vectorization and inserting new documents into PgvectorDocumentStore. Due to this, during the time window when this operation is ongoing, any RAG pipelines that depend on the impacted document store are "offline". We need to either turn off the impacted pipelines or be okay with partial data.
Describe the solution you'd like
What would be cool is if there was an option to "swap" document stores. I am imagining a process like this: When we need to run vectorization, we create a new document store as a "temp" store. We insert all new documents in this "temp store". When ready, we ask haystack to switch the real store with this temp store. And then delete the temp store. Behind the scenes, it's essentially doing PG table renames in an atomic way.
Note sure what the API would look like to be honest!
Describe alternatives you've considered
Given the current implementation of PgvectorDocumenstore, didn't find any way to swap two stores.
Additional context
An alternative we could try would be to just insert documents in the store with the correct overwrite policy. The challenge is that in our setup, we don't have stable ids for the documents. So we can't reliably de-dup new inserts into the store.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the current PgvectorDocumentStore implementation and its table creation, vectorization, and insertion flow. Define how a temporary store can be populated and atomically swapped while pipelines remain available, including cleanup and failure behavior. Done means the API and zero-downtime swap behavior are specified and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100