deepset-ai / deepset-ai/haystack-core-integrations

Support zero-downtime vectorization on PgvectorDocumentStore

Open
#1,696 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

integration:pgvector P2
Dominant language
Python
Stars
203
Forks
332
Avg merge
2d 4h
Merged PRs (30d)
80

Description

Is your feature request related to a problem? Please describe.

In our current use case, we need to always recreate the table when running vectorization and inserting new documents into PgvectorDocumentStore. Due to this, during the time window when this operation is ongoing, any RAG pipelines that depend on the impacted document store are "offline". We need to either turn off the impacted pipelines or be okay with partial data.

Describe the solution you'd like

What would be cool is if there was an option to "swap" document stores. I am imagining a process like this: When we need to run vectorization, we create a new document store as a "temp" store. We insert all new documents in this "temp store". When ready, we ask haystack to switch the real store with this temp store. And then delete the temp store. Behind the scenes, it's essentially doing PG table renames in an atomic way.

Note sure what the API would look like to be honest!

Describe alternatives you've considered
Given the current implementation of PgvectorDocumenstore, didn't find any way to swap two stores.

Additional context
An alternative we could try would be to just insert documents in the store with the correct overwrite policy. The challenge is that in our setup, we don't have stable ids for the documents. So we can't reliably de-dup new inserts into the store.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the current PgvectorDocumentStore implementation and its table creation, vectorization, and insertion flow. Define how a temporary store can be populated and atomically swapped while pipelines remain available, including cleanup and failure behavior. Done means the API and zero-downtime swap behavior are specified and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.