pathwaycom / pathwaycom/pathway
External vector database indexing should be exposed as connectors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Currently, Pathway supports primitives for indexing vectors and performing vector search. This indexing has two main types:
- Internal indexes implemented within Pathway, such as in-memory indexes;
- External indexes, such as QDrant.
When indexing vectors into external storages, it makes sense to expose these indexes as standard output connectors, while still allowing users to use the existing indexing API.
Exposing these indexes externally provides several benefits:
- A valid use case exists where users simply read certain indexes and want to save and reuse them in another pipeline;
- It improves discoverability of the functionality for indexing data into external storage from Pathway.
Describe the solution you'd like
Introduce aliases for external indexes in Pathway that allow them to be accessed as standard output connectors (pw.io.*). Users can then perform indexing operations via these output connectors without losing access to the original API.
Describe alternatives you've considered
None.
Additional context
The existing indexing approach should remain unchanged; this change only adds aliases. Therefore, this is a non-breaking change and does not affect backward compatibility.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Pathway's existing vector indexing API and the standard output connector surface under pw.io.*. Determine how external indexes such as QDrant can be exposed through connector aliases while preserving the existing indexing API; done means both access paths remain available without breaking compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100