Optional Weaviate native multi-tenancy, vector compression, and advanced index options
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Summary
The Weaviate vector backend currently uses one collection per dataset with no multi-tenancy, no vector compression, and no configurable index/distance settings. This proposes adding these as **opt-in, default-off** options.
### Motivation
- At scale, one-collection-per-dataset creates thousands of collections. Weaviate native multi-tenancy (one shared collection, many isolated tenants) is the intended pattern for this.
- Vector compression (RQ/PQ/BQ/SQ) and index-type selection (hnsw/flat/dynamic) reduce memory/storage cost and are currently unavailable.
### Proposed change (all default-off, backward compatible)
- `WEAVIATE_MULTI_TENANCY_ENABLED`: new datasets sharing an embedding model become tenants in one shared collection (tenant = dataset id). Existing datasets keep their persisted layout (recorded in `index_struct`).
- `WEAVIATE_COMPRESSION` (rq/pq/bq/sq) + tuning knobs; flat index restricted to bq.
- `WEAVIATE_INDEX_TYPE` (hnsw/flat/dynamic) + `WEAVIATE_DYNAMIC_INDEX_THRESHOLD` + `WEAVIATE_DISTANCE_METRIC`. Dynamic index requires `ASYNC_INDEXING=true` on the server (added to compose).
- Optional `WEAVIATE_REPLICATION_FACTOR`.
### Scope
All changes confined to `api/providers/vdb/vdb-weaviate/` + config + compose env. Existing behavior is unchanged when flags are off. Depends on the 1.38/client-4.22 upgrade (#38211).
I'd like to work on this and have a PR ready.
Contributor guide
Research direction
Start by reviewing api/providers/vdb/vdb-weaviate/ alongside the config and compose environment changes, then check dependency upgrade issue #38211. Trace how index_struct records existing dataset layouts and verify that all new options remain default-off and backward compatible; completion should cover the listed tenancy, compression, index, distance, async-indexing, and replication settings.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100