langgenius / langgenius/dify

Optional Weaviate native multi-tenancy, vector compression, and advanced index options

Open
#38,212 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Summary

The Weaviate vector backend currently uses one collection per dataset with no multi-tenancy, no vector compression, and no configurable index/distance settings. This proposes adding these as **opt-in, default-off** options.

### Motivation

- At scale, one-collection-per-dataset creates thousands of collections. Weaviate native multi-tenancy (one shared collection, many isolated tenants) is the intended pattern for this.
- Vector compression (RQ/PQ/BQ/SQ) and index-type selection (hnsw/flat/dynamic) reduce memory/storage cost and are currently unavailable.

### Proposed change (all default-off, backward compatible)

- `WEAVIATE_MULTI_TENANCY_ENABLED`: new datasets sharing an embedding model become tenants in one shared collection (tenant = dataset id). Existing datasets keep their persisted layout (recorded in `index_struct`).
- `WEAVIATE_COMPRESSION` (rq/pq/bq/sq) + tuning knobs; flat index restricted to bq.
- `WEAVIATE_INDEX_TYPE` (hnsw/flat/dynamic) + `WEAVIATE_DYNAMIC_INDEX_THRESHOLD` + `WEAVIATE_DISTANCE_METRIC`. Dynamic index requires `ASYNC_INDEXING=true` on the server (added to compose).
- Optional `WEAVIATE_REPLICATION_FACTOR`.

### Scope

All changes confined to `api/providers/vdb/vdb-weaviate/` + config + compose env. Existing behavior is unchanged when flags are off. Depends on the 1.38/client-4.22 upgrade (#38211).

I'd like to work on this and have a PR ready.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing api/providers/vdb/vdb-weaviate/ alongside the config and compose environment changes, then check dependency upgrade issue #38211. Trace how index_struct records existing dataset layouts and verify that all new options remain default-off and backward compatible; completion should cover the listed tenancy, compression, index, distance, async-indexing, and replication settings.

Written by the indexing model from the issue text.

Assessment

Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.