apache / apache/paimon-vector-index
Support reusing a training result across multiple vector index writers
- Dominant language
- Rust
- Stars
- 21
- Forks
- 21
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 10
Description
## Motivation
Callers that build one vector index file per segment currently need a separate training pass for every writer. The trained centroids, quantizers, and rotations can often be shared when those segments come from the same table or data distribution, but the unified API consumes `VectorIndexTraining` when it creates a writer.
For `N` segments, this makes the lifecycle effectively:
```text
train -> writer 1
train -> writer 2
...
train -> writer N
```
IVF-PQ already has an internal `from_trained` path, but this capability is not available through the unified API or consistently across the supported index types and language bindings.
## Proposed API
Allow a completed training result to create multiple independent writers:
```rust
let training = VectorIndexTrainer::train(config, training_vectors, training_count)?;
let first = training.create_writer();
let second = training.create_writer();
```
Each writer should start with the same trained model and an empty vector/ID payload. The existing consuming APIs should remain unchanged for one-shot callers.
Equivalent opt-in entry points should be exposed through C, C++, Java/JNI, and Python.
## Expected benefit
In a release-mode synthetic build with 128-dimensional vectors, 8,192 training rows, and eight segments of 2,048 vectors each, training once reduced writer preparation time by 84-87% across IVF-Flat, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN. Including vector insertion and serialization, elapsed time fell by 73-84%.
The benefit comes from reducing `N` training passes to one; it does not change the index algorithms themselves.
## Compatibility and scope
- No default behavior change: callers opt in to reusable training explicitly.
- Existing consuming APIs remain available.
- No on-disk format change.
- No cross-process training-model serialization or index merging in this issue.
Contributor guide
Research direction
Start by tracing VectorIndexTrainer::train, the proposed training.create_writer() entry point, and IVF-PQ's existing from_trained path across the supported index types. Then inspect the unified API and its C, C++, Java/JNI, and Python bindings. Done means independent writers reuse one completed training result while retaining empty payloads, existing consuming APIs, and the current on-disk format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp, java, python, rust
- Domain
- api, data-engineering, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100