apache / apache/paimon-vector-index

Support reusing a training result across multiple vector index writers

Open
#86 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21
Forks
21
Avg merge
2d 10h
Merged PRs (30d)
10

Description

## Motivation

Callers that build one vector index file per segment currently need a separate training pass for every writer. The trained centroids, quantizers, and rotations can often be shared when those segments come from the same table or data distribution, but the unified API consumes `VectorIndexTraining` when it creates a writer.

For `N` segments, this makes the lifecycle effectively:

```text
train -> writer 1
train -> writer 2
...
train -> writer N
```

IVF-PQ already has an internal `from_trained` path, but this capability is not available through the unified API or consistently across the supported index types and language bindings.

## Proposed API

Allow a completed training result to create multiple independent writers:

```rust
let training = VectorIndexTrainer::train(config, training_vectors, training_count)?;
let first = training.create_writer();
let second = training.create_writer();
```

Each writer should start with the same trained model and an empty vector/ID payload. The existing consuming APIs should remain unchanged for one-shot callers.

Equivalent opt-in entry points should be exposed through C, C++, Java/JNI, and Python.

## Expected benefit

In a release-mode synthetic build with 128-dimensional vectors, 8,192 training rows, and eight segments of 2,048 vectors each, training once reduced writer preparation time by 84-87% across IVF-Flat, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN. Including vector insertion and serialization, elapsed time fell by 73-84%.

The benefit comes from reducing `N` training passes to one; it does not change the index algorithms themselves.

## Compatibility and scope

- No default behavior change: callers opt in to reusable training explicitly.
- Existing consuming APIs remain available.
- No on-disk format change.
- No cross-process training-model serialization or index merging in this issue.

Contributor guide

Open the contributing guide

Research direction

Start by tracing VectorIndexTrainer::train, the proposed training.create_writer() entry point, and IVF-PQ's existing from_trained path across the supported index types. Then inspect the unified API and its C, C++, Java/JNI, and Python bindings. Done means independent writers reuse one completed training result while retaining empty payloads, existing consuming APIs, and the current on-disk format.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp, java, python, rust
Domain
api, data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.