alibaba / alibaba/zvec

[Tracking][Phase II] Improve optimize()/compaction performance and resource efficiency

Open
#728 0 comments 0 reactions 1 assignee Claimed by @feihongxu0824 View on GitHub
Dominant language
C++
Stars
15.9k
Forks
998
Avg merge
4d 7h
Merged PRs (30d)
34

Description

### Background

Phase I ([#98](https://github.com/alibaba/zvec/issues/98)) introduced a safe segment-reuse path for compaction:

- Reuse the first compatible vector index as the merge base.
- Append tail segments instead of rebuilding the complete index.
- Support FLAT, HNSW, and HNSW_RABITQ when there is no filter and schemas are compatible.
- Fall back to a full rebuild for filters, incompatible schemas, IVF, VAMANA, and other unsafe cases.

This significantly reduced compaction time for reusable paths. Phase II will build on that work and address the remaining rebuild, file-copy, memory, and I/O costs.

### Goals

- Make compaction cost scale primarily with the changed/tail data where possible.
- Reduce write amplification, temporary disk usage, and peak RSS.
- Expand incremental/reuse support to more index and compaction scenarios.

Tracking fixes and improvments here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.