[Tracking][Phase II] Improve optimize()/compaction performance and resource efficiency
- Dominant language
- C++
- Stars
- 15.9k
- Forks
- 998
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 34
Description
### Background
Phase I ([#98](https://github.com/alibaba/zvec/issues/98)) introduced a safe segment-reuse path for compaction:
- Reuse the first compatible vector index as the merge base.
- Append tail segments instead of rebuilding the complete index.
- Support FLAT, HNSW, and HNSW_RABITQ when there is no filter and schemas are compatible.
- Fall back to a full rebuild for filters, incompatible schemas, IVF, VAMANA, and other unsafe cases.
This significantly reduced compaction time for reusable paths. Phase II will build on that work and address the remaining rebuild, file-copy, memory, and I/O costs.
### Goals
- Make compaction cost scale primarily with the changed/tail data where possible.
- Reduce write amplification, temporary disk usage, and peak RSS.
- Expand incremental/reuse support to more index and compaction scenarios.
Tracking fixes and improvments here.
Contributor guide
Assessment
This issue has not been assessed yet.