zilliztech / zilliztech/VectorDBBench
[Question/Enhancement] Unconditional force-merge compaction in Milvus client `_optimize()` may be too costly and non-reproducible at 100M scale (LAION-100M)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 438
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 1
Description
Summary
In the Milvus client, _optimize() unconditionally triggers a force-merge compaction
before search for all non-GPU index types:
MILVUS_FORCE_MERGE_TARGET_SIZE_MB = ((1 << 63) - 1) // (1024**2) # line 21
...
self._wait_for_segments_sorted()
self._wait_for_index()
compaction_id = self.client.compact(
self.collection_name, target_size=MILVUS_FORCE_MERGE_TARGET_SIZE_MB
)
Concerns at LAION-100M scale
I'd like to ask whether this is intended to run at 100M-vector scale (e.g. the
Performance768D100M / LAION-100M case), and whether it should be made configurable.
LAION-100M is 100M × 768-dim float32 ≈ 307 GB of raw vector data.
- Compaction cost: force merge rewrites the segments; at 300 GB+ this is a very long,
IO/CPU-heavy operation on DataNodes. _optimize() is supposed to be a "prepare for
search" step, but here it becomes a multi-hour data rewrite. - Index rebuild memory: after merging into much larger segments, the vector index is
rebuilt per segment. Index construction memory scales with segment size, so very large
merged segments raise real OOM risk on the index/data node — especially for
memory-hungry builds (HNSW, DiskANN).
Questions
1、Was unconditional force merge validated on the 100M-scale cases, and if so, what was
the observed compaction duration and peak DataNode / IndexNode memory?
2、Is the intent to normalize segment layout for fair comparison? If yes, wouldn't an
explicit, fixed target_size be more reproducible than the memory-dependent auto mode?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with vectordb_bench/backend/clients/milvus/milvus.py, especially MILVUS_FORCE_MERGE_TARGET_SIZE_MB near line 21 and _optimize() near lines 312-316. Trace the benchmark paths for Performance768D100M and LAION-100M, then gather the requested compaction-duration and peak-memory evidence. Done should be a maintainer-approved resolution for the force-merge behavior and its reproducibility or configurability at large scale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100