opensearch-project / opensearch-project/OpenSearch
[Segment Replication] Replicate *.liv file may cause performance issue
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
In document replication scenario, *.liv file are only be written to disk when flush operation (Lucene commit) is performed. But in segment replication, the *.liv file must be written every refresh (by setting writeAllDeletes to true in DirectoryReader#open method).
It may not cause any problem in append-only scenario (no delete will be issued in old segment). But in update scenario, as long as there is a delete operation performed in old segment, primary shard's refresh will write full liv bitmap to disk and replicate to replica shard.
*.liv file may be very large in some merged segment (for example *.liv file for a segment with 16,000,000 docs takes up ~2MB disk space). Differ with segment data file, we cannot reuse old *.liv file when new *.liv file is generated, even if only one doc is deleted in segment, we must replicate the full *.liv file. So in segrep, write and replicate *.liv file may introduce greater network and CPU (write and load *.liv file) load.
Several ways to fix this issue:
- Write diff other than full
bitmapwhenrefreshis performed - Compress liv doc file with LZ4 or zstd
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DirectoryReader#open entry point and the segment replication refresh path described in the issue, focusing on writeAllDeletes and generation of *.liv files. Compare the proposed diff-based and compressed approaches, then verify that refreshes reduce unnecessary *.liv disk, CPU, and network load without breaking replication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, performance, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100