opensearch-project / opensearch-project/OpenSearch

[Segment Replication] Replicate *.liv file may cause performance issue

Open
#3,929 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Indexing:Replication lucene
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

In document replication scenario, *.liv file are only be written to disk when flush operation (Lucene commit) is performed. But in segment replication, the *.liv file must be written every refresh (by setting writeAllDeletes to true in DirectoryReader#open method).

It may not cause any problem in append-only scenario (no delete will be issued in old segment). But in update scenario, as long as there is a delete operation performed in old segment, primary shard's refresh will write full liv bitmap to disk and replicate to replica shard.

*.liv file may be very large in some merged segment (for example *.liv file for a segment with 16,000,000 docs takes up ~2MB disk space). Differ with segment data file, we cannot reuse old *.liv file when new *.liv file is generated, even if only one doc is deleted in segment, we must replicate the full *.liv file. So in segrep, write and replicate *.liv file may introduce greater network and CPU (write and load *.liv file) load.

Several ways to fix this issue:

  1. Write diff other than full bitmap when refresh is performed
  2. Compress liv doc file with LZ4 or zstd

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the DirectoryReader#open entry point and the segment replication refresh path described in the issue, focusing on writeAllDeletes and generation of *.liv files. Compare the proposed diff-based and compressed approaches, then verify that refreshes reduce unnecessary *.liv disk, CPU, and network load without breaking replication.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, performance, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.