apache / apache/lucene

IO write throttle rate will beyond the Ceiling(1024MB/s) in the merge [LUCENE-10265]

Open
#11,301 2 comments 0 reactions 0 assignees View on GitHub
affects-version:8.6.2 legacy-jira-priority:Major module:core/other type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

It's known that merge io write throttle rate is under the control of `targetMBPerSec` In ConcurrentMergeSchedule, it should't beyond the Ceiling(1024MB/s).

`targetMBPerSec` is shared by many merge threads, it will be changed by the way:

```java
if (newBacklog) {
// This new merge adds to the backlog: increase IO throttle by 20%
targetMBPerSec *= 1.20;
if (targetMBPerSec > MAX_MERGE_MB_PER_SEC) {
targetMBPerSec = MAX_MERGE_MB_PER_SEC;
}
......
} else {
// We are not falling behind: decrease IO throttle by 10%
targetMBPerSec /= 1.10;
if (targetMBPerSec < MIN_MERGE_MB_PER_SEC) {
targetMBPerSec = MIN_MERGE_MB_PER_SEC;
}
......
}
```

The modification process is not a atomic operation:
1. `targetMBPerSec` is changed by the first merge thread from 1024 to 1024\*1.2
1. other merge thread will read the new value(1024\*1.2).
1. the first merge thread limit the value to be 1024.

The bad case will happen.

In product, we do find that IO write throttle rate is beyond the Ceiling(1024MB/s) in the merge.

```java
[2021-11-26T15:27:19,861][TRACE][o.e.i.e.E.MS ] [data1] [test1][25] elasticsearch[data1][refresh][T#5] MS: io throttle: current merge backlog; leave IO rate at 3589.1 MB/sec
[2021-11-26T15:27:20,304][TRACE][o.e.i.e.E.MS ] [data1] [test1][13] elasticsearch[data1][write][T#3] MS: io throttle: current merge backlog; leave IO rate at 192.4 MB/sec
[2021-11-26T15:27:25,330][TRACE][o.e.i.e.E.MS ] [data1] [test1][22] elasticsearch[data1][[test1][22]: Lucene Merge Thread #1026] MS: io throttle: current merge backlog; leave IO rate at 96.3 MB/sec
[2021-11-26T15:27:25,995][TRACE][o.e.i.e.E.MS ] [data1] [test1][16] elasticsearch[data1][[test1][16]: Lucene Merge Thread #1063] MS: io throttle: current merge backlog; leave IO rate at 419.2 MB/sec
[2021-11-26T15:27:38,335][TRACE][o.e.i.e.E.MS ] [data1] [test1][19] elasticsearch[data1][write][T#2] MS: io throttle: current merge backlog; leave IO rate at 3051.5 MB/sec
```

If we shoud do the following:
1. changing it by the atomic operation.
2. adding the `volatile` attribute to `targetMBPerSec`.

---
Migrated from [LUCENE-10265](https://issues.apache.org/jira/browse/LUCENE-10265) by kkewwei (@kkewwei), updated Nov 28 2021

Contributor guide

Open the contributing guide

Research direction

Start by locating and reading ConcurrentMergeSchedule, especially the targetMBPerSec update paths and the MAX_MERGE_MB_PER_SEC checks. Reproduce or inspect the concurrent merge-throttle behavior described in the issue; done means the shared rate remains at or below 1024 MB/sec during concurrent updates and the change is safely visible across merge threads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.