IO write throttle rate will beyond the Ceiling(1024MB/s) in the merge [LUCENE-10265]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
It's known that merge io write throttle rate is under the control of `targetMBPerSec` In ConcurrentMergeSchedule, it should't beyond the Ceiling(1024MB/s).
`targetMBPerSec` is shared by many merge threads, it will be changed by the way:
```java
if (newBacklog) {
// This new merge adds to the backlog: increase IO throttle by 20%
targetMBPerSec *= 1.20;
if (targetMBPerSec > MAX_MERGE_MB_PER_SEC) {
targetMBPerSec = MAX_MERGE_MB_PER_SEC;
}
......
} else {
// We are not falling behind: decrease IO throttle by 10%
targetMBPerSec /= 1.10;
if (targetMBPerSec < MIN_MERGE_MB_PER_SEC) {
targetMBPerSec = MIN_MERGE_MB_PER_SEC;
}
......
}
```
The modification process is not a atomic operation:
1. `targetMBPerSec` is changed by the first merge thread from 1024 to 1024\*1.2
1. other merge thread will read the new value(1024\*1.2).
1. the first merge thread limit the value to be 1024.
The bad case will happen.
In product, we do find that IO write throttle rate is beyond the Ceiling(1024MB/s) in the merge.
```java
[2021-11-26T15:27:19,861][TRACE][o.e.i.e.E.MS ] [data1] [test1][25] elasticsearch[data1][refresh][T#5] MS: io throttle: current merge backlog; leave IO rate at 3589.1 MB/sec
[2021-11-26T15:27:20,304][TRACE][o.e.i.e.E.MS ] [data1] [test1][13] elasticsearch[data1][write][T#3] MS: io throttle: current merge backlog; leave IO rate at 192.4 MB/sec
[2021-11-26T15:27:25,330][TRACE][o.e.i.e.E.MS ] [data1] [test1][22] elasticsearch[data1][[test1][22]: Lucene Merge Thread #1026] MS: io throttle: current merge backlog; leave IO rate at 96.3 MB/sec
[2021-11-26T15:27:25,995][TRACE][o.e.i.e.E.MS ] [data1] [test1][16] elasticsearch[data1][[test1][16]: Lucene Merge Thread #1063] MS: io throttle: current merge backlog; leave IO rate at 419.2 MB/sec
[2021-11-26T15:27:38,335][TRACE][o.e.i.e.E.MS ] [data1] [test1][19] elasticsearch[data1][write][T#2] MS: io throttle: current merge backlog; leave IO rate at 3051.5 MB/sec
```
If we shoud do the following:
1. changing it by the atomic operation.
2. adding the `volatile` attribute to `targetMBPerSec`.
---
Migrated from [LUCENE-10265](https://issues.apache.org/jira/browse/LUCENE-10265) by kkewwei (@kkewwei), updated Nov 28 2021
Contributor guide
Research direction
Start by locating and reading ConcurrentMergeSchedule, especially the targetMBPerSec update paths and the MAX_MERGE_MB_PER_SEC checks. Reproduce or inspect the concurrent merge-throttle behavior described in the issue; done means the shared rate remains at or below 1024 MB/sec during concurrent updates and the change is safely visible across merge threads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100