apache / apache/incubator-pegasus
perf: seperate copy checkpoint from `learn` flow
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
Copy large checkpoint when learn has huge impact, In the past, we only think it because of the increase of `io write load`. However, I did two comparative tests and find the design of `nfs copy checkpoint` may be unreasonable.
### Test One: `restore` vs `learn` with same rate
In theroy, if io write rate is same, `restore` should has same impact with `learn`, but the test show different result:
| Rate(MB/s) | case | io util | P99 |
| -------- | ----- | -----: | :----: |
| 400 | restore | 20% | 50+ms |
| 400 | learn | 80% | 1000+ms |
From above, I found same io rate has different IO util and cause different latency. But if you think only IO utill casue the result, please see the next test
### Test Two: `learn` vs `no learn` but with same write rate and same io utils:
In this test, I keep the `io utils` and `write rate` same, the only difference is another test has no `learn` flow, we get the result:
| Rate(MB/s) | case | io util | P99 |
| -------- | ----- | -----: | :----: |
| 400 | no learn | 80% | 200+ms |
| 400 | learn | 80% | 1000+ms |
From above, I found that the latency of `io util load of 80%`caused only account for a small part, the large impact is casused by `learn` logic
## Conclusion
The reason of write latency has two part:
- io load increase: 200ms
- learn thread task queue: 800ms
So if we re-implement `learn checkpoint`, surpose current latency is 1000ms, and then:
1. if seprate copy checkpoint from `learn` flow in `write thread pool` so as to it won't competing thread resources, latency can decrease 1000ms to 200ms( see test2)
2. if improve copy checkpoint has same io utils impact with `restore`, latecny can decrease from 200ms to 50ms(see test1)
That is to say, actually, `copy checkpoint` should be a independent task like `restore`download file from
remote filesystem and only cause small io utils and little latency impact.
Contributor guide
Research direction
Start by locating the learn checkpoint-copy flow and the restore flow, then compare how each uses the write thread pool, task queue, and remote filesystem I/O. Done means the checkpoint copy is an independent task with the lower contention and latency impact described in the two comparative tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100