apache / apache/bookkeeper

Adjust checkpoint timing adaptively according to throughput

Open
#2,896 0 comments 0 reactions 0 assignees View on GitHub
type/feature
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**FEATURE REQUEST**

1. Please describe the feature you are requesting.

When using the singleentrylog mode, depending on the throughput, for example, under the throughput of 200m / s, The default value of the configuration parameter logsizelimit is 1.2g. At this time, due to the large throughput, the generation of checkpoint tasks is fast, but the execution time of checkpoint tasks is long. The speed of the two is inconsistent. A large number of tasks accumulate in the checkpoint thread, resulting in the checkpoint bits recorded in the lastmark file greatly lagging behind the real-time data. Restarting bookie will be abnormally slow. Unnecessary checkpoint tasks can be eliminated according to the execution time of checkpoint tasks. The next effective checkpoint task is generated only after one checkpoint task is executed. The specific way is to clear the accumulated tasks in the checkpoint thread pool every time a newly generated checkpoint task is added.

cpt

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
must-have

3. Provide any additional detail on your proposed use case for this feature.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.