Compaction Limits

**FEATURE REQUEST**

Currently the compaction feature allows for two compaction specifications (minor and major) that define how the system will perform compaction. Each minor and major can be given a interval and threshold which tells the duration between runs, and the usage % of the entrylog to be considered for compaction. There also exists a rate limit on either (bytes or entries to be written that can be used to balance the system load with the compaction load.)

What we observe as a issue, is that there is no limit to the amount of time that is spent in compaction or limit on the number of logs to be compacted. This may force a delay in garbage collection, which may in turn affect other aspects of the system. In a heavily loaded system the number of entry logs can grow, which will extend the compaction time.

While compaction is good, the amount of time spent on compaction compared to processing messages should be a limited cost. If the system is under heavy load, then the compaction is also delayed.

- What are the impacts of continued delay of garbage collection? Can/should the garbage collection be interleaved between compaction of the logs if the compaction spans across gc interval?

Start by locating the compaction and garbage-collection entry points in the BookKeeper codebase and trace how compaction duration, log count, and garbage-collection intervals currently interact. The issue needs an agreed limit or scheduling behavior before implementation can begin; done means the chosen behavior is implemented and validated under heavy load.

Written by the indexing model from the issue text.