apache / apache/bookkeeper

BP-23: Ledger Balancer

Open
#846 2 comments 1 reaction 0 assignees View on GitHub
area/autorecovery area/bookie BP triage/week-33 type/feature
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BP**

This is the master ticket for tracking BP-23:

In **Messaging/Streaming/Logging** oriented use case (where old ledgers/segments are most likely will be deleted at some point), we don't actually need to rebalance the ledgers stored on bookies.

However,
In **Storage** oriented use cases (where data most likely will never be deleted), BookKeeper data might not always be placed uniformly across bookies. One common reason is addition of new bookies to an existing cluster. This proposal is proposing to provide a balancer mechanism (as an utility, also as part of `AutoRecovery` daemon), that analyzes ledger distributions and balances ledgers across bookies.

It replicated ledgers to new bookies (based on resource-aware placement policies) until the cluster is deemed to be balanced, which means that disk utilization of every bookie (ratio of used space on the node to the capacity of the node) differs from the utilization of the cluster (ratio of used space on the cluster to total capacity of the cluster) by no more than a given threshold percentage.

The balancer will replicate ledgers away from disk-filled bookies as first priority.

Proposal PR - #847

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.