Prevent endless AutoRecovery retries for single-replica ledgers
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
## 1. Feature requested
BookKeeper AutoRecovery should recognize failed-Bookie underreplication tasks for
ledgers whose `writeQuorumSize == 1` and avoid entering an endless recovery loop
when the failed Bookie is permanently unavailable.
For these ledgers, each entry has only one data source. Once that source is
permanently lost, AutoRecovery cannot reconstruct the missing data from another
Bookie. The requested behavior is therefore to:
- avoid publishing new failed-Bookie underreplication tasks for known
single-replica ledgers;
- remove matching historical failed-Bookie tasks created before this behavior
was introduced;
- always release the underreplication lock during the cleanup path;
- preserve placement-policy repair tasks, including `writeQuorumSize == 1`
tasks with an empty replica list; and
- expose dedicated Auditor and ReplicationWorker metrics so operators can
identify possible data loss and take manual recovery action.
This feature must not claim to recover data that has no surviving replica. It
should only stop the impossible automatic recovery path and make the condition
visible to operators.
## 2. Importance and workarounds
Importance: **must-have** for deployments that create ledgers with
`writeQuorumSize == 1`; **should-have** otherwise.
Without this behavior, a permanently lost Bookie leaves an underreplication task
that the Worker repeatedly locks, attempts to open or read, fails to recover,
releases the lock, and retries. This consumes Worker, BookKeeper client, and
ZooKeeper resources, produces noisy failure logs, and can delay processing of
ledgers that are actually recoverable.
There is no safe automatic workaround for the lost data itself. Operators can
restore the original Bookie or recover data from backups, but those options may
not be available. Disabling AutoRecovery or manually deleting the
underreplication task can stop the retry noise, but disabling AutoRecovery also
stops unrelated recovery and deleting the task can hide a data-loss condition.
## 3. Proposed use case
A cluster contains a ledger created with `writeQuorumSize == 1`. The ledger's
only Bookie fails permanently. The Auditor detects the failed Bookie, reads the
ledger metadata, and identifies that automatic reconstruction is impossible.
Instead of creating a task that will fail forever, the Auditor skips publication
and increments a dedicated metric.
During an upgrade, the cluster may already contain persistent underreplication
znodes created by an older BookKeeper version. When a Worker consumes one of
these historical tasks, it should inspect the task shape and ledger metadata,
clean up only failed-Bookie tasks for single-replica ledgers, release the lock,
and report a skipped outcome. A placement-policy task with an empty replica list
must continue through the normal replication path.
The metrics should allow operators to alert on skipped ledgers and decide
whether to restore the original Bookie, use backups, or apply an
application-specific recovery procedure.
Contributor guide
Research direction
Start by tracing the Auditor flow that publishes failed-Bookie underreplication tasks and the ReplicationWorker path that consumes and cleans them up. Check how ledger metadata, task shape, lock release, placement-policy tasks, and existing metrics are handled. Done means impossible single-replica recovery tasks are skipped or removed safely, locks are released, valid placement repairs continue, and dedicated outcomes are observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100