addEntry succeeds on replacement bookies after ensemble recovery, but entry is silently dropped — lh.close() fails with BKMetadataVersionException
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
## Summary
When an open ledger's ensemble bookies are decommissioned and replaced with fresh bookies at the same network addresses, the original client's subsequent `addEntry` call **returns success** (entry ID assigned, LAC advanced), yet the entry is **permanently lost**: the recovery process had already closed the ledger at a lower `lastEntry` in ZK metadata, and `lh.close()` then fails with `BKMetadataVersionException`. The client receives no `BKLedgerFencedException`; the data loss is silent.
## Environment
| | |
|---|---|
| BookKeeper version | 4.17.3 (bundled in Apache Pulsar 4.0.10) |
| Also reproduced on | BK 4.16.x (bundled in Pulsar 3.0.x) |
| ZooKeeper | embedded in `apachepulsar/pulsar:4.0.10` |
| Java | OpenJDK 21 |
## Steps to Reproduce
**Cluster setup**
- 1 ZooKeeper node
- 3 bookies (`bk1`, `bk2`, `bk3`) on the same Docker bridge network
**Client**
Opens a ledger with `E=2, WQ=2, AQ=2`. The ensemble is assigned two of the three bookies (say `bk2` and `bk3`); `bk1` is the spare.
Writes `entry-0`, then blocks waiting for a proceed signal.
**Decommission + replace (one at a time)**
Because E=2/WQ=2 with only 3 bookies leaves one spare, the order must be interleaved:
1. **D1** — `docker stop bk2` → `bookkeeper shell decommissionbookie -bookieid :3181`
The ReplicationWorker copies bk2's fragments to bk1 (the spare) and deletes bk2's ZK cookie.
2. **E1** — Start a fresh `bk2-new` at the **exact same IP** as the original `bk2`.
3. **D2** — `docker stop bk3` → `bookkeeper shell decommissionbookie -bookieid :3181`
During this step the ReplicationWorker on bk1 detects ledger 0 as under-replicated. After `openLedgerRereplicationGracePeriod` (30 s default) it performs recovery-mode open: fences the ledger, reads up to the last confirmed entry (`lastEntry=0`), closes it in ZK metadata. ZK version is bumped.
4. **E2** — Start a fresh `bk3-new` at the **exact same IP** as the original `bk3`.
**Proceed signal**
After both replacements are registered, the client's proceed sentinel is created.
```
addEntry("entry-1") → id=1, LAC=1 ← SUCCESS, no exception
lh.close() → BKMetadataVersionException: Bad ledger metadata version
```
## Observed Behaviour
```
[client] entry-1 id=1 LAC=1
[client] ERROR lh.close(): BKMetadataVersionException: Bad ledger metadata version
```
Log from the recovery side (during D2):
```
INFO LedgerFragmentReplicator - Updated ZK to point ledger fragments from old bookies
to new bookies: {:3181=:3181}
```
And from the original client's close path:
```
ERROR Metadata conflict when closing ledger 0. Another client may have recovered
the ledger while there were writes outstanding.
(local lastEntry:1 length:14) (metadata lastEntry:0 length:7)
```
## Expected Behaviour
The original client should receive `BKLedgerFencedException` (or equivalent) on the `addEntry` call — **not** a success acknowledgement — because the ledger was already fenced and closed by the recovery process before the write was attempted.
Alternatively, if `addEntry` is allowed to succeed, `lh.close()` should not silently discard the entry: it should either succeed (reconcile the metadata) or propagate a clear error that signals the committed entry can no longer be confirmed.
## Root Cause Analysis
BookKeeper's fencing protocol works by sending `fenceLedger` RPCs to the **current ensemble** bookies. Those bookies record the fence in memory and reject future writes with `BKLedgerFencedException`. This relies on the **same** bookie processes being present at the time of both the fence and any subsequent write attempt.
When a bookie is replaced by a **fresh process at the same IP and port**, the new bookie has no in-memory fence state for any previously open ledger. The original client — whose `LedgerHandle` still holds a valid connection to the (now new) bookies — sends `addEntry` RPCs which the new bookies accept without complaint, because from their perspective this ledger has never been fenced.
The result is that `addEntry` completes with a success acknowledgement from `AQ` (=2) replicas, advancing `LAC` to 1. The entry is physically written to the new bookies' journals. But since ZK metadata records `lastEntry=0` (set during recovery before the writer wrote), the entry will never be visible to any future reader and is functionally lost.
The `BKMetadataVersionException` from `lh.close()` is the only signal the writer receives, and it arrives **after** `addEntry` has already confirmed the write — too late for the application to react.
## Impact
- **Silent data loss**: writes that receive a successful `addEntry` acknowledgement can be permanently dropped with no indication to the writer.
- This affects any workflow where bookies are decommissioned via `bookkeeper shell decommissionbookie` while writers hold open `LedgerHandle` objects whose ensembles include the decommissioned bookies — a routine operational procedure.
- Reproduced on two independent Pulsar/BookKeeper releases (3.0.x / 4.0.x), suggesting this is a fundamental design gap rather than a version-specific regression.
Contributor guide
Assessment
This issue has not been assessed yet.