apache / apache/bookkeeper

addEntry succeeds on replacement bookies after ensemble recovery, but entry is silently dropped — lh.close() fails with BKMetadataVersionException

Open
#4,812 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

## Summary

When an open ledger's ensemble bookies are decommissioned and replaced with fresh bookies at the same network addresses, the original client's subsequent `addEntry` call **returns success** (entry ID assigned, LAC advanced), yet the entry is **permanently lost**: the recovery process had already closed the ledger at a lower `lastEntry` in ZK metadata, and `lh.close()` then fails with `BKMetadataVersionException`. The client receives no `BKLedgerFencedException`; the data loss is silent.

## Environment

| | |
|---|---|
| BookKeeper version | 4.17.3 (bundled in Apache Pulsar 4.0.10) |
| Also reproduced on | BK 4.16.x (bundled in Pulsar 3.0.x) |
| ZooKeeper | embedded in `apachepulsar/pulsar:4.0.10` |
| Java | OpenJDK 21 |

## Steps to Reproduce

**Cluster setup**

- 1 ZooKeeper node
- 3 bookies (`bk1`, `bk2`, `bk3`) on the same Docker bridge network

**Client**

Opens a ledger with `E=2, WQ=2, AQ=2`. The ensemble is assigned two of the three bookies (say `bk2` and `bk3`); `bk1` is the spare.

Writes `entry-0`, then blocks waiting for a proceed signal.

**Decommission + replace (one at a time)**

Because E=2/WQ=2 with only 3 bookies leaves one spare, the order must be interleaved:

1. **D1** — `docker stop bk2` → `bookkeeper shell decommissionbookie -bookieid :3181`
The ReplicationWorker copies bk2's fragments to bk1 (the spare) and deletes bk2's ZK cookie.

2. **E1** — Start a fresh `bk2-new` at the **exact same IP** as the original `bk2`.

3. **D2** — `docker stop bk3` → `bookkeeper shell decommissionbookie -bookieid :3181`
During this step the ReplicationWorker on bk1 detects ledger 0 as under-replicated. After `openLedgerRereplicationGracePeriod` (30 s default) it performs recovery-mode open: fences the ledger, reads up to the last confirmed entry (`lastEntry=0`), closes it in ZK metadata. ZK version is bumped.

4. **E2** — Start a fresh `bk3-new` at the **exact same IP** as the original `bk3`.

**Proceed signal**

After both replacements are registered, the client's proceed sentinel is created.

```
addEntry("entry-1") → id=1, LAC=1 ← SUCCESS, no exception
lh.close() → BKMetadataVersionException: Bad ledger metadata version
```

## Observed Behaviour

```
[client] entry-1 id=1 LAC=1
[client] ERROR lh.close(): BKMetadataVersionException: Bad ledger metadata version
```

Log from the recovery side (during D2):

```
INFO LedgerFragmentReplicator - Updated ZK to point ledger fragments from old bookies
to new bookies: {:3181=:3181}
```

And from the original client's close path:

```
ERROR Metadata conflict when closing ledger 0. Another client may have recovered
the ledger while there were writes outstanding.
(local lastEntry:1 length:14) (metadata lastEntry:0 length:7)
```

## Expected Behaviour

The original client should receive `BKLedgerFencedException` (or equivalent) on the `addEntry` call — **not** a success acknowledgement — because the ledger was already fenced and closed by the recovery process before the write was attempted.

Alternatively, if `addEntry` is allowed to succeed, `lh.close()` should not silently discard the entry: it should either succeed (reconcile the metadata) or propagate a clear error that signals the committed entry can no longer be confirmed.

## Root Cause Analysis

BookKeeper's fencing protocol works by sending `fenceLedger` RPCs to the **current ensemble** bookies. Those bookies record the fence in memory and reject future writes with `BKLedgerFencedException`. This relies on the **same** bookie processes being present at the time of both the fence and any subsequent write attempt.

When a bookie is replaced by a **fresh process at the same IP and port**, the new bookie has no in-memory fence state for any previously open ledger. The original client — whose `LedgerHandle` still holds a valid connection to the (now new) bookies — sends `addEntry` RPCs which the new bookies accept without complaint, because from their perspective this ledger has never been fenced.

The result is that `addEntry` completes with a success acknowledgement from `AQ` (=2) replicas, advancing `LAC` to 1. The entry is physically written to the new bookies' journals. But since ZK metadata records `lastEntry=0` (set during recovery before the writer wrote), the entry will never be visible to any future reader and is functionally lost.

The `BKMetadataVersionException` from `lh.close()` is the only signal the writer receives, and it arrives **after** `addEntry` has already confirmed the write — too late for the application to react.

## Impact

- **Silent data loss**: writes that receive a successful `addEntry` acknowledgement can be permanently dropped with no indication to the writer.
- This affects any workflow where bookies are decommissioned via `bookkeeper shell decommissionbookie` while writers hold open `LedgerHandle` objects whose ensembles include the decommissioned bookies — a routine operational procedure.
- Reproduced on two independent Pulsar/BookKeeper releases (3.0.x / 4.0.x), suggesting this is a fundamental design gap rather than a version-specific regression.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.