ZKMetadataStore: BadVersion for /ledgers/underreplication/LAYOUT
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
When starting several bookies, I ran into this error:
```
2022-06-27T16:15:06,204+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /ledgers/underreplication/LAYOUT
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handlePutResult(ZKMetadataStore.java:210) ~[com.datastax.oss-pulsar-metadata-2.10.0.6.jar:2.10.0.6]
```
Note that the error is in the `org.apache.pulsar.metadata.impl.ZKMetadataStore` class. After hitting the error, the bookie exited. I'm not sure what would have cause this issue, but it seems like a concurrent modification issue.
**Expected behavior**
The bookie should not fail to start without retrying first.
**Additional context**
This is on the DataStax build of pulsar 2.10. plus some commits. In looking at the code in ZKMetadataStore, there are no relevant differences.
Contributor guide
Research direction
Start with the failure path in org.apache.pulsar.metadata.impl.ZKMetadataStore, especially ZKMetadataStore.java around line 210, and trace how concurrent bookie startup updates /ledgers/underreplication/LAYOUT. Reproduce the concurrent startup failure if possible. Done means a BadVersion error is retried and the bookie does not exit immediately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100