apache / apache/pulsar

ZKMetadataStore: BadVersion for /ledgers/underreplication/LAYOUT

Open
#16,455 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
When starting several bookies, I ran into this error:

```
2022-06-27T16:15:06,204+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /ledgers/underreplication/LAYOUT
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handlePutResult(ZKMetadataStore.java:210) ~[com.datastax.oss-pulsar-metadata-2.10.0.6.jar:2.10.0.6]
```

Note that the error is in the `org.apache.pulsar.metadata.impl.ZKMetadataStore` class. After hitting the error, the bookie exited. I'm not sure what would have cause this issue, but it seems like a concurrent modification issue.

**Expected behavior**
The bookie should not fail to start without retrying first.

**Additional context**
This is on the DataStax build of pulsar 2.10. plus some commits. In looking at the code in ZKMetadataStore, there are no relevant differences.

Contributor guide

Open the contributing guide

Research direction

Start with the failure path in org.apache.pulsar.metadata.impl.ZKMetadataStore, especially ZKMetadataStore.java around line 210, and trace how concurrent bookie startup updates /ledgers/underreplication/LAYOUT. Reproduce the concurrent startup failure if possible. Done means a BadVersion error is retried and the bookie does not exit immediately.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.