[Bug] bookie-2 is not able to recover after lossing the filesystem
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
Kubernetes 1.29
Pulsar 3.2.3
### Minimal reproduce step
scale down to 1 bookies
delete the related pvc/pv for bookie-2
### What did you expect to see?
a full recover by 1 bookie
### What did you see instead?
`2024-07-17T10:10:23,047+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current]
2024-07-17T10:10:23,048+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:113) ~[org.apache.bookkeeper-bookkeeper-server-4.16.5.jar:4.16.5]
at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:408) ~[org.apache.bookkeeper-bookkeeper-server-4.16.5.jar:4.16.5]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.5.jar:4.16.5]
at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.5.jar:4.16.5]
at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.5.jar:4.16.5]`
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Research direction
Reproduce the failure by scaling to one bookie and deleting bookie-2's PVC/PV. Start with LegacyCookieValidation.java and the EmbeddedServer builder shown in the stack trace; determine why the recovered directories fail cookie validation. Done means the remaining bookie completes recovery without the InvalidCookieException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100