Failed to build bookie server in kubernetes environment
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
we have deployed pulsar using helm chart in openshfit k8s environment. when we scale up the deleted volume bookie we encounter with below issue.
```
2.284: Total time for which application threads were stopped: 0.0001663 seconds, Stopping threads took: 0.0001040 seconds
10:28:55.441 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
10:28:55.441 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
2.404: Total time for which application threads were stopped: 0.0002793 seconds, Stopping threads took: 0.0001671 seconds
2.405: Total time for which application threads were stopped: 0.0002462 seconds, Stopping threads took: 0.0001874 seconds
2.406: Total time for which application threads were stopped: 0.0002902 seconds, Stopping threads took: 0.0001068 seconds
2.413: Total time for which application threads were stopped: 0.0005099 seconds, Stopping threads took: 0.0003770 seconds
2.413: Total time for which application threads were stopped: 0.0001357 seconds, Stopping threads took: 0.0000875 seconds
2.415: Total time for which application threads were stopped: 0.0004179 seconds, Stopping threads took: 0.0001708 seconds
2.417: Total time for which application threads were stopped: 0.0001661 seconds, Stopping threads took: 0.0000410 seconds
2.418: Total time for which application threads were stopped: 0.0003534 seconds, Stopping threads took: 0.0002680 seconds
2.419: Total time for which application threads were stopped: 0.0003882 seconds, Stopping threads took: 0.0003552 seconds
2.419: Total time for which application threads were stopped: 0.0000419 seconds, Stopping threads took: 0.0000138 seconds
2.420: Total time for which application threads were stopped: 0.0000399 seconds, Stopping threads took: 0.0000124 seconds
10:28:55.461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:468) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
2.425: Total time for which application threads were stopped: 0.0000542 seconds, Stopping threads took: 0.0000156 seconds
```
**To Reproduce**
Steps to reproduce the behavior :
1. scale down the bookie pod.
2. delete persistent volume claim and persistent volume of deleted bookie pod.
3. scale up the bookie pod
**Expected behavior**
scale up the bookie pod should not cause the problem.
Contributor guide
Research direction
Start with the Bookie.checkEnvironmentWithStorageExpansion path in Bookie.java and the buildBookieServer path in Main.java, then reproduce the listed scale-down, volume deletion, and scale-up steps in the Helm-based Kubernetes deployment. Determine why the recreated bookie lacks a valid cookie and verify that scaling the bookie back up no longer fails with InvalidCookieException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, java, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100