pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
A clear and concise description of what the bug is.
pulsar 2.7.1
```
k get pod -n pulsar
NAME READY STATUS RESTARTS AGE
pulsar-bookie-0 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-1 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-2 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-3 0/1 CrashLoopBackOff 73 5h58m
pulsar-broker-0 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-1 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-2 0/1 CrashLoopBackOff 71 5h57m
pulsar-proxy-0 1/1 Running 0 5h57m
pulsar-proxy-1 1/1 Running 0 5h57m
pulsar-proxy-2 1/1 Running 0 5h57m
pulsar-pulsar-manager-6bf99b4d88-jk22g 1/1 Running 0 27d
pulsar-recovery-0 1/1 Running 1 27d
pulsar-toolset-0 1/1 Running 1 27d
pulsar-zookeeper-0 1/1 Running 1 27d
pulsar-zookeeper-1 1/1 Running 1 27d
pulsar-zookeeper-2 1/1 Running 1 27d
```
bookie
```
k logs -n pulsar pulsar-bookie-0 --tail 20 -f
22:20:20.891 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22:20:20.895 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
22:20:20.904 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
22:20:20.918 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/172.16.35.208:2181. Will not attempt to authenticate using SASL (unknown error)
22:20:20.924 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.16.67.24:45748, server: pulsar-zookeeper/172.16.35.208:2181
22:20:20.931 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/172.16.35.208:2181, sessionid = 0x2000001f885027b, negotiated timeout = 30000
22:20:20.934 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
22:20:21.104 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current]
22:20:21.104 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
22:20:21.113 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:476) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:712) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
```
broker
```
22:28:45.687 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal Server Errorjava.io.ByteArrayInputStream@7c3c22f9
```
so is there anyone who can help me out?
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: centos 7.9
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
Start with the Kubernetes logs for pulsar-bookie-0 and pulsar-broker-0, focusing on Bookie.InvalidCookieException and the broker's HTTP 500 error. Compare the Pulsar 2.7.1 deployment and storage state to determine why the bookie directories lack cookies; done means the bookie and broker pods start without CrashLoopBackOff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100