apache / apache/pulsar

pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes

Open
#10,692 10 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
A clear and concise description of what the bug is.
pulsar 2.7.1
```
k get pod -n pulsar
NAME READY STATUS RESTARTS AGE
pulsar-bookie-0 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-1 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-2 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-3 0/1 CrashLoopBackOff 73 5h58m
pulsar-broker-0 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-1 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-2 0/1 CrashLoopBackOff 71 5h57m
pulsar-proxy-0 1/1 Running 0 5h57m
pulsar-proxy-1 1/1 Running 0 5h57m
pulsar-proxy-2 1/1 Running 0 5h57m
pulsar-pulsar-manager-6bf99b4d88-jk22g 1/1 Running 0 27d
pulsar-recovery-0 1/1 Running 1 27d
pulsar-toolset-0 1/1 Running 1 27d
pulsar-zookeeper-0 1/1 Running 1 27d
pulsar-zookeeper-1 1/1 Running 1 27d
pulsar-zookeeper-2 1/1 Running 1 27d
```
bookie

```
k logs -n pulsar pulsar-bookie-0 --tail 20 -f
22:20:20.891 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22:20:20.895 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
22:20:20.904 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
22:20:20.918 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/172.16.35.208:2181. Will not attempt to authenticate using SASL (unknown error)
22:20:20.924 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.16.67.24:45748, server: pulsar-zookeeper/172.16.35.208:2181
22:20:20.931 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/172.16.35.208:2181, sessionid = 0x2000001f885027b, negotiated timeout = 30000
22:20:20.934 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
22:20:21.104 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current]
22:20:21.104 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
22:20:21.113 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:476) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:712) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
```

broker

```
22:28:45.687 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal Server Errorjava.io.ByteArrayInputStream@7c3c22f9

```

so is there anyone who can help me out?

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: centos 7.9
**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start with the Kubernetes logs for pulsar-bookie-0 and pulsar-broker-0, focusing on Bookie.InvalidCookieException and the broker's HTTP 500 error. Compare the Pulsar 2.7.1 deployment and storage state to determine why the bookie directories lack cookies; done means the bookie and broker pods start without CrashLoopBackOff.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.