apache / apache/bookkeeper

[Question] Correct procedure for re-IPing a Pulsar cluster with ZooKeeper and BookKeeper

Open
#4,694 0 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

Hello,

I’m looking for information on the correct way to re-IP an Apache Pulsar cluster using ZooKeeper and BookKeeper.

Before performing the re-IP, I wanted to make some configuration changes so that the BookieID uses \:\ instead of the default \:\. To do this, I set `useHostNameAsBookieID=true`. However, after restarting the BookKeeper systemd service, the service fails to start and the following error appears in the logs:
```
2025-12-23T10:22:39,564 - ERROR [main:Main@218] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [5
bookieHost: “.dummy.com:3181"
journalDir: “/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
] is not matching with [5
bookieHost: “10.0.0.1:3181"
journalDir: "/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
]
```

I tried the following steps to resolve the issue:
- Stopping the Bookkeeper systemd service
- Moving the Cookie/VERSION files from the journal01 and ledger01 directories to /tmp
- Deleting the old cookie using:
- `bin/bookkeeper shell cookie_delete 10.0.0.1:3181`
- Running updateBookieInLedger with the old srcBookie and new destBookie
- `bin/bookkeeper shell updateBookieInLedger --srcBookie ’10.0.0.1:3181' --destBookie ‘.dummy.com:3181' --updatespersec 5`
- Generating a new cookie and coping the created file to the VERSION files for the journal01 and ledger01 directories.
- `bin/bookkeeper shell cookie_generate -j /data/bookkeeper/journal01 -l /data/bookkeeper/ledger01 -o /tmp/test-cookie .dummy.com:3181`

None of these steps unfortunately seemed to work. The only way I was able to recover was by clearing the entire journal and ledger directories and letting the node re-sync from the other bookies.

So my main questions are:
- What is the correct/recommended approach to re-IP Bookkeeper nodes while preserving existing ledger data?
- Are there any additional steps required when changing `useHostNameAsBookieID` on an existing cluster like clearing or updating the Cookie?

Some additional context about the environment:
Pulsar version: 3.0.5
Bookkeeper version: 4.16.5
Zookeeper version: 3.8.4
openJDK version: 17.0.17
Running on RHEL8 VM's

Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.