apache / apache/pulsar

Pulsar Handshake was not completed within timeout, closing connection

Open
#11,246 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

我有三个 pulsar broker 节点,但是经常遇到部分 broker 节点 6650 端口服务不可用。节点故障情况下, 从 broker 的日志可以看出 healthcheck 一直处于失败的状态。healthcheck 失败的原因,我判断就是 handshake timeout,下面会贴具体日志。

另外我发现即使 broker 节点不可用,但是其 8080 端口依旧能正常提供服务,可以执行 ./bin/pulsar-admin persistent get-partitioned-topic-metadata persistent://pulsar/pulsar-cluster/10.2.1.26:8080/healthcheck 命令,也能获取到正确的结果,所以只有 6650 端口服务异常。

我通过在故障 broker 节点,通过 tcpdump 抓取 6650 端口报文,发现 pulsar client 和 pulsar broker 可以正常建立 tcp 连接,但是 pulsar client 发送了一笔 tcp data 后,pulsar broker 只会返回 tcp ack ,接下来 pulsar broker 没有任何 tcp 报文。我在 pulsar client 里面加日志,也如期的发现 pulsar client 连接一直处于 SentConnectFrame 状态。等上 30s 后,连接都超时异常。

**我使用的 pulsar 版本为 2.7.0**

pulsar broker 日志:
![image](https://user-images.githubusercontent.com/9940360/124737222-51ed1d00-df4a-11eb-8347-272831ec1cc7.png)

pulsar client 日志:
![image](https://user-images.githubusercontent.com/9940360/124738110-2fa7cf00-df4b-11eb-98b5-4bdf89c650b3.png)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the broker and client logs alongside the tcpdump capture for port 6650 on Pulsar 2.7.0, and compare that path with the working 8080 healthcheck command. Confirm the failure with the reported three-broker topology; done means the handshake completes without the 30-second timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.