apache / apache/pulsar-client-go

Failed to lookup topic (request timeout) after pulsar broker restart

Open
#349 7 comments 0 reactions 0 assignees View on GitHub
triage/week-33
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

Consumers/Producers are created normally after broker restart.

#### Actual behavior

```
level=info msg="Connecting to broker" remote_addr="pulsar://10.0.0.4:6650"
level=info msg="TCP connection established" local_addr="10.0.0.2:55310" remote_addr="pulsar://10.0.0.4:6650"
level=info msg="Connection is ready" local_addr="10.0.0.2:55310" remote_addr="pulsar://10.0.0.4:6650"
level=warning msg="Failed to lookup topic" error="request timed out" name=zwvsg subscription=my-sub topic="persistent://public/default/telegram"
level=error msg="Failed to create consumer" error="request timed out"
```

On broker side:
```
INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.0.2:55354
INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.0.2:55354
INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.0.2:55358
INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.0.2:55358
INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.0.2:55362
```

#### Steps to reproduce

1. Start pulsar (3 node deployment)
2. Connect to first node with producer/consumer
3. Restart first node
4. Observe `request timed out` errors

#### System configuration
**Pulsar version**: 2.6.0

Hi, while testing 3 node deployment durability I've encountered weird behaviour: go client was able to reconnect, but any request was timing out. This state is quite flaky and I was able to reproduce it once in standalone deployment.

One or multiple broker restarts fixes this problem, but it looks scary.

Can it be pulsar-client-go specific or I should create issue in apache/pulsar? I'm not familiar with java, so can't provide any useful debug info now, sorry.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.