apache / apache/pulsar-client-go

Producer Send and SendAsyn is blocked for forever when pulsar is down

Open
#515 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

Producer Send and SendAsyn is blocked forever when pulsar is down if the MaxReconnectToBroker is set to unlimited retry. In case of pulsar down scenarios, within `runEventsLoop`in `producer_partition.go`, the call enters `reconnectToBroker` and remains in a forever loop until the pulsar broker connection is established. Due to this, no more events are consumed from `eventsChan` channel causing both Send and SendAsyn to be blocked. Due to this, the `SendTimeout` would also be not honoured.

#### Expected behavior
Producer Send must not be blocked forever when the pulsar broker is down. It must honour the SendTimeout and return back with an error.
Producer SendAsyn must never be blocked when the pulsar broker is down. It must honour the SendTimeout and call the callback function.

#### Actual behavior
Due the above mentioned issue Producer Send/SendAsyn blocks forever when the pulsar broker is down

#### Steps to reproduce
1. Create a pulsar producer and set the MaxReconnectToBroker as unlimited retry and SendTimeout as a fixed value
2. Send messages to pulsar using Send or SendAsyn API
3. Bring down pulsar broker or inject a connection error between broker and client
4. In case of Send, the call is blocker forever. In case of SendAsyn, the callback is never called and once the `pendingQueue` is filled, the call is blocked forever.

#### System configuration
Pulsar client version - v0.4.0

Contributor guide

Open the contributing guide

Research direction

Start in producer_partition.go, focusing on runEventsLoop and reconnectToBroker, and trace how eventsChan and pendingQueue are handled when MaxReconnectToBroker allows unlimited retries. Reproduce the failure with a down Pulsar broker, unlimited retries, and a fixed SendTimeout. Done means Send returns an error within the timeout, while SendAsyn invokes its callback instead of blocking indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.