apache / apache/pulsar-client-go

Consumer message loss due to duplicate reconnection

Open
#1,461 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

no message loss

#### Actual behavior

During the Consumer reconnection process, if two consecutive reconnectToBroker calls occur, it leads to message loss. Specifically: messages received from Broker during the first reconnection (e.g., M1, M2) are cleared during the second reconnection, and users ultimately only see subsequent messages (e.g., M3, M4).

#### Steps to reproduce

Prerequisites:
1. Consumer uses Durable subscription mode (default mode)
2. Messages have not been acked yet
3. Broker owner changed

Trigger Conditions:
1. Broker sends CommandCloseConsumer to close the Consumer
2. Before the first reconnection completes, TCP connection breaks triggering a second reconnection

_**Maybe Race Condition in Duplicate Reconnection**_:
`handleCloseConsumer` calls `ConnectionClosed` before deleting the handler. If the TCP connection breaks before deletion, the `Close` method will call `ConnectionClosed` again

```
// First call: Broker actively closes
func (c *connection) handleCloseConsumer(closeConsumer *pb.CommandCloseConsumer) {
consumerID := closeConsumer.GetConsumerId()
c.log.Infof("Broker notification of Closed consumer: %d", consumerID)

if consumer, ok := c.consumerHandler(consumerID); ok {
consumer.ConnectionClosed(closeConsumer) // First call
c.DeleteConsumeHandler(consumerID) // Delete handler afterwards
}
}

// Second call: TCP connection breaks
func (c *connection) Close() {
c.closeOnce.Do(func() {
listeners, consumerHandlers, cnx := c.closeAndEmptyObservers()

// If DeleteConsumeHandler hasn't executed, snapshot still contains the consumer
for _, handler := range consumerHandlers {
handler.ConnectionClosed(nil) // Second call
}
})
}
```

_**Then, Key Problem Points Maybe:**_
The second reconection clears previous local queue: `pc.clearReceiverQueue` clears dispatcher.messages, causing M1, M2 to be cleared during the second reconnection ?
```
// pulsar/consumer_partition.go
func (pc *partitionConsumer) grabConn() error {
// ...
if seekMsgID := pc.seekMessageID.get(); seekMsgID != nil {
pc.startMessageID.set(seekMsgID)
pc.seekMessageID.set(nil)
} else {
pc.startMessageID.set(pc.clearReceiverQueue())
}

// In Durable mode, the StartMessageId is not sent to the broker
if pc.options.subscriptionMode != Durable {
cmdSubscribe.StartMessageId = convertToMessageIDData(pc.startMessageID.get())
}
// ...
}
```

Duplicate reconnected log
```
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=696 name=qmjlt subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=792 name=xicnk subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=861 name=keabn subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=547 name=wxznx subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=996 name=ebecn subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=696 name=qmjlt subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=792 name=xicnk subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=861 name=keabn subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=547 name=wxznx subscription=sub-2 topic="persistent://public/default/test-partition-68"
time="2026-01-22T20:49:33+08:00" level=info msg="Reconnected consumer to broker" consumerID=996 name=ebecn subscription=sub-2 topic="persistent://public/default/test-partition-68"
```

#### System configuration
Pulsar version: 4.1
pulsar-client-go: 0.18.0

Contributor guide

Open the contributing guide

Research direction

Start by tracing the duplicate callbacks in the connection handleCloseConsumer and Close paths, then inspect grabConn in pulsar/consumer_partition.go, especially clearReceiverQueue and seekMessageID handling. Reproduce two consecutive reconnections with an unacked Durable subscription and verify that previously received messages are not lost and reconnection is not performed twice.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.