apache / apache/pulsar-client-go

Deduplication fails when the batch message contains duplicate message and valid message

Open
#1,395 1 comment 9 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

When a producer resends a **duplicate SequenceID** while batching is enabled, the client should **flush the current batch** and start a new one.
This guarantees that a batch never contains both *old/duplicate* and *new* `SequenceId`s, allowing the broker‑side deduplication logic to drop only the duplicates and accept the genuinely new messages.

This is the behavior of the Java client since PR [apache/pulsar#6326](https://github.com/apache/pulsar/pull/6326).

#### Actual behavior

With the Go client (v0.15.0) the duplicate and new messages are added to the **same** batch.
Because the outer `CommandSend` carries only the *lowest* `SequenceId` in the batch, the broker cannot determine that newer messages are present; it treats the **entire** batch as a duplicate and drops it — including the fresh messages.

#### Steps to reproduce

```go
package main

import (
"context"
"fmt"
"log"
"sync"
"time"

"github.com/apache/pulsar-client-go/pulsar"
)

func main() {
// ---------- set up ----------
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://localhost:6650",
OperationTimeout: 5 * time.Second,
ConnectionTimeout: 5 * time.Second,
})
if err != nil {
log.Fatalf("client: %v", err)
}
defer client.Close()

const topic = "persistent://public/default/dedupe-go-batch-bug"
const producerName = "go-dedupe-test"

prod, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: topic,
Name: producerName,
DisableBatching: false,
BatchingMaxPublishDelay: 1 * time.Second,
})
if err != nil {
log.Fatalf("producer: %v", err)
}
defer prod.Close()

// helper to send one async msg and wait for ack
send := func(seq int64, wg *sync.WaitGroup) {
wg.Add(1)
prod.SendAsync(context.Background(),
&pulsar.ProducerMessage{
Payload: []byte(fmt.Sprintf("seq-%d", seq)),
SequenceID: &seq,
},
func(id pulsar.MessageID, _ *pulsar.ProducerMessage, err error) {
log.Printf("Send cb seq=%d err=%v", seq, err)
wg.Done()
})
}

// ---------- 1) first batch: seq 0,1 ----------
var wg sync.WaitGroup
send(0, &wg)
send(1, &wg)
wg.Wait() // ensure broker acks first batch
time.Sleep(1 * time.Second) // let batch window expire

// ---------- 2) second batch with duplicate + new ----------
// we issue them back‑to‑back so they land in the SAME batch
send(0, &wg) // duplicate
send(2, &wg) // new
wg.Wait()
prod.Flush()

// ---------- consume to see what really persisted ----------
cons, _ := client.Subscribe(pulsar.ConsumerOptions{
Topic: topic,
SubscriptionName: "check",
Type: pulsar.Exclusive,
SubscriptionInitialPosition: pulsar.SubscriptionPositionEarliest,
})
defer cons.Close()

got := map[string]struct{}{}
for {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
msg, err := cons.Receive(ctx)
cancel()
if err != nil {
break
}
got[string(msg.Payload())] = struct{}{}
cons.Ack(msg)
}

fmt.Println("messages stored:")
for k := range got {
fmt.Println(" ", k)
}
}

```

#### System configuration
**Pulsar version**: 4.0.5

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Go reproduction against Pulsar 4.0.5, then trace the Go client's batching and sequence-ID handling for SendAsync. Done means a duplicate SequenceID flushes the current batch before the new message is added, so consuming the topic retains the genuinely new message while dropping only the duplicate.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.