apache / apache/pulsar-client-go

message add to batch failed: AddToBatchFailed

Open
#847 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

Settings: Schema=StringSchema, CompressionType=pulsar.LZ4[, DisableBatching=true]
pulsar-client-go version: v0.8.0 & v0.8.1
when: string size > 132000 and size < 5MB, producer can send message normally

#### Actual behavior

when string size > 132000, program will throw "unable to add message to batch" (message add to batch failed: AddToBatchFailed)

#### Steps to reproduce

```
func main(){
ProducerString(genStr(132000)) // when the string size > 132000,program will throw "unable to add message to batch" using pulsar-client-go version v0.8.0 & v0.8.1
}

// generate test string
func genStr(size int) string {
var builder strings.Builder
for i:=0; i < size; i++ {
builder.WriteString("a")
//builder.WriteString("abcdefghi,")
}
return builder.String()
}

// producer to send msg
func ProducerString(str string) {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: pulsar://10.200.19.3:6650,
OperationTimeout: 30 * time.Second,
ConnectionTimeout: 30 * time.Second,
})
if err != nil {
utils.Logger.Fatal(err)
return
}
defer client.Close()

producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: "persistent://test-zdh/test/pt-longstr-v1",
Schema: pulsar.NewStringSchema(nil),
CompressionType: pulsar.LZ4,
DisableBatching: true,
})
defer producer.Close()

msgID, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
Value: str,
})
if err != nil {
utils.Logger.Fatal(err)
}
utils.Logger.Info(msgID)
}
```

#### System configuration
**Pulsar version**: v2.6.3
**pulsar-client-go bug version**: v0.8.0 & v0.8.1
**go version**: 1.15.12

Contributor guide

Open the contributing guide

Research direction

Start with the ProducerString and genStr reproduction using pulsar-client-go v0.8.0 or v0.8.1, with LZ4 compression and DisableBatching enabled. Trace the producer send path for strings larger than 132000 bytes and compare it with smaller payloads. Done means the reproduction sends the larger message without an AddToBatchFailed error, or the issue documents the confirmed limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.