apache / apache/pulsar-client-go
message add to batch failed: AddToBatchFailed
- Dominant language
- Go
- Stars
- 745
- Forks
- 389
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 3
Description
#### Expected behavior
Settings: Schema=StringSchema, CompressionType=pulsar.LZ4[, DisableBatching=true]
pulsar-client-go version: v0.8.0 & v0.8.1
when: string size > 132000 and size < 5MB, producer can send message normally
#### Actual behavior
when string size > 132000, program will throw "unable to add message to batch" (message add to batch failed: AddToBatchFailed)
#### Steps to reproduce
```
func main(){
ProducerString(genStr(132000)) // when the string size > 132000,program will throw "unable to add message to batch" using pulsar-client-go version v0.8.0 & v0.8.1
}
// generate test string
func genStr(size int) string {
var builder strings.Builder
for i:=0; i < size; i++ {
builder.WriteString("a")
//builder.WriteString("abcdefghi,")
}
return builder.String()
}
// producer to send msg
func ProducerString(str string) {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: pulsar://10.200.19.3:6650,
OperationTimeout: 30 * time.Second,
ConnectionTimeout: 30 * time.Second,
})
if err != nil {
utils.Logger.Fatal(err)
return
}
defer client.Close()
producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: "persistent://test-zdh/test/pt-longstr-v1",
Schema: pulsar.NewStringSchema(nil),
CompressionType: pulsar.LZ4,
DisableBatching: true,
})
defer producer.Close()
msgID, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
Value: str,
})
if err != nil {
utils.Logger.Fatal(err)
}
utils.Logger.Info(msgID)
}
```
#### System configuration
**Pulsar version**: v2.6.3
**pulsar-client-go bug version**: v0.8.0 & v0.8.1
**go version**: 1.15.12
Contributor guide
Research direction
Start with the ProducerString and genStr reproduction using pulsar-client-go v0.8.0 or v0.8.1, with LZ4 compression and DisableBatching enabled. Trace the producer send path for strings larger than 132000 bytes and compare it with smaller payloads. Done means the reproduction sends the larger message without an AddToBatchFailed error, or the issue documents the confirmed limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100