apache / apache/pulsar-client-go

aynsc send [hasSpace()] control has bad performance when send large message

Open
#771 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

1. when i send large message body,the client has a queue to hold it and send by batch, there are two triggers: 1. message
count lager then `maxMessages`, OR 2. messge body sum is larger then 'BatchingMaxSize'
2. Know the codes are 【bc.numMessages+1 <= bc.maxMessages && bc.buffer.ReadableBytes()+msgSize <= uint32(bc.maxBatchSize)】,and it will brings somemany Flush opertion when i send large message body

#### Actual behavior

1. 【unable to add message to batch】will called when send message body size nearly 'BatchingMaxSize'

#### Steps to reproduce

1. send body size lager then 3M
2. make send fast than 1000 qps

#### System configuration
**Pulsar version**: 2.9.0

Contributor guide

Open the contributing guide

Research direction

Start at the async send path's hasSpace() control and inspect the batching condition involving maxMessages, BatchingMaxSize, buffer.ReadableBytes(), and Flush operations. Reproduce with messages larger than 3 MB at over 1000 QPS, then verify that batching avoids the reported “unable to add message to batch” behavior and unnecessary flushes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.