apache / apache/pulsar-client-cpp
[pulsar-client-cpp] Excessive locking cause significant performance degradation
- 主要語言
- C++
- 星號
- 71
- 分支
- 90
- 平均合併
- 2 小時 33 分鐘
- 30 天內合併 PR
- 3
描述
**Describe the bug**
Implementation of statistics in cpp client have two concurrency issues.
1. ProducerStatsImpl (and ConsumerStatsImpl) classes use a single shared lock to protect access to internal data. The lock is taken on each sent or received message. Under high load this shared lock causes signficant contention and performance degradation.
Profiler shows that sending and receiving threads block each-other.

Since sending and receving functions access different member subset they should be protected by different mutex or other approach should be selected.
As example after patching issue I've got about 1/3 throughtput improvement. As you can see on screenshot below threads are witing on I/O but not on mutexes.

2. ProducerStatsImpl implementation has races between destructor and DeadlineTimer callback. Consider following scenario:
1. ProducerStatsImpl destructor acquire the mutex
2. DeadlineTimer calls calback flushAndReset and blocked on mutex
3. ProducerStatsImpl calls timer.cancel and cancel any pending operation but it cannot cancel already executed callback at step 2
4. ProducerStatsImpl destructor release mutex
5. DeadlineTimer acquire the mutex
6. ProducerStatsImpl destructor destroy object
7. DeadlineTimer callback access to deallocated memory
Are you willing accept PR for issue number one or both?
貢獻指南
研究方向
首先閱讀 ProducerStatsImpl 和 ConsumerStatsImpl 的實作及其 DeadlineTimer callback 路徑。使用高負載 profiling 驗證鎖競爭;對於已接受的範圍,完成標準是降低 send/receive mutex 競爭,並防止 callback 存取已銷毀的 ProducerStatsImpl。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- cpp
- 領域
- distributed-systems, performance
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100