apache / apache/pulsar-client-cpp

[pulsar-client-cpp] Excessive locking cause significant performance degradation

Aperta
#116 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C++
Stelle
71
Fork
90
Merge medio
2h 33m
PR unite (30g)
3

Descrizione

**Describe the bug**
Implementation of statistics in cpp client have two concurrency issues.

1. ProducerStatsImpl (and ConsumerStatsImpl) classes use a single shared lock to protect access to internal data. The lock is taken on each sent or received message. Under high load this shared lock causes signficant contention and performance degradation.
Profiler shows that sending and receiving threads block each-other.

![original-profiling](https://user-images.githubusercontent.com/2276675/142137028-b1dab92d-d6a4-47c3-84fd-666bccfd188a.png)

Since sending and receving functions access different member subset they should be protected by different mutex or other approach should be selected.
As example after patching issue I've got about 1/3 throughtput improvement. As you can see on screenshot below threads are witing on I/O but not on mutexes.
![pathed-profiling](https://user-images.githubusercontent.com/2276675/142137475-36f31817-29da-43d5-9ddd-ecbbb4948d8b.png)

2. ProducerStatsImpl implementation has races between destructor and DeadlineTimer callback. Consider following scenario:

1. ProducerStatsImpl destructor acquire the mutex
2. DeadlineTimer calls calback flushAndReset and blocked on mutex
3. ProducerStatsImpl calls timer.cancel and cancel any pending operation but it cannot cancel already executed callback at step 2
4. ProducerStatsImpl destructor release mutex
5. DeadlineTimer acquire the mutex
6. ProducerStatsImpl destructor destroy object
7. DeadlineTimer callback access to deallocated memory

Are you willing accept PR for issue number one or both?

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Read the ProducerStatsImpl and ConsumerStatsImpl implementations and their DeadlineTimer callback path first. Use high-load profiling to verify lock contention; done means reducing send/receive mutex contention and preventing callbacks from accessing destroyed ProducerStatsImpl, for the accepted scope.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cpp
Ambito
distributed-systems, performance
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.