apache / apache/pulsar-client-cpp

[pulsar-client-cpp] Excessive locking cause significant performance degradation

Aberta
#116 3 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
C++
Estrelas
71
Forks
90
Merge médio
2h 33min
PRs com merge (30d)
3

Descrição

**Describe the bug**
Implementation of statistics in cpp client have two concurrency issues.

1. ProducerStatsImpl (and ConsumerStatsImpl) classes use a single shared lock to protect access to internal data. The lock is taken on each sent or received message. Under high load this shared lock causes signficant contention and performance degradation.
Profiler shows that sending and receiving threads block each-other.

![original-profiling](https://user-images.githubusercontent.com/2276675/142137028-b1dab92d-d6a4-47c3-84fd-666bccfd188a.png)

Since sending and receving functions access different member subset they should be protected by different mutex or other approach should be selected.
As example after patching issue I've got about 1/3 throughtput improvement. As you can see on screenshot below threads are witing on I/O but not on mutexes.
![pathed-profiling](https://user-images.githubusercontent.com/2276675/142137475-36f31817-29da-43d5-9ddd-ecbbb4948d8b.png)

2. ProducerStatsImpl implementation has races between destructor and DeadlineTimer callback. Consider following scenario:

1. ProducerStatsImpl destructor acquire the mutex
2. DeadlineTimer calls calback flushAndReset and blocked on mutex
3. ProducerStatsImpl calls timer.cancel and cancel any pending operation but it cannot cancel already executed callback at step 2
4. ProducerStatsImpl destructor release mutex
5. DeadlineTimer acquire the mutex
6. ProducerStatsImpl destructor destroy object
7. DeadlineTimer callback access to deallocated memory

Are you willing accept PR for issue number one or both?

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Leia primeiro as implementações de ProducerStatsImpl e ConsumerStatsImpl e o respectivo caminho de callback de DeadlineTimer. Use profiling sob alta carga para verificar a contenção de locks; considera-se concluído quando a contenção dos mutexes de send/receive for reduzida e os callbacks forem impedidos de acessar uma ProducerStatsImpl destruída, dentro do escopo aceito.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
cpp
Domínio
distributed-systems, performance
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Razoavelmente clara
Facilidade para iniciantes
35/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.