apache / apache/pulsar-client-cpp

Key-based batching and hashingScheme parameter for Python client

未關閉
#83 5 則留言 6 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
C++
星號
71
分支
90
平均合併
2 小時 33 分鐘
30 天內合併 PR
3

描述

**Short intro**
We're running a platform that's effectively just a bunch of microservices written in Python, Java and Go which are communicating through Kafka. Currently we're considering moving from Kafka to Pulsar.
However, one of the main features we need is key-based message routing for solving concurrency problems. Pulsar has that option with KeyShared subscription mode.

**The problem**
In Python, KeyShared subscription works only if producer disables batching. I ran some tests in my local environment and disabling batching results in high decrease of throughput (~25k/s with batching vs ~9k/s without).

**Features that would've solved the problem**
- Key-based batching for Python client
- HashingScheme parameter when creating producer so all of our services (Java, Python and Go) could have the same hashing scheme

貢獻指南

開啟貢獻指南

研究方向

首先追蹤 Python 客戶端目前如何處理 producer batching,以及 producer 建立如何暴露設定。比較 Python、Java 和 Go 客戶端使用的雜湊方案。完成的標準是:Python 支援基於 key 的 batching,且 producer 可以選擇一種能在這些客戶端之間產生相容路由的雜湊方案。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
go, java, kafka, python
領域
distributed-systems
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
基本清楚
新手友好度
30/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。