apache / apache/pulsar-client-cpp

Key-based batching and hashingScheme parameter for Python client

未关闭
#83 5 条评论 6 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C++
星标
71
派生
90
平均合并
2 小时 33 分钟
30 天内合并 PR
3

描述

**Short intro**
We're running a platform that's effectively just a bunch of microservices written in Python, Java and Go which are communicating through Kafka. Currently we're considering moving from Kafka to Pulsar.
However, one of the main features we need is key-based message routing for solving concurrency problems. Pulsar has that option with KeyShared subscription mode.

**The problem**
In Python, KeyShared subscription works only if producer disables batching. I ran some tests in my local environment and disabling batching results in high decrease of throughput (~25k/s with batching vs ~9k/s without).

**Features that would've solved the problem**
- Key-based batching for Python client
- HashingScheme parameter when creating producer so all of our services (Java, Python and Go) could have the same hashing scheme

贡献指南

打开贡献指南

调研方向

首先追踪 Python 客户端目前如何处理 producer batching,以及 producer 创建如何暴露配置。比较 Python、Java 和 Go 客户端使用的哈希方案。完成的标准是:Python 支持基于 key 的 batching,并且 producer 可以选择一种能够在这些客户端之间生成兼容路由的哈希方案。

由索引模型根据 Issue 内容生成。

评估

技术栈
go, java, kafka, python
领域
distributed-systems
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。