apache / apache/pulsar-client-cpp

Key-based batching and hashingScheme parameter for Python client

Offen
#83 5 Kommentare 6 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C++
Sterne
71
Forks
90
Ø Merge
2 Std. 33 Min.
Gemergte PRs (30 T.)
3

Beschreibung

**Short intro**
We're running a platform that's effectively just a bunch of microservices written in Python, Java and Go which are communicating through Kafka. Currently we're considering moving from Kafka to Pulsar.
However, one of the main features we need is key-based message routing for solving concurrency problems. Pulsar has that option with KeyShared subscription mode.

**The problem**
In Python, KeyShared subscription works only if producer disables batching. I ran some tests in my local environment and disabling batching results in high decrease of throughput (~25k/s with batching vs ~9k/s without).

**Features that would've solved the problem**
- Key-based batching for Python client
- HashingScheme parameter when creating producer so all of our services (Java, Python and Go) could have the same hashing scheme

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by tracing how the Python client currently handles producer batching and how producer creation exposes configuration. Compare the hashing schemes used by the Python, Java, and Go clients. Done means Python supports key-based batching and producers can select a hashing scheme that produces compatible routing across these clients.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
go, java, kafka, python
Bereich
distributed-systems
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.