dpkp / dpkp/kafka-python

Twisted interface

Open
#560 16 comments 6 reactions 1 assignee Claimed by @dpkp View on GitHub
Dominant language
Python
Stars
5.9k
Forks
1.5k
Avg merge
3d 11h
Merged PRs (30d)
3

Description

Kafka-Python claims to be "async" but this seems to just mean "uses lots of threads to put your I/O into the background" and not actually "event-driven". It would be nice to have an interface which avoids the overhead (thread stacks, GIL switching, etc) and risks (sharing too much state, deadlocks, difficult to debug differing behavior at scale) of threads, but presents a similar interface.

I filed the ticket with the summary "twisted" because of my particular biases, but this should be done in a way which is as framework-neutral as possible, since supporting the `asyncio` stdlib module at some point in the future would probably be interesting as well, not to mention Tornado and Vanilla and so on.

Given that one would be consuming a queue, perhaps this would be a good landmark project for https://github.com/twisted/tubes which, despite its github URL, has an (almost) framework-neutral architecture and lots of good abstractions around flow control and propagating backpressure.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.