aio-libs / aio-libs/aiokafka

[QUESTION] Heartbeat thread for CPU intensive consumer: difference to kafka-python

オープン
#700 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
question
主要言語
Python
スター
1.4k
フォーク
269
平均マージ
1日 1時間
マージ済み PR(30日)
6

説明

We use `kafka-python` and have a problem that maybe `aiokafka` could solve. We have a system that consumes messages one-at-a-time and can take a long time to process (up to 10s of minutes). These are math intensive calculations with a lot of work done by C-libraries (called by pandas, numpy, scipy). Once in a while we get a "heartbeat poll expired" error even though we are still inside the `max_poll_interval_ms` and our hypothesis is that the processing thread is not allowing the background heartbeat thread to do it's thing.
Now my questions:
- Would `aiokafka` handle better the heartbeat thread for CPU intensive processes? I'm asking because I looked through the code of both libraries and I got the impression that both libraries implement the heartbeat functionality in a similar way.
- Is the async advantage of `aiokafka` only on the IO side, i.e. communication with the Broker to poll and send messages to topics? Would the performance increase only be apparent if multiple messages are polled together?

BTW Thanks a lot for the awesome work!! 💪 not only here but on `kafka-python` too!

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。