[QUESTION] Heartbeat thread for CPU intensive consumer: difference to kafka-python
- 主要言語
- Python
- スター
- 1.4k
- フォーク
- 269
- 平均マージ
- 1日 1時間
- マージ済み PR(30日)
- 6
説明
We use `kafka-python` and have a problem that maybe `aiokafka` could solve. We have a system that consumes messages one-at-a-time and can take a long time to process (up to 10s of minutes). These are math intensive calculations with a lot of work done by C-libraries (called by pandas, numpy, scipy). Once in a while we get a "heartbeat poll expired" error even though we are still inside the `max_poll_interval_ms` and our hypothesis is that the processing thread is not allowing the background heartbeat thread to do it's thing.
Now my questions:
- Would `aiokafka` handle better the heartbeat thread for CPU intensive processes? I'm asking because I looked through the code of both libraries and I got the impression that both libraries implement the heartbeat functionality in a similar way.
- Is the async advantage of `aiokafka` only on the IO side, i.e. communication with the Broker to poll and send messages to topics? Would the performance increase only be apparent if multiple messages are polled together?
BTW Thanks a lot for the awesome work!! 💪 not only here but on `kafka-python` too!
コントリビューションガイド
評価
この issue はまだ評価されていません。