aio-libs / aio-libs/aiokafka

[QUESTION] extremly high Fetch Rate

Đang mở
#1,081 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
question
Ngôn ngữ chính
Python
Star
1.4k
Fork
269
Merge trung bình
1 ngày 1 giờ
Pull request đã merge (30 ngày)
6

Mô tả

I have http API service on fastapi that only write to kafka by aiokafka. So I only one producer cause very high fetch rate on monitoring graphs - usually 3.5-4k - that amount is sum of counts rates for different topics. Producer only send to one topic but fetch rate is also high for topics that are have no new messages for along time.
For monitoring we use metric `kafka_server_brokertopicmetrics_totalfetchrequests_total` by topic and see that problem. Searching about metrics got https://newrelic.com/blog/how-to-relic/tuning-apache-kafka-consumers where found following text:
```
A reasonable fetch rate is somewhere around 1 to 5 requests per second per broker because it corresponds to 200 to 1,000 milliseconds of fetch latency. If your average fetch size is consistently hitting a limit and your average fetch request rate is higher than five requests per second per broker, you’re likely creating many small fetch requests instead of fewer large ones. This not only limits the throughput of your application, it can very quickly consume a large amount of your Kafka Broker capacity due to the high volume request rate which directly translates to significant costs.
```
As additional info: we also have java producers that don't generate parasitic fetches. Checked by stopping all kafka services one by one to find problem.

So few questions:
1. why aiokafka producer generate parasitic load?
2. why producer try fetch from topic even if it empty (we have some)?
3. is the way to disable this behavior to stop parasitic load on kafka?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.