[QUESTION] extremly high Fetch Rate
- Ngôn ngữ chính
- Python
- Star
- 1.4k
- Fork
- 269
- Merge trung bình
- 1 ngày 1 giờ
- Pull request đã merge (30 ngày)
- 6
Mô tả
I have http API service on fastapi that only write to kafka by aiokafka. So I only one producer cause very high fetch rate on monitoring graphs - usually 3.5-4k - that amount is sum of counts rates for different topics. Producer only send to one topic but fetch rate is also high for topics that are have no new messages for along time.
For monitoring we use metric `kafka_server_brokertopicmetrics_totalfetchrequests_total` by topic and see that problem. Searching about metrics got https://newrelic.com/blog/how-to-relic/tuning-apache-kafka-consumers where found following text:
```
A reasonable fetch rate is somewhere around 1 to 5 requests per second per broker because it corresponds to 200 to 1,000 milliseconds of fetch latency. If your average fetch size is consistently hitting a limit and your average fetch request rate is higher than five requests per second per broker, you’re likely creating many small fetch requests instead of fewer large ones. This not only limits the throughput of your application, it can very quickly consume a large amount of your Kafka Broker capacity due to the high volume request rate which directly translates to significant costs.
```
As additional info: we also have java producers that don't generate parasitic fetches. Checked by stopping all kafka services one by one to find problem.
So few questions:
1. why aiokafka producer generate parasitic load?
2. why producer try fetch from topic even if it empty (we have some)?
3. is the way to disable this behavior to stop parasitic load on kafka?
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.