Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Consumer offsets not committed when using KeyRecordGrouper

未關閉
#271 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
S3
主要語言
Java
星號
58
分支
39
平均合併
2 天 10 小時
30 天內合併 PR
5

描述

When using the S3 connector and grouping records by key we have encountered this error:

> consumer` poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records

Increasing the interval doesn't fix this issue. Looking at the consumer offsets I can see that it's failing to commit any offset, however it is still writing the records to S3 and it's writing more than one batch. Unfortunately the connector appears to be then retry uploading the fetched batches until the connector is killed.

The topic in question has millions of records on it, and if I set the offset manually to skip most of the messages it appears to work OK. When not grouping by key the same topic is ingested to S3 without any issues. In short it appears to be a load issue specific to grouping by key.

With the default `max.poll.records` of 500 and `max.poll.interval.ms` of 300000, I would have thought the consumer offsets would have been written every ~500 records. It's certainly able to process more than 500 records in that interval. Can you advise on what triggers the offsets being committed?

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。