Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Consumer offset not updated in Kafka with storage API + batch mode
- Dominant language
- Java
- Stars
- 37
- Forks
- 45
- Avg merge
- 19h 50m
- Merged PRs (30d)
- 5
Description
When using the BigQuery Storage API and the batch mode, the connector consumer offset is not updated in Kafka, even though records are successfully written and committed in BigQuery and therefore we can't monitor the consumer lag.
As I understand, the offsets are managed by the BigQuery stream, but after getting the committable offsets from the stream, we do not flush them.
https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/main/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java#L210
Is there a specific reason for not flushing the offsets? I see the `flush()` function would raise a version compatibility exception in that case.
_Happy to raise a PR if it would be accepted._
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.