Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Consumer offset not updated in Kafka with storage API + batch mode
- Linguagem predominante
- Java
- Estrelas
- 37
- Forks
- 45
- Merge médio
- 19h 50min
- PRs com merge (30d)
- 5
Descrição
When using the BigQuery Storage API and the batch mode, the connector consumer offset is not updated in Kafka, even though records are successfully written and committed in BigQuery and therefore we can't monitor the consumer lag.
As I understand, the offsets are managed by the BigQuery stream, but after getting the committable offsets from the stream, we do not flush them.
https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/main/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java#L210
Is there a specific reason for not flushing the offsets? I see the `flush()` function would raise a version compatibility exception in that case.
_Happy to raise a PR if it would be accepted._
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.