Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Consumer offset not updated in Kafka with storage API + batch mode

Open
#34 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
37
Forks
45
Avg merge
19h 50m
Merged PRs (30d)
5

Description

When using the BigQuery Storage API and the batch mode, the connector consumer offset is not updated in Kafka, even though records are successfully written and committed in BigQuery and therefore we can't monitor the consumer lag.

As I understand, the offsets are managed by the BigQuery stream, but after getting the committable offsets from the stream, we do not flush them.

https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/main/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java#L210

Is there a specific reason for not flushing the offsets? I see the `flush()` function would raise a version compatibility exception in that case.

_Happy to raise a PR if it would be accepted._

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.