Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Consumer offset not updated in Kafka with storage API + batch mode

Aberta
#34 3 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Java
Estrelas
37
Forks
45
Merge médio
19h 50min
PRs com merge (30d)
5

Descrição

When using the BigQuery Storage API and the batch mode, the connector consumer offset is not updated in Kafka, even though records are successfully written and committed in BigQuery and therefore we can't monitor the consumer lag.

As I understand, the offsets are managed by the BigQuery stream, but after getting the committable offsets from the stream, we do not flush them.

https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/main/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java#L210

Is there a specific reason for not flushing the offsets? I see the `flush()` function would raise a version compatibility exception in that case.

_Happy to raise a PR if it would be accepted._

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.