Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Add configurable gRPC keepalive for Storage Write API streams

Aberta
#226 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Java
Estrelas
37
Forks
45
Merge médio
19h 50min
PRs com merge (30d)
5

Descrição

When using the BigQuery Storage Write API (`useStorageWriteApi: true`), gRPC streams are silently closed by Google's server-side infrastructure after ~600 seconds of inactivity. The next `AppendRows` call on the dead stream fails with:

ABORTED: Closing the stream because it has been inactive for 600 seconds

or:

io.grpc.StatusRuntimeException: UNAVAILABLE: Connection reset by peer

This triggers task failures and automatic restarts. In low-traffic connectors (e.g., topics with variable producer rates), this creates a **flapping pattern** — tasks repeatedly fail and restart during idle windows, causing:

- Temporary data delivery delays during stream recovery
- Downstream measurement gaps (consumers expecting continuous flow see holes)
- Log noise that masks real failures

There is no way to configure gRPC keepalive via connector config, JVM system properties, or environment variables. The `BigQueryWriteSettings` builder in `StorageWriteApiBase` does not expose transport channel options.

## Root Cause

In `StorageWriteApiBase.java`, the `BigQueryWriteSettings` is constructed without any transport-level configuration:

```java
BigQueryWriteSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(credentials))
.build();

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start by reading StorageWriteApiBase.java, especially the BigQueryWriteSettings builder, and trace how useStorageWriteApi is configured. Determine how connector configuration should reach the transport channel options and identify the existing tests covering Storage Write API streams. Done means low-traffic streams can use a configured gRPC keepalive and no longer fail solely after the server's idle period.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
google-cloud, grpc, java, kafka
Domínio
data-engineering, stream-processing
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Pouca atividade
Clareza
Razoavelmente clara
Facilidade para iniciantes
48/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.