Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Add configurable gRPC keepalive for Storage Write API streams
- Linguagem predominante
- Java
- Estrelas
- 37
- Forks
- 45
- Merge médio
- 19h 50min
- PRs com merge (30d)
- 5
Descrição
When using the BigQuery Storage Write API (`useStorageWriteApi: true`), gRPC streams are silently closed by Google's server-side infrastructure after ~600 seconds of inactivity. The next `AppendRows` call on the dead stream fails with:
ABORTED: Closing the stream because it has been inactive for 600 seconds
or:
io.grpc.StatusRuntimeException: UNAVAILABLE: Connection reset by peer
This triggers task failures and automatic restarts. In low-traffic connectors (e.g., topics with variable producer rates), this creates a **flapping pattern** — tasks repeatedly fail and restart during idle windows, causing:
- Temporary data delivery delays during stream recovery
- Downstream measurement gaps (consumers expecting continuous flow see holes)
- Log noise that masks real failures
There is no way to configure gRPC keepalive via connector config, JVM system properties, or environment variables. The `BigQueryWriteSettings` builder in `StorageWriteApiBase` does not expose transport channel options.
## Root Cause
In `StorageWriteApiBase.java`, the `BigQueryWriteSettings` is constructed without any transport-level configuration:
```java
BigQueryWriteSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(credentials))
.build();
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Start by reading StorageWriteApiBase.java, especially the BigQueryWriteSettings builder, and trace how useStorageWriteApi is configured. Determine how connector configuration should reach the transport channel options and identify the existing tests covering Storage Write API streams. Done means low-traffic streams can use a configured gRPC keepalive and no longer fail solely after the server's idle period.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- google-cloud, grpc, java, kafka
- Domínio
- data-engineering, stream-processing
- Tipo de issue
- Funcionalidade
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Pouca atividade
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 48/100