Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Add configurable gRPC keepalive for Storage Write API streams

Aperta
#226 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
37
Fork
45
Merge medio
19h 50m
PR unite (30g)
5

Descrizione

When using the BigQuery Storage Write API (`useStorageWriteApi: true`), gRPC streams are silently closed by Google's server-side infrastructure after ~600 seconds of inactivity. The next `AppendRows` call on the dead stream fails with:

ABORTED: Closing the stream because it has been inactive for 600 seconds

or:

io.grpc.StatusRuntimeException: UNAVAILABLE: Connection reset by peer

This triggers task failures and automatic restarts. In low-traffic connectors (e.g., topics with variable producer rates), this creates a **flapping pattern** — tasks repeatedly fail and restart during idle windows, causing:

- Temporary data delivery delays during stream recovery
- Downstream measurement gaps (consumers expecting continuous flow see holes)
- Log noise that masks real failures

There is no way to configure gRPC keepalive via connector config, JVM system properties, or environment variables. The `BigQueryWriteSettings` builder in `StorageWriteApiBase` does not expose transport channel options.

## Root Cause

In `StorageWriteApiBase.java`, the `BigQueryWriteSettings` is constructed without any transport-level configuration:

```java
BigQueryWriteSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(credentials))
.build();

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.