Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Add configurable gRPC keepalive for Storage Write API streams
- Lingua principale
- Java
- Stelle
- 37
- Fork
- 45
- Merge medio
- 19h 50m
- PR unite (30g)
- 5
Descrizione
When using the BigQuery Storage Write API (`useStorageWriteApi: true`), gRPC streams are silently closed by Google's server-side infrastructure after ~600 seconds of inactivity. The next `AppendRows` call on the dead stream fails with:
ABORTED: Closing the stream because it has been inactive for 600 seconds
or:
io.grpc.StatusRuntimeException: UNAVAILABLE: Connection reset by peer
This triggers task failures and automatic restarts. In low-traffic connectors (e.g., topics with variable producer rates), this creates a **flapping pattern** — tasks repeatedly fail and restart during idle windows, causing:
- Temporary data delivery delays during stream recovery
- Downstream measurement gaps (consumers expecting continuous flow see holes)
- Log noise that masks real failures
There is no way to configure gRPC keepalive via connector config, JVM system properties, or environment variables. The `BigQueryWriteSettings` builder in `StorageWriteApiBase` does not expose transport channel options.
## Root Cause
In `StorageWriteApiBase.java`, the `BigQueryWriteSettings` is constructed without any transport-level configuration:
```java
BigQueryWriteSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(credentials))
.build();
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.