Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
BigQuerySinkTask.preCommit() hangs indefinitely when Storage Write API gRPC call never completes
- Lenguaje dominante
- Java
- Estrellas
- 37
- Forks
- 45
- Merge medio
- 19 h 50 min
- PR fusionados (30 d)
- 5
Descripción
## Summary
`BigQuerySinkTask.preCommit()` → `flush()` → `KcbqThreadPoolExecutor.awaitCurrentTasks()`
blocks on `CountDownLatch.await()` with no timeout. If a Storage Write API `AppendRows`
gRPC call never completes (observed after `ConnectionWorker.resetConnection()` during
stream pool scaling), the write task never counts down the latch. `flush()` waits
forever. The Kafka Connect task thread is zombied until the consumer group fences it
(`max.poll.interval.ms`), triggering a rebalance. The task restarts, makes progress
for a few minutes, and hangs again. Repeats indefinitely.
## Environment
- bigquery-connector 2.13.0
- Kafka Connect 3.8 (Debezium connect image 3.4.3.Final)
- GCP BigQuery Storage Write API
- `useStorageWriteApi: true`, `enableBatchMode: false`
## Thread dump
"task-thread-batch-production-bigquery-sink-stream-2" — WAITING (parking)
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007251724d0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at com.wepay.kafka.connect.bigquery.write.batch.KcbqThreadPoolExecutor.awaitCurrentTasks(KcbqThreadPoolExecutor.java:99)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:191)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.preCommit(BigQuerySinkTask.java:211)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:445)
Gax/gRPC threads are all idle (`waiting on condition` in `ThreadPoolExecutor.getTask()`),
confirming no in-flight gRPC work — the latch is waiting on a write task that will
never complete.
Preceded in the logs by:
com.google.cloud.bigquery.storage.v1.ConnectionWorkerPool createConnectionWorker
INFO: Scaling up new connection for stream name: .../batch_requests/_default, pool size after scaling up 1
com.google.cloud.bigquery.storage.v1.ConnectionWorker resetConnection
## Impact
Consumer lag grows unboundedly. Observed ~1.4B offset lag over 3 days. 94 rebalances
in 13 hours from the zombie→fence→rebalance cycle.
## Workaround
`useStorageWriteApi: false` — the legacy streaming inserts HTTP path has timeouts,
so failed write tasks complete (with an exception) and the latch counts down.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.